Minecraft is not that much of a video memory prostitute, its more a CPU problem than anything else.. First of all, use parameters in java, there is a post in the forums somewhere with recommended parameters that will help a lot, also, BE SURE YOU ARE USING 64 BITS JAVA. I run minecraft with a GT 520 (Its a bitch around the Ge Force bitches, but for me, makes the job done (even in skyrim)) and 4 gigas of ram with a dual core 64 bits, and I dont encounter much problems.. So im sure you are doing something wrong somewhere down the line.. Some things to check:
Java 64 bits?
O.S. 64 bits?
Java process using more than 1 core?
Does your graphic card drivers support correctly OpenGL?
Do you have minecraft installed in a hard drive that has a quick format? (Anything but FAT16/32 should work.. If windows, ntfs if linux ext4)
Do you have your harddrive in good health? Check for fragmentation and for cluster errors, your OS should give you the tools needed.
Is your Power Supply enough for you video card and all the stuff? Running that machine with a 300 or 400 watts is a really bad idea.
Have you checked in your bios if you can overclock you cpu? (only do this if you have good ventilation, and never overclock it more than it can take without getting too hot)