Nothing to do with upping the heap or permgen sizes. Those are not total RAM allocated anyway, but rather just a small part of java's total memory consumption.
Simply running a heavily modded Minecraft client with 32bit java fails. Some mods are heavy enough to do this on their own. Try terrafirmacraft for example. With only Forge, PlayerAPI and TFC installed, Minecraft will crash and burn within 2 minutes of gameplay on my system, guaranteed, 100% of the time. If I simply use 64bit java instead, it works fine without a hitch. Without any java arguments set.
Generally, try using the mininmum amount of java arguments (and memory allocation) necessary to make your client run, unless you really know what you're doing. Many things will actually decrease performance if set incorrectly.
For reference, my standard arguments string is: -Xms256M -Xmx768M -XX:MaxPermSize=128m -XX:UseSSE=3 -XX:+AggressiveOpts -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
The last three of which are set by the FTB launcher automatically, and the last two of which are redundant if you use java 7, because CMSGC is on by default then. MaxPermSize default is 64, and I doubled it for safety, but it certainly doesn't need any more. UseSSE=3 directs java to specifically target the capabilities of my CPU architecture when it does runtime compilation, though I have no idea whether or not that's on by default. Finally, my heap size settings are 256 minimum, 768 maximum. I have yet to see my client need more than 600 in the ingame F3 screen.
