First try increasing the amount of memory for your client under the launcher options. It should let you go up to 2GB.
If that fails then you will need to install 64bit java, so you can have access to all your RAM. You will also need to change how you start FTB, as the default setup only uses...