Why do you have an i3 with 16GB of RAM? Low power CPU with high power RAM? Were you told by someone that the only thing that matters to gaming is RAM? Because it doesn't. But, I digress.
Don't try adding too much memory, or too little. A good balance is about 2 GB. Too much and Java doesn't know what to do with it, too little, Java can't do anything anyway.
I'm going to guess that FTB isn't running with your graphics card, which seems to be the most common cause of this problem. You'll need to open the control panel for your graphics card (usually the Nvidia or Catalyst Control Panel) and set the FTB launcher to use your graphics card.