I did change a few of the arguments, earlier, with no success. This is what it looks like currently.
java -Xms2048m -Xmx3072m -XX:PermSize=512m -jar FTBServer-1.6.4-965.jar nogui
I have also done it on my own client too. The server has been tested with both vanilla and FTB Unleashed 1.1.4...