The server loads the chunks , you should only need fastcraft on the server ;)
For what I can understand it is indeed terrain generation and chunk loading , which did NOT improve in 1.8 significantly if at all.
Minecraft is odd to run , requires resources inconsistent and does not balance load...