Sorry, I was in a bit of rush posting the response. Anyway, I figured out the problem. It seemed that the server and client were running together on one core and it couldn't quite keep up. I just shifted the affinity on it to any of the cores he had except for the first and it fixed all the stutters.
