How close are your bases? I find a minimum of ten chunks to be a good measure, so that you'll tend to have very little loading overlap. IE you're not loading their base, and they aren't loading yours.
If the two of you had lots of chunk loaders going, that could be an issue as well.
For RAM, I always prefer Minecraft + 2GB for the OS. So if Minecraft needs 4, you'd want at least 6.
Try starting the server fresh, and having each player ping it, then join in combination, in case it's someone's internet instead of the server.