Yeah, it happens when I'm close as well. I did a ping on the server, 100 ms and no packet loss. Not great but I don't think it's traditional lag. I would agree someone is doing something that is draining the server.
I know it's not me since the most advanced thing I own is a stone pick axe and...