Solved Unusual ping spikes/high ping

[PsionicArchon]

I'm running into a peculiar issue with a small server I run.

I'm looking to bounce ideas off of someone as I'm completely stumped.

I've run every test I can in an attempt to diagnose the problem.

Some information:
I am running mcpc+
I am using a VPS
Mod list: http://pastebin.com/5ADQEp5J

Processor usage sits between five and fourteen percent.
Disk IO is within acceptable ranges (75+)
Ping to the actual vps fluctuates between ten and fifteen ms.

Ping plotter shows no routing issues, all paths are between ten and twenty as I would expect.
Pinging the server for three minutes using command prompt yielded no unusual fluctuations.

Hell, I even installed a Teamspeak 3 server so that I could connect to it simply to get another readout on my ping. It never went above twenty ms.

Available heap space is well with an acceptable one gigabyte out of the two I've dedicated. Permgen used is also with an acceptable range at one hundred megabytes out of the three hundred and fifty I've dedicated.

Using htop to monitor strain on individual processor cores suggested nothing abnormal. Performance across all four cores is within an acceptable five to fifteen percent.

Tickrate on the server never drops below 19.99, and there are no run away grid tick handlers to speak of.

Nothing is happening that would immediately indicate a problem.. which in and of its self is a problem. There's no pattern to these strange ping spikes. There's nothing happening when they occur (I'm standing in my base running tests). There's nothing wrong with the VPS. My connection to the VPS outside of the server is perfect.

Beyond that, my ping to the server is abnormally high, idling at around 107ms.

Here's an example screenshot of Opis detecting a spike: http://imgur.com/I4MGFDb

Can anyone shed some light on this, or at least point me in the direction of someone that can?

 

[Connor Gavitt]

Erm what units are you using for disc IO?

I'll assume you are on Linux- run this to find your real read and write per second

dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync

Also provide your Sever.log and your forge-0.log


What modpack are you using?

These will help provide us with info as to what's going on.

 

[PsionicArchon]

The lowest I've seen my disk IO drop was 60MB/s. I was running that exact command. It's usually somewhere between 75-85MB/s. At times Disk IO will drift upwards of 105MB/s, which seems pretty reasonable for a budget VPS.

It's a basic Linux box running the latest version of the Sun JVM on Ubuntu 11.10.

It's a custom modpack closely resembling Monster. I posted the mod list in the original post.

Here's the server.log: http://pastebin.com/3X5qqHqy
Forge log: https://www.dropbox.com/s/xun6ral1d1azs0f/ForgeModLoader-server-0.log.1

I had to dropbox it. Apparently pastebin has a paste size limit.

Edit: I just ran a few tests locally on a secondary computer. My ping to the server, over a lan mind you, is over 70ms... It appears that a mod has gone screwy.

Edit 2: JMX profiling reveals nothing out of the ordinary. http://i.imgur.com/FATWzx1.png
Those initial spikes were generated while the server was booting. Operational processor usage is quite low.

Also, let me know if anything here looks out of the ordinary: http://i.imgur.com/1ggaIr7.png

Edit 3: I just ran a massive benchmarking utility on the VPS. Everything appears to have scored pretty well. Plenty well enough to run a small server..
UnixBench: 2059.1
I/O rate: 171.0 MB/second
Bandwidth rate: 76.7 MB/second

Welp. That's all the testing I can do. Hopefully I've provided more then enough information.

 

[Connor Gavitt]

What are you using for the anti X-ray? With a low read write such as 60MB/s that may be causing it.

 

[DZCreeper]

Is this VPS using normal or solid state drives? That's quite a bit of IO for normal hard disks, especially if they are SATA drives and not SAS drives.

 

[PsionicArchon]

There are no anti x-ray features enabled (not intentionally at least). I have the orebfuscator disabled as well.

This is an older node, as far as I can tell it hasn't been upgraded, and still uses traditional hard drives. I/O speeds vary throughout the day, though when/if they do drop it isn't for long.

I've been monitoring that JVM profiler all day as I played, nothing red flagged for me. I'm going to run warmrost just to get a bit more detail. Unfortunately this is an OpenVZ VPS, and thus I am not able to monitor active disk I/O usage. Unless someone knows of a way to do this.

 

[Connor Gavitt]

Is this VPS using normal or solid state drives? That's quite a bit of IO for normal hard disks, especially if they are SATA drives and not SAS drives.

Oh yes HDDs easily get 100MB/s SAS gets even up to 300MB/s at 10k rpm 80-300Mb/s per drive, if you have had raid 0 or 10 array it doubles some high end HDDs can outperform low end SSDs as well and cost about the same.

There are no anti x-ray features enabled (not intentionally at least). I have the orebfuscator disabled as well.

This is an older node, as far as I can tell it hasn't been upgraded, and still uses traditional hard drives. I/O speeds vary throughout the day, though when/if they do drop it isn't for long.

I've been monitoring that JVM profiler all day as I played, nothing red flagged for me. I'm going to run warmrost just to get a bit more detail. Unfortunately this is an OpenVZ VPS, and thus I am not able to monitor active disk I/O usage. Unless someone knows of a way to do this.

can you list your plugins.

 

[PsionicArchon]

There are two plugins installed, Openinv, and WorldEdit. It's a small server now, and the friends who use it all visit our Teamspeak so the need for administrative plugins is low.

I noticed that these random ping spikes increase with time. After leaving the server running for an entire night I could barely move, ping sat at around 400 consistently. Tickrate however was a flat 20. In fact, nothing was out of the ordinary.. A mod is definitely doing something that it isn't supposed to.

At the moment, I'm leaning towards the calclavia updater. I've seen it spike pretty high on the jvmprofiler.

Edit: Had this happen while I was away: http://pastebin.com/kKnp4FMP

I came home having been gone for an hour to a locked up server. My jvmprofiler refused to poll the server so I couldn't get a readout on the offending thread. The VPS was completely responsive, I had no trouble logging in, and running a top command. %wa was at 0.0 and st was at 0.0. CPU usage according to htop was around twelve percent.

I'm going to assume that either 1. Linux is doing something derpy with I/O priority or 2. A mod is killing the server. Either way, I'm about done. I'm going to cobble together a 1.7 pack today, and move on. This has been an on and off problem for over six months now.

Edit 2: Welp. I found one of the culprits. It turns out that it was the Calclavia Core updater nuking my server periodically. I have no idea why this even occurs. Out of sheer desperation I made a fresh backup, and began removing likely suspects one by one. Upon removing Calclavia Core (and it's adjoining mods) I am no longer being hit with these massive ping spikes.

 

[PsionicArchon]

Well, turns out my lucky guess wasn't so lucky.

The problem 'appears' to be on the VPS providers end.

Using a utility known as glances gave me a more accurate live readout on what the CPU is doing during these lag spikes.

For some reason the CPU% will drop to 0.0 or 1.0 without provocation. Is this an issue on the providers end? Has anyone else experienced this before? I've never seen CPU usage plummet into a lag spike, normally it's the other way around.

Edit: Contacted my provider, turns out there was a cap on my CPU usage after all. After a quick back and forth they ended up increasing the cap. Everything's running smoothly. Good service all around, and a six month old problem finally solved. You can close this thread.