Unleashed. TPS drop and CPU usage spike.

[eyeball114]

Unleashed version 1.1.3

Server specs
MotherBoard: ASRock B75M R2.0 LGA 1155 Intel B75
CPU: Intel Core i5-3470 Ivy Bridge 3.2GHz (3.6GHz Turbo Boost)
RAM: CORSAIR Vengeance LP 16GB 240-Pin DDR3 SDRAM DDR3 1600
Boot Drive: Toshiba 240gb Notebook Drive
Storage HDD 1x WD 1TB Green + Notebook Drive x1 WD 160GB(backups)
PSU: Corsair CX430M 430w Modular

This all started this week.

We are always at 20 - 19 TPS. All of a sudden we are dropping to around 15 and my CPU usage goes from around 16% and under for each core to bouncing all the way up to 100% and under.

I narrowed this down to when a player joins. Hes been with me on my servers forever and I don't believe this is anything he is doing on purpose. His home is massive. Its probably the biggest real estate on the server and has 20+ boilers and a massive AE system. The CPU usage jumps insanely when he logs in and the TPS drops shortly after to about 15.

He, and 1 other player, have also told me that something is loading their bases. Chunk loaders are disabled on my server. Their boilers remain on burning fuel but everything else stops. Their fuel production stops meanwhile their boilers are burning through their supply.

After he is on for a few minutes the TPS settles back around 19 and stays there until someone else logs in. Dropping it again to around 15 and then slowly rising back to normal. We have never had any TPS when people join on in the past 2 years.

Max memory is 5461 MB
Allocated is 5065
and the Free memory goes anywhere from 3500 down to 2300

Any help is greatly appreciated. I can't understand whats up. If anyone wants to help and need any other info just let me know and Id be happy to produce it.


Plugins:
Announcer
Citizens
ClearLag
Essentials
GriefLog
GroupManager
LWC
TreeAssist
WorldBorder
WorldEdit


EDIT: We disconnected his AE system and it drops the core usage a bit. The highest they now go is 66% which is still very high for what we usually have.

 

[MigukNamja]

Re: "Their boilers remain on burning fuel but everything else stops"

If the server can't keep up, the boilers *may* not be completing their loop and returning to an idle state. I'm not saying the boilers are to blame, per se, but that their symptoms may be being caused by the server not being able to keep up at the standard 20 TPS.

Unfortunately, Railcraft code is not on github as far as I can (can't) find, so I can't verify this theory by code inspection.

Re: "We disconnected his AE system and it drops the core usage a bit."

When AE is disconnected and he logs out and no-one else is in the area, do the boilers shutdown and stop burning fuel ?

 

[eyeball114]

He tells me that when no one is in his chunk that the fuel keeps burning in the boilers but all production of his fuel stops. Eventually running the boilers dry.
We havent tested the AE disconnected while he is offline. Just when he is out of the chunk. Ill have to wait until he gets back on to see if his boilers keep going.
The CPU usage is the most confusing to me. I can literally watch my monitor go from a steady flat line around 20% to all out 4 year old coloring outside the lines.

 

[MigukNamja]

Does it happen only when he logs in, or anytime someone loads his base's chunks ?

Also, what happens if logs out somewhere else, and then logs back in ?

In other words, him, or his base's chunks are to blame ?

 

[eyeball114]

Just Tp'd to his base and its anyone in his chunk that will make the usage spike. So its not him but his base.

 

[MigukNamja]

Since his boilers are burning fuel anyways, would he mind it if you disabled them by breaking 1 block on each one, preferably one of the fuelbox blocks ?

Narrowing it down to 1 or more CPU-hungry mods or components of a mod is likely going to be a PITA and a search for a needle in the haystack. Just start with the most suspected/obvious ones that stick out. 20 boilers sticks out. I stopped at ten(10) 36HP boilers with 18 engines each when I was experiencing client lag.

Also, shutting down MJ and EU power generation entirely may help find the issue. If that 'solves' it, then you can start turning on components at a time until the issue comes back.

Does he use a lot of BC pipes, by any chance ?

 

[eyeball114]

Well since we started having this issue this past week we narrowed it down to anything he placed in the past 7 days. Thats was 8 boilers and 5 mining laser drills. We shut off all 8 new boilers and let them run dry including the buffers in the steam engines. The lasers were shut off as well. With little to no change in CPU usage.

He uses no BC pipes. His AE System is massive. I mean if you would like to come on the server and see it you would be speechless for a minute lol.

 

[MigukNamja]

LOL, yeah, that might be it. If simply turning off AE causes CPU to resume to slightly more sane levels, then....

Exactly how massive we talking ? I think I have a "big" AE system when my power consumption as reported by the ME Controller is in the 200+ units/t range. I'm at around 300+ units/t now, not including 4x Quantum Gates, (2 pairs), and it's one of the biggest AE systems I've ever built. But, it's also not in 1 chunk. I have my Controller in 1 chunk and most of the rest of AE stuff in a MystCraft age.

 

[eyeball114]

Everything is hooked to one Controller. Here it all is:

 

[MigukNamja]

Oh my, am I seeing that correctly ? MAC looks sane (5x5x7 ?), but fourty-five (45 !!!!) ME Drives ?

I don't see any/many ME Storage Buses. He could switch to DSUs and save a *lot*.

And...holy cow. 1393 ME cable blocks ?

Yeah, that's pretty big. Not sure if AE was meant to scale well to those levels.

 

[eyeball114]

He had 2300 cables last week but its cutting down with the use of tesseracts. Could tesseracts be an issues here?

 

[MigukNamja]

I use maybe 10 Item Tesseracts total in my builds, up to 30 or 40 Liquid Tesseracts, and a handful of Energy Tesseracts. All of my Item Tesseract pairs have an ME Basic Export bus pointed at one side of the pair. Never had a problem with them, though.[DOUBLEPOST=1380662832][/DOUBLEPOST]Is he using an Item Tesseract channel for 1 kind of item, or trying to fit multiple item types though ? Unless something on the other side is always pulling everything out, it will head-of-line block.

 

[eyeball114]

He has 24 boilers total, we are in the process of shutting those off now. This is becoming a pain in the ass.

 

[MigukNamja]

Oh, I feel your pain. I had to abandon a world and revert to a much older save when I couldn't figure out a resource issue. Mine was so bad I couldn't login. This was on 1.4.7 and I was admittedly doing several non-server-friendly things.

 

[eyeball114]

Yeah when i first started out I had that happen a few times lol. These guys have been loyal to the community and never caused any issues and I would feel horrible suggest a chunk reload. Its not causing any server lag. I just dont want my CPU exploding.

 

[MigukNamja]

I forgot to ask - what are your FTB server java start arguments ? I don't have experience with a server that big, but you may need to bump up the defaults if you haven't already. I'm only using 1GB of RAM in total out of a small 2GB VPS system, but here are my java args:

MAXHEAP=1500
MINHEAP=512
PERMSIZE=128
HISTORY=1024
CPU_COUNT=2
INVOCATION="java -Xmx${MAXHEAP}M -Xms${MINHEAP}M -XXermSize=${PERMSIZE}m -XX:+UseConcMarkSweepGC \
-XX:+CMSIncrementalPacing -XXarallelGCThreads=$CPU_COUNT -XX:+AggressiveOpts \
-jar

 

[eyeball114]

#!/bin/bash
java -Xmx6G -Xms5G -jar mcpc.jar nogui
read -p "Press [Enter] key to continue..."

We barely use half of whats allocated even when this area is loaded.

We shut his boilers down and saw a slight decrease in CPU usage but still jumping to around 73% at times.

 

[MigukNamja]

What's your permSize ? I'm not an expert on Java, but if it's too small or missing, it has been known to cause server CPU issues. I'm using 128MB, but you may need to use more, like 256MB or more, ex:

java ... -XX:permSize=256m

 

[eyeball114]

I have no idea where may I find this? lol...

 

[MigukNamja]

Based upon your previous response with your server args, it looks like you're not, and I'm pretty sure the default is non-optimal (too small). Insert this just before "-jar ..."

-XX:permSize=128m

If that doesn't work, try 256m