Solved Thermal Expansion 70,000+ u/s cause TPS drop, HELP!!!

[Phyrax]

Since a month ago, I've not needed to use opis at all. In fact, server TPS has been great. The server has only been up a little more than a month. Recently, we've had a rash of TPS drops before the 4 hour restart. Not mentioning my cron job sometimes fails, curse you CentOS...

Anyhow, as of this moment, my server is becoming almost unplayable. Server tick rate is down to just 6, sometimes 10-11. So as a bench mark test, I went through the world yesterday and broke player's chunk loaders, putting them in chests for later use. Swapped World Anchors for personal anchors, and even nerfed the chickenchunks.cfg to load no more than 25 chunks per loader. So I figured, if the chunks would unload, surely this would fix TPS issues.

My hope was in vain... 15 minutes later, TPS is still less than 10. I then remembered, dynmap. After pausing that, still no change. Then I remembered Extra Utilities, the lava pump, I spent the better part of an hour going through the nether breaking all of them and placing them in a chest for the player. As you can see in the screenshot, only two were left.

As of right now, I'm the only one online and tick rate is absolutely terrible. 6TPS with one player loaded chunk set, which is spawn and no machines etc... (aside from 6 trade-o-mats).

A further investigation with Opis reveals a total update time of 70,000+ updates a second, with the bulk of ticks coming from ThermalExpansion Grid Tick Handler. WHY? Why when no chunks are loaded with TE items does TE need that many ticks??? Is it not properly cleaning up after itself?

Following a reboot, the Grid Tick Handler is under 10,000, and tick rate is once again 19.8TPS

In the following imgur album you can find all the screenshots from opis, aswell as output from the /gc command.

http://imgur.com/a/wPTht

Current modpack: FTB Monster (Recommended)
Server: CentOS 8GB Ram, 6GB allocated to MC
Quad Core CPU 3.2GHz

Jar Flavor: MCPC Plus 1.6.4 R2.1 Forge965 Build 251

Before you say, "you need more ram" notice the ram isn't maxed out, besides if it were, java would crash causing an out of memory issue. There is no excuse for a mod not cleaning up after itself when the chunk unloads...

Has anyone else had this issue?

 

[DZCreeper]

Conduits are lightweight but when you have 70000 of anything, do you really expect it to be lightweight?

 

[Phyrax]

Take the time to look at the provided screenshot album. You'll notice that I'm the ONLY one online and that no chunks are loaded aside from spawn, and two ender thermic pumps, which include one pipe a piece. Show, how can 70,000 pipes be loaded? I don't "expect" anything, other than information that is preferably detailed and informative...

 

[DZCreeper]

By chance do you have any untested features enabled like Logistics Pipes routing enabled through Thermal Expansion itemducts?

Any MFR farms using itemducts? I have seen and personally witnessed them having clogging and routing issues with those.

 

[Phyrax]

I personally only use Ender IO and have zero TE pipes in my home. Furthermore, in the data provided, I am using my alt character with no player data on the server. This toon has never placed blocks or anything.

Also, effective yesterday, all players on my server are required to turn in any TE pipes in favor of ender IO at spawn within 3 days. Being that it's a small community I won't have any problems with that. However, to ensure that I don't, following the deadline I will be teleporting around breaking any pipes at anyone's home via HawkEye logs.

It still doesn't make sense though, why would the pipes stay loaded even if the chunks arent?

 

[DZCreeper]

Can you try without MCPC+ ? Sounds like the routing thread for the pipes is staying open even when the pipes don't technically exist in the world.

 

[Phyrax]

I will gladly try it. I want to give my users a notice though since the bulk of them, me included, love the /home and /tp functionality that we're so accustomed to. Will post my findings afterwards.

 

[DZCreeper]

If the cause does turn out to be MCPC+, a lot of the functionality you most likely want is replicated in ServerTools, a mod that runs on Forge with no need for MCPC+.

For instance, it has /home, /tp, and plenty of other useful commands which are well documented. http://servertools.matthewprenger.com/doku.php?id=start

 

[Connor Gavitt]

If the cause does turn out to be MCPC+, a lot of the functionality you most likely want is replicated in ServerTools, a mod that runs on Forge with no need for MCPC+.

For instance, it has /home, /tp, and plenty of other useful commands which are well documented. http://servertools.matthewprenger.com/doku.php?id=start

But then no prism

 

[DZCreeper]

But then no prism

It actually has the ability to log every block broken by a player or fake player. Creates a different file per day but other than that its okay, probably not as fast as Prism running on MySQL though. http://servertools.matthewprenger.com/doku.php?id=modules:core

 

[Connor Gavitt]

It actually has the ability to log every block broken by a player or fake player. Creates a different file per day but other than that its okay, probably not as fast as Prism running on MySQL though. http://servertools.matthewprenger.com/doku.php?id=modules:core

^

 

[Phyrax]

I believe I found the culprit. I ran the server for approximately 8 hours today on the default jar file. Unfortunately players kept crashing one after the other during that time, but TPS stayed at 20, the TPS command reported 23-26 to be honest.

Then, to alleviate the constant crashes by players I switched the jar file to MCPC, a few hours later, TPS went to crap. So I buckled down and checked my Hawkeye logs (Yes I still use it) to find placed conduits. Being that it's a small server, it only took me an hour to clear out all item ducts and energy conduits. In one persons home, i found his boiler setup with pipe that crossed the chunk boundary. Guess what happened when I broke it... 3 stacks of items flew out at me. This happened at another individuals house, again when the pipe crossed the chunk boundary, I broke it, and seeds spit at me this time.

So anyhow, now I'm the only one on the server, and TPS is at 20.

I'm going to leave this thread set to 'problem' for at least three days to see if there are any more problems since I removed those conduits/ducts. Though this brings up another issue, read time outs on EVERY player when running the default jar... ugh idk, and I'm done troubleshooting for the day.

 

[Jadedcat]

Since a month ago, I've not needed to use opis at all. In fact, server TPS has been great. The server has only been up a little more than a month. Recently, we've had a rash of TPS drops before the 4 hour restart. Not mentioning my cron job sometimes fails, curse you CentOS...

Anyhow, as of this moment, my server is becoming almost unplayable. Server tick rate is down to just 6, sometimes 10-11. So as a bench mark test, I went through the world yesterday and broke player's chunk loaders, putting them in chests for later use. Swapped World Anchors for personal anchors, and even nerfed the chickenchunks.cfg to load no more than 25 chunks per loader. So I figured, if the chunks would unload, surely this would fix TPS issues.

My hope was in vain... 15 minutes later, TPS is still less than 10. I then remembered, dynmap. After pausing that, still no change. Then I remembered Extra Utilities, the lava pump, I spent the better part of an hour going through the nether breaking all of them and placing them in a chest for the player. As you can see in the screenshot, only two were left.

As of right now, I'm the only one online and tick rate is absolutely terrible. 6TPS with one player loaded chunk set, which is spawn and no machines etc... (aside from 6 trade-o-mats).

A further investigation with Opis reveals a total update time of 70,000+ updates a second, with the bulk of ticks coming from ThermalExpansion Grid Tick Handler. WHY? Why when no chunks are loaded with TE items does TE need that many ticks??? Is it not properly cleaning up after itself?

Following a reboot, the Grid Tick Handler is under 10,000, and tick rate is once again 19.8TPS

In the following imgur album you can find all the screenshots from opis, aswell as output from the /gc command.

http://imgur.com/a/wPTht

Current modpack: FTB Monster (Recommended)
Server: CentOS 8GB Ram, 6GB allocated to MC
Quad Core CPU 3.2GHz

Jar Flavor: MCPC Plus 1.6.4 R2.1 Forge965 Build 251

Before you say, "you need more ram" notice the ram isn't maxed out, besides if it were, java would crash causing an out of memory issue. There is no excuse for a mod not cleaning up after itself when the chunk unloads...

Has anyone else had this issue?

We see this a lot on servers using MCPC+. The TE Tick Handler grid isn't just TE conduits, its item ducts, conduits, fluiduct, MFR rednet energy cables and any other mod hooking into the TE Grid. Monster has several problem mods that cause cross mod issues with TE3 and other mods. That's why this never occurs on FTBLite2 servers. One thing that will cause that every single time is Open Block tanks connected to fluiducts. Railcraft frequently does interesting things with item ducts.


Honestly 95% of all bugs in Monster can be solved with the following steps

• Remove Railcraft
• Remove Universal Electricity mods
• Turn off Open Block tanks
• Remove MCPC+
• Try to duplicate.

 

[PsionicArchon]

I'm going to assume that the TE Handler deals with anything hooking into the TE API. Right now there are dozens of mods that use the TE API, most of which do so for Redstone Flux, including Ender IO.

I have thousands upon thousands of Energy Conduits in my base. Right now they're still the most server friendly power transfer method, with the new Mekanism cables a close runner up. I have my doubts that anything TE related is causing your tickrate issues.

Some things also worth mentioning: I use MCPC+ (latest version).
I have TE fluiducts pumping liquid experience away from Open Blocks tanks.
I have a Steam Boiler outputting into fluiducts.
I have itemducts providing charcoal to my boilers.
I have Galacticraft and MFFS installed (both UE mods) alongside Calslavia Core.

Everything's green on my screen. My tick handler barely jumps above eighty.

It may be worth it for you to manually update a few troublesome mods on the off chance that the FTB team hasn't. Namely, Ender IO (which may very well be the culprit). The latest version can be found here: http://ci.algent.fr/job/EnderIO MC/

According to the wiki Monster is still using version 0.11.0.268. Holy bajesus, that's one hundred versions behind... Quite a few optimizations were made to the Ender IO power network. If the users of your server don't mind manually downloading a few configs, check these mods as well, to be sure you're using the latest version.

Extra Utilities
Logistics Pipes
MFFS
Calclavia Core
Universal Electricity Core
Modular Powersuits
Open Blocks
Minefactory Reloaded: According to the wiki on this one Monster is using version 2.7.5-352. Great googly moogly that's pretty far behind. I'm using 2.7.7B1-471 right now. You may have trouble acquiring this version though as the jenkins is down.


Anything using the Thermal Expansion RF API for that matter.

 

[Jadedcat]

I'm going to assume that the TE Handler deals with anything hooking into the TE API. Right now there are dozens of mods that use the TE API, most of which do so for Redstone Flux, including Ender IO.

I have thousands upon thousands of Energy Conduits in my base. Right now they're still the most server friendly power transfer method, with the new Mekanism cables a close runner up. I have my doubts that anything TE related is causing your tickrate issues.

Some things also worth mentioning: I use MCPC+ (latest version).
I have TE fluiducts pumping liquid experience away from Open Blocks tanks.
I have a Steam Boiler outputting into fluiducts.
I have itemducts providing charcoal to my boilers.
I have Galacticraft and MFFS installed (both UE mods) alongside Calslavia Core.

Everything's green on my screen. My tick handler barely jumps above eighty.

It may be worth it for you to manually update a few troublesome mods on the off chance that the FTB team hasn't. Namely, Ender IO (which may very well be the culprit). The latest version can be found here: http://ci.algent.fr/job/EnderIO MC/

According to the wiki Monster is still using version 0.11.0.268. Holy bajesus, that's one hundred versions behind... Quite a few optimizations were made to the Ender IO power network. If the users of your server don't mind manually downloading a few configs, check these mods as well, to be sure you're using the latest version.

Extra Utilities
Logistics Pipes
MFFS
Calclavia Core
Universal Electricity Core
Modular Powersuits
Open Blocks
Minefactory Reloaded: According to the wiki on this one Monster is using version 2.7.5-352. Great googly moogly that's pretty far behind. I'm using 2.7.7B1-471 right now. You may have trouble acquiring this version though as the jenkins is down.


Anything using the Thermal Expansion RF API for that matter.

The wiki is not correct. Correct version numbers are found here: https://feedthebeast.atlassian.net/wiki/display/PML/Monster

 

[epidemia78]

Upon reading this thread I checked my list (mine not FTB's) and quite a few mods needed to be updated. Heres hoping it doesnt corrupt my world.

 

[MajPayne21]

Stuffed itemducts cause TPS issues on my server all the time.

The most common culprit is itemducts moving wood from a DSU to Redstone furnaces. Even if we use servos to reduce output to 1 item, over time they get severely stuffed (we're talking 20,000+ items) and wreck TPS.

I have mandated AE export buses anytime people are pulling from a large source into a finite sink (such as DSU to single stack capacity machines). There shouldn't be any issues if the destination has plenty of room.

I don't know why the itemducts get stuffed in the above scenario, but we see it all the time.

Sent from my SCH-I535 using Tapatalk

 

[Phyrax]

Thanks for all the input guys and gals. I've logged in today to see the TPS at just 15, while this isn't nowhere near the crappy TPS I was getting, it's still under 20, and the perfectionist in me wants to make sure I've removed all conduits and ducts from TE. However, I've found a flaw in Hawkeye, 200+ pages of duct/conduit activity.... I'm really not going to go through all that page by page. So i logged into ssh and figured I would share a little code snippet for you guys. What I'm doing instead is getting a list of players who have logged in since the last log rotation, seeing as I manually rotate my logs every two weeks, this will give me an idea of who's homes were loaded and where I should look for conduits and ducts.

Here's the command, note you will need to have terminal access and this command is specific to a unix/linux environment, not minecraft:

cat server.log | grep logged | awk '{print $4}' | cut -d '[' -f 1 | sort | uniq

What this does is read your server.log file (change as needed) and prints out a list of users without duplicates that have logged in, which is logged by the file.

What I've done thus far, breaking pipes etc, has proved that it's working, and I assume I've left a few untouched, by looking at my list of users, I know I have. So time to teleport and clear some pipes. Oh WorldEdit, if ducts/conduits weren't an invalid block type... I'd love it even more. I'll post more status tomorrow.

Currently the GTH (Grid Tick Handler) for TE is clocking at ~24,000 updates a second, hopefully tomorrow it will be even less. Regardless, it's way better than ~75,000....

 

[PsionicArchon]

...

What I've done thus far, breaking pipes etc, has proved that it's working, and I assume I've left a few untouched, by looking at my list of users, I know I have. So time to teleport and clear some pipes. Oh WorldEdit, if ducts/conduits weren't an invalid block type... I'd love it even more. I'll post more status tomorrow.

Currently the GTH (Grid Tick Handler) for TE is clocking at ~24,000 updates a second, hopefully tomorrow it will be even less. Regardless, it's way better than ~75,000....

Loading up your world in mcedit would be a might bit faster. You can just select all chunks, and remove every conduit/duct simultaneously using its item id. It sure beats trying to find all of the ducts by hand.

 

[Phyrax]

Right, I've already thought of that. But being that it's an online public server, I would have to download the entire map, edit the chunks, then re-upload it. With a 25GB world folder, I would rather not...