Problem Server Can't Keep Up

[UntoldForce]

Hi, I am running Infinity (previously 1.3.4, now 1.4.1) with a few added mods that my friends enjoyed. I am encountering massive lag spikes of 10 seconds or more after moving to 1.4.1 and I was hoping that someone here could help me identify what is causing it. The console is spamming this approximately every 20 seconds:

[Server thread/WARN]: Can't keep up! Did the system time change, or is the server overloaded? Running 3613ms behind, skipping 72 tick(s)

Here is the FML startup log file for the server, which should detail some worrisome things (the world has a memory leak, and I don't know what is causing it).

https://www.dropbox.com/s/9m7q2ahvndvjon1/fml-server-latest - Copy.log?dl=0

I also ran a /debug profiler to figure out what is going on, and there is some crazy stuff happening here. I'm no mod developer but it looks like absurdly bad garbage collection or some kind of recursion going on.

http://pastebin.com/6dGgktgJ

Also very strange... Mystcraft created a DIM_MYST-2147483648 after moving to 1.41 which I've never seen before. Since I know that it is most likely one of the extra mods causing this, I will list them all here:

Base: FTB Infinity 1.4.1
Added mods:
1.7.10_AM2-1.4.0.008 (Ars Magica 2)
AnimationAPI-1.7.10-1.2.4 (required for Ars Magica 2)
backpack-2.0.1-1.7.x (People love backpacks)
ExtraCells-1.7.10-2.2.67bNIGHTLY (Extra cells for fluid ME storage)
LycanitesMobsComplete 1.10.13.2 [1.7.10] (this is a challenge server, Lycanites makes it that way)
MCA-5.0.1-1.7.10-universal (Minecraft Comes Alive, for having kids)
Mekanism-1.7.10-8.1.1.225
MekanismGenerators-1.7.10-8.1.1.225
MekanismTools-1.7.10-8.1.1.225
ModularPowersuits-0.11.0.7-QMX
ModularPowersuitsAddons-0.11.0.7-QMX
Numina-0.4.0.7-QMX (required for Modular Powersuits)
RadixCore-2.0.1-1.7.10-universal (required for MCA)
Reliquary-1.2
secretroomsmod-1.7.10-4.7.1.413

If anyone can help me find the mod(s) that are causing this, I would be eternally grateful.

 

[UntoldForce]

My apologies for forgetting to add this in but I am running Win 7 x64 on a quad-core with 4 GB RAM allocated to Minecraft. The pack is FTB Infinity 1.4.1 (as stated above) but the map has been migrated since the original Infinity public launch. Average server load is about 4-6 people, but this happens even without anyone in the server. Thanks in advance to any knowledgeable people who can help.

 

[Shybella]

It's difficult to tell what the issue may actually be. I'd start by profiling JVM with Warmroast: https://bukkit.org/threads/warmroas...ol-to-figure-out-why-a-mc-server-lags.153336/
( Other than mem leak detections but I don't think thats directly causing the problem)

Run that and post back the results when it starts happening. We can continue from there.

 

[Baron]

I got the same problem too.

People say its a cauldron problem. Unsure. Do you use cauldron? If so what version.

https://github.com/XCompWiz/Mystcraft-Issues/issues/66

 

[UntoldForce]

Thanks Shybella, Warmroast seems to be a better overall profiling tool than Opis. I find that Opis is very handy for finding out specifics but not great for the overall picture. I found out that one of the players set up a chunk loader in the Deep Dark and decided to put all of his BigReactors reactors and turbines in there. I asked him to move them to a different dimension and that helped somewhat though not all the way.

I let Warmroast profile my server for about 12 hours to get a good idea what is happening. Unfortunately the "Server can't keep up" warnings didn't happen last night but it did let me see that Lycanite's Mobs is taking up about 1/3 of the server CPU time. I estimated that it would be high, but not that high. I'll probably have to change Lycanite's spawn rates in the config, but this is a hardmode server meant to be brutal. Lycanite's keeps the brutality even for late-game players, since the flying mobs can still attack those with armor that allows flying.

Baron, I am using Forge instead of cauldron. I would use cauldron if my server wasn't 100% whitelisted friends. The majority of users I've known for at least 3 years, so we allow things that bigger servers couldn't allow. No claims and such since everyone respects each other (I know it's rare, but it works for us since we're so small).

I'll take a screenshot of the Warmroast profile to get a baseline since I'm not experiencing issues at this time. (since I'm a new user on the forums I'll post the screenshot in the next post--links have to be approved by a moderator) Thank you again both of you. I'll keep this updated as things progress.

 

[UntoldForce]

As promised, here is the Warmroast profile screenshot (attached)

 

[Shybella]

Yes, Warmroast is a beautiful JVM profiling tool. If you drop --thread "Server thread" from the start script it'll show all threads besides your main Server Thread. This is very handy too.

@UntoldForce I'm overall more than certain this has to do with Mystcraft. As you're not using Cauldron and still having that issue shows that Forge Version incompatibility is the problem not Cauldron. Which seems to be the issue now that xCompWiz has suggested. You've proven this to be correct. I'm working on a fix for this now or in hopes xCompwiz just implements a simple Forge check and just implements the old IWorldGenerator class. There's a Github with Cauldron updated with 1388 Forge which contains the missing maps but you should have no issues just updating Forge with your current mods.

@Baron I'm hoping xCompWiz replies back with the issue reguarding Mystcraft.

 

[UntoldForce]

@Shybella thanks for the suggestion... believe it or not I asked all players to stop generating new Mystcraft ages since generating structures on the boundary would cause a server crash. It didn't seem to extend to already-generated worlds but that's a fantastic idea. I haven't been keeping up with the forums as much as I should have (damn you, work) but I'll try updating tonight. The profiles were taken without people jumping into Mystcraft ages. We do use RFtools quite a bit (it's delightfully broken for someone with a Draconic Evolution stabilized spawner and grinder) which doesn't seem to have any issues like Mystcraft with worldgen.

Noob question--is there any downside to using Cauldron instead of Forge? I know that I could more easily manage my server with the Bukkit plugins but I was always a bit leery of switching. I am sorry, I am not as knowledgeable with things as I should be.

 

[Shybella]

Cauldron works as a simple bridge between Forge and Spigot, Spigot implements the Bukkit API which allows for plugins. How Cauldron does this is by creating bindings. These bindings do nothing but allow interpretations to be passed without conflict between Spigot and Forge which both directly modify/interpret default NMS.

Theoretically, Cauldron implementing Spigot and other various patches that improve many default NMS and java classes without breaking those important functions while providing more functionality and performance enchantments. BloodMC's(MCPC/Cauldron) and MD_5(Spigot) improvements have indeed done what their suppose to do.

Now, one of the major issues is Forge and mod compatibility. I remember years back Linux Kernel releases had the same issue and still do but distributions like Debian have allowed for LTS(Long Term Support) for single(Stable) Kernels that are extended to improve compatibility. Forge along with Sponge will do this but that's still far away.

Cauldron is no longer supported publically by BloodMC. He still does release patches via one of his IRCs. You should check his Cauldron website for more information on that. Although, there are a couple projects running now that still update Cauldron. BloodMC has been focusing on Mixin for Sponge which is a Java to ASM Translator and he also works on Sponges core. I try to keep up to date on his work, he's the man.

Nearly every Vanilla server uses Spigot due to the performance enchantments and functionality. As they also update Bukkit, support for plugins like Essentials exist and for Cauldron and work the sameway they would on Vanilla.

In your case, setup a second testing server for Cauldron running Infinity 1.4.1. Learn how things work before implementing that live for your friends and players, it's not much different than standalone Forge. When you do set this up, remove FastCraft as Spigot implements the ASMTransformation for Chunk Generation that FastCraft does. It'll cause a conflict.

Let me know how things go.

Links:

Cauldron: http://cauldron.minecraftforge.net/
Sponge: https://spongepowered.org/
Spigot: http://www.spigotmc.org/

 

[Shybella]

Ok, after further investigation. It has to deal with Fastcraft making modification to a field for loaded chunk hashmaps that store chunk data for said loaded chunks.

So, this isn't a Forge problem. It's Mystcraft trying to improve performance when another mod has already made modifications. So, it's Fastcraft not having compatibility for Mystcraft.

 

[UntoldForce]

Would it be more prudent to disable Mystcraft or Fastcraft until the problem is patched? I'm guessing your earlier advice to upgrade Forge is no longer a good idea? Also thanks again for the knowledge @Shybella, you're like a veritable treasure trove of knowledge. I'd ask where you learned it all but a server admin has to keep their secrets!

 

[Shybella]

Yes, updating Forge wouldn't resolve the issue but It still would be a good idea given the recent Forge changes for fluid registry. Changelog: http://files.minecraftforge.net/mav...orge-1.7.10-10.13.3.1395-1710ls-changelog.txt

The generation changes with Fastcraft can be helpful. You have a couple choices. If you value Mystcraft. Here's a couple ways to improve performance for chunk generation and loading and unloading.

#Changing your view distance will lower the view distance which will lower the amount of chunks loaded around a player. This can be found in your server.properties
#I adjust this depending on my player count, but a good number to start with is 5.
- View-distance: 10

# In your ChickenChunk's config this will unload chunks that are not occupied by players. This will greatly reduce memory overhead and overall performance with tick entities.
#The number of ticks to wait between attempting to unload orphaned chunks
#1 second = 20 ticks
cleanuptime=1200

You can also check out some JVM flags to apply to your startup. Here's a startup script with the flags and examples.

-server -XX:+AlwaysPreTouch -XX:+UseStringDeduplication -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:""ParallelGCThreads=8 -XX:ConcGCThreads=2 ( Remove the "" before Parallel )

My example flag is overtop for small servers. You can knock those threads down. 2/2 should be well morre than enough.

• -XX:+UseG1GC: Use the Garbage First (G1) Collector.
• -XX:MaxGCPauseMillis: Sets a target for the maximum GC pause time. This is a soft goal, and the JVM will make its best effort to achieve it.
• -XX:""ParallelGCThreads: Sets the number of threads used during parallel phases of the garbage collectors. The default value varies with the platform on which the JVM is running. ( Remove the "" before Parallel )
• -XX:ConcGCThreads: Number of threads concurrent garbage collectors will use. The default value varies with the platform on which the JVM is running.
• -XX:+UseStringDeduplication: https://blog.codecentric.de/en/2014/08/string-deduplication-new-feature-java-8-update-20-2/
• -XX:+AlwaysPreTouch: Set the JVM to touch every page on initialization, thus getting all the page in memory before entering main() (this will increase your startup time). The AlwaysPreTouch flag is useful in testing when simulating a long running system which has all the virtual memory already mapped into physical memory.

Important

Make sure you're using the latest Java Update. G1GC and AlwaysPreTouch had a issue that caused very long GC pause times. It was a bug with AlwaysPreTouch which has been resolved. The update was released on 4/20/2015.

 

[UntoldForce]

Good find about the Forge update (I think I saw something on the forums about the problems with fluid registry). I did decrease the view distance to 7, which I thought should be more than appropriate for the small # of concurrent users (as I said, average is about 6 players). Do you decrease the chunk cleanup time significantly? Also I run my server instance with 4G of RAM... is that a decent number for the view distance and player count? I know the CPU speed is the most important factor for Minecraft servers but I also understand that memory is important, especially for heavily modded servers.

This is the startup string I run because someone told me that it improves garbage collection.

java -server -Xms512m -Xmx4G -XXermSize=256m -d64 -XX:+UseParNewGC -XX:+CMSIncrementalPacing -XX:+CMSClassUnloadingEnabled -XXarallelGCThreads=2 -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:-UseAdaptiveSizePolicy -jar FTBServer-1.7.10-1352.jar nogui

Will the additional GC flags cause any conflicts with the existing flags? Thanks in advance.

 

[Shybella]

-XX:+CMSIncrementalPacing does nothing but statistic logging. Isn't needed and extra processing.
-XX:+UseParNewGC is another type of Garbage Collector. UseParNewGC is typically used for applications that use < 1GB. G1GC handles large memory sizes with way greater optimized collecting times.
-XX:+CMSClassUnloadingEnabled this is a bad choice for modding applications such as Minecraft. Most of Minecraft classes are called at runtime so when a garbage collection happens they'll be wiped out.

-XX:MinHeapFreeRatio=5 - This goes by percent of your min -Xms or heap size. This will start collect when the met occupancy min free size is met.
-XX:MaxHeapFreeRatio=10 This goes by percent of your max -Xmx or heap size. This will start collect when the met occupancy max free size is met.

I wouldn't use none of the above as the defaults handle well for Minecraft.

Also, are you using Java 8?

 

[Shybella]

This is a very good guide. https://blog.codecentric.de/en/2012/07/useful-jvm-flags-part-4-heap-tuning/

 

[UntoldForce]

Running Java 8 update 45 (64 bit)... also have 32 bit installed (same version)

I switched my flags to the following and will check the results tomorrow:

java -server -Xms512m -Xmx4G -XXermSize=256m -d64 -XX:+AlwaysPreTouch -XX:+UseStringDeduplication -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XXarallelGCThreads=2 -XX:ConcGCThreads=2 -jar FTBServer-1.7.10-1352.jar nogui

I understand that PermSize is deprecated in jre8 so I'll remove that next time. On an earlier FTB server (a long time ago on a Java version far, far away) I ran into unknown server crashes when adding a new mod until I realized the PermSize was too low. Took a look at the article you posted about heap tuning. It's a shame I'm not a dev. Knowing the "proper" amount of memory to allocate to a Minecraft server seems to be more trial and error for admins than formulaic. I know for clients that sometimes too much RAM causes problems (just because one has 32 GB of RAM does not mean one should dedicated 24 GB to the Minecraft client!)

I didn't change from Forge to Cauldron tonight, though. Was far too busy with work. I'll have to look into that tomorrow. Still getting periodic "server can't keep up" warnings so I'll keep on it. Thanks again Shybella for your encyclopedic knowledge.

 

[Shybella]

Anytime, I'm glad I could help.

 

[ScottulusMaximus]

Having this again myself, previously it was an AE interface causing it.... Now I have no idea, removing Mystcraft doesn't help.

And it's not intermittent, it's constant every 30sec or so and skipping nearly 300 ticks a time.

 

[ScottulusMaximus]

Anyone got a step by step to get WarmRoast running, this "simple" tutorial on the page has left me clueless.

I paste this into CMD(the suggested command without the "." before com.sk89... leads it to not finding the class):

java -Djava.library.path=C:\ProgramFiles\Java\jdk1.7.0_79/jre/bin -cp C:\ProgramFiles\Java\jdk1.7.0_79/lib/tools.jar;warmroast-1.0.0-SNAPSHOT.jar.com.sk89q.warmroast.WarmRoast --thread "Server Thread"

And get this in return:

Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

XXXX>java -Djava.library.path=C:\ProgramFiles\Java\jdk1.7.0_79/j
re/bin -cp C:\ProgramFiles\Java\jdk1.7.0_79/lib/tools.jar;warmroast-1.0.0-SNAPSH
OT.jar.com.sk89q.warmroast.WarmRoast --thread "Server Thread"
Unrecognized option: --thread
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

 

[ScottulusMaximus]

Ok it's the hungry node I'm growing causing it this time, or something in it's feeding setup/chunkloading.

But seriously what is with this pack and skipping ticks without using all available resources? There must be an underlying factor somewhere that's letting 2 completely different mods do this:/