[Script,Linux] Auto-Start FTB Server on Crash + Automatic Backups

[Eremite]

This is still a work in progress, but I figured I'd post it early so I could get input or see if something like it already exists:

https://github.com/Sindern/rtb

It's meant to be placed in the server's folder along with the jar file, then run instead of the normal command to launch the server. It should monitor the server and re-start it if it ever crashes. It will also make full backups of your server folder every 4 hours (by default) just in case something horrible happens to your map and you can't get it working again.

Check the readme on the repo and be sure to edit the config at the top of the script before running it.

Please let me know of any features/bugs you notice with it.

 

[Greedseed]

Cool The only problem i see with all of these scripts is that most of the time, or lets even say 99% of the time, the server will not crash but just lock up. A reboot would fix this issue but i dont see how that could be detected without reading down the info output from the console.

 

[Eremite]

Yeah, it's still a work in progress at the moment. So far I haven't been able to make the server crash except by making it run out of memory by setting a VPS to super-low RAM. I'm going to try some stuff to make it break some more... I'm thinking maybe have it try a telnet connection via the server port to see if it's able to connect, or maybe nmap. I'll figure that out once I can get my hands on a locked up server.

 

[Mercury]

I assume that this crash detection only works when FTB-Beta-A.jar (java) process is not working.

My FTB server crashes once per day with main/server thread running what means server is running partially. I can do save-all and stop commands but it will timeout all current players and won't accept new ones.

Description: Exception in server tick loop
 
java.lang.NullPointerException
    at ic2.advancedmachines.common.TileEntityAdvancedMachine.operate(TileEntityAdvancedMachine.java:210)
    at ic2.advancedmachines.common.TileEntityAdvancedMachine.g(TileEntityAdvancedMachine.java:96)
    at xe.h(World.java:2020)
    at ie.h(WorldServer.java:463)
    at net.minecraft.server.MinecraftServer.r(MinecraftServer.java:649)
    at hg.r(DedicatedServer.java:241)
    at net.minecraft.server.MinecraftServer.q(MinecraftServer.java:585)
    at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:492)
    at fp.run(SourceFile:818)

 

[Greedseed]

Yeah, it's still a work in progress at the moment. So far I haven't been able to make the server crash except by making it run out of memory by setting a VPS to super-low RAM. I'm going to try some stuff to make it break some more... I'm thinking maybe have it try a telnet connection via the server port to see if it's able to connect, or maybe nmap. I'll figure that out once I can get my hands on a locked up server.

Cool, i wil look more into this if i can drop you some code for CentOS. If i dont forget at least Got way to many projects i am working on xD.

 

[Dekar]

I have not had our server crash yet, but when checking in on the server, the console for MC basically does not respond to any commands. Doing a ctrl+c will stop the server and I can restart it. However, even in that state people can connect and play on the server.

Any thoughts to just have a daily restart for the MC server in these scripts? My thought was to have the server do a backup around 4am and after that just do a restart of the server.

 

[Eremite]

Yeah, currently, it only works if the process crashes completely and closes so that it's not in the list of running processes.

The daily restart thing might be a good thing to add. I think I found a nifty trick to test it though. If anyone has their server freeze up where the process is still running and they can't connect, try this command below and let me know the output. (Needs netcat from http://netcat.sourceforge.net/ or your distro's repo)

user@box:~/ftb$ nc -zv localhost 25565
localhost [127.0.0.1] 25565 (?) open
user@box:~/ftb$ nc -zv localhost 25565
localhost [127.0.0.1] 25565 (?) : Connection refused
If the port's got a service listening on it, it should say open like the top one. I tested it with the server off and I get Connection refused. Not sure if it'll reply with the same if it's open but locked up.

 

[glasspelican]

I forked the git-hub. I see something that might be an bug.
Wont know for sure until I test it next week after exams.

Once I have more time I would be glad to make a pull request with new features.

 

[Mercury]

Yeah, currently, it only works if the process crashes completely and closes so that it's not in the list of running processes.

The daily restart thing might be a good thing to add. I think I found a nifty trick to test it though. If anyone has their server freeze up where the process is still running and they can't connect, try this command below and let me know the output. (Needs netcat from http://netcat.sourceforge.net/ or your distro's repo)

user@box:~/ftb$ nc -zv localhost 25565
localhost [127.0.0.1] 25565 (?) open
user@box:~/ftb$ nc -zv localhost 25565
localhost [127.0.0.1] 25565 (?) : Connection refused
If the port's got a service listening on it, it should say open like the top one. I tested it with the server off and I get Connection refused. Not sure if it'll reply with the same if it's open but locked up.

It will stay open if server is locked / server thread running.

 

[Eremite]

Still fighting with this in my free time. :/ Netcat and nmap both seem to respond that the port is open even if the server is frozen.

Trying to test to see if there's anything special about how lsof displays the socket if the server's frozen. Here's what it looks like on a healthy server.

root ~/ftb # lsof -i | grep :25565
java 3835 root 45u IPv6 2850289046 0t0 TCP *:25565 (LISTEN)

-edit-
And when the server's not running:

root ~/ftb # lsof -i  | grep :25565
root ~/ftb #

Google's not finding much information on this. :/ I wish I knew more about Java to see how it's handling the connections, maybe.

 

[rmbell]

Just wrote a quick bash script that will auto-restart your server upon detected crash, that might help someone out, so I thought I would share it.

#!/bin/bash
### Requires inotifywait & tail
## Config
# Location of your /crash-reports/ folder
crashreps="/home/minecraft/ftb/crash-reports/"
# Location to store a detected crashes file
detectedcrashes="/home/minecraft/ftb/detected.crashes.txt"
# Your force-restart cmd
forcerestart="sudo service ftb force-restart"
## End of config
inotifywait -m -r --format '%f' -e modify -e create $crashreps >> $detectedcrashes
        if tail -n1 $detectedcrashes ; then
                $forcerestart
        fi

 

[Mercury]

^ It didn't come to my mind to check for crash-reports folder instead of querying server. Very nice

 

[Eremite]

Oh what a goose I am. This is proof that you can beat your head against the wall for hours trying to find a solution to something and have it be the simplest thing. mtime of the crash file... brilliant.

I'll try to incorporate that this weekend. Thanks for showing me the light.

-edit-
https://github.com/rvoicilas/inotify-tools/wiki
Posting link to inotify-tools for when I get home to mess with it.

 

[ncreen_same]

Perhaps to see if the server is frozen you could use the Query port (obviously this would need to be configured and turned on)

 

[glasspelican]

The script works too well.
I cant get the server to shut-down.

 

[Miamore]

The script works too well.
I cant get the server to shut-down.

If you're running the server in a seperate screen (you totally should) then just save the world and close the screen (ctrl +A, then type = ":quit")

 

[mooner93]

Hi, I never got around thanking you for this post. I have been using this script on my server for a while now and I like it a lot thank you very much and keep up the good work.

 

[Lathanael]

I have been using this one for a time now and really like it

http://www.minecraftforum.net/topic/121830-multi-world-linux-minecraft-server-control-script/

 

[Eremite]

Just updated it and ran it through a few simple tests. It *should* catch freeze-up crashes too now, thanks to rmbell's suggestion to use inotifywait.

Also, it will run the server in screen now, so you can just attach/detach to/from the screen and still be able to input console commands even after a crash. Plus the output of the backup/crash logging looks prettier without the server text.

Needs to have the screen and inotify tools installed.

Hopefully I didn't add any bugs while I was at it.

-edit-
Ugh, I herped a derp. I just realized that the way I have backups added by default will create recursive backups. Yo dawg, I herd you like backups, so it'll put backups in your backups so you can have backups of your backups of your backups ....
Easy fix is to just set that to something outside the server folder.

-edit2-
I double derp'd. When I changed the backups, it messed up the backup checks so it was backing up continuously. Should be fixed now, but if you ran it between Friday and now, (sunday 11PM Central), my apologies.

 

[gustav9797]

rtb.sh: line 4: syntax error near unexpected token `newline'
rtb.sh: line 4: `<!DOCTYPE html>'

Missing something?
Ubuntu 12.04 x64