[Linux] Watchdog script, auto crash recovery

[lukasni]

Hey all

I have recently been having problems with my horizons server freezing up, still trying to track down the cause, but that's not the issue here. Due to those crashes, I decided it would be nice to set up some kind of auto-recovery script that will detect if the client is crashed and restart it if need be.

Now, if the actual java process has died that is pretty simple, just check if the pid is still active and restart if not. However, most of the crashes I have been getting were actually not crashing the process, the process was just hanging and bogging up the CPU of my server.

I'm thinking of maybe checking the timestamp of the last log message, that should never be more than 5 minutes due to autosaving, but that seems a bit clunky.

Does anyone have a good way of dealing with the problem?

 

[Francis Baster]

I have a python script which repeatedly asks the server for its player count and message of the day, and if the server is unable to respond to 15 consecutive requests a restart command is sent to my server wrapper (remote toolkit). I have found that this is the most reliable means of crash-detection. My script simply stuffs the .forcerestart command into the screen session which contains my server, causing Minecraft Remote Toolkit to kill the java process and then start a new one. If you do not use remote toolkit then you will need to modify the restart command that is to be executed by the shell. This should be easy enough to do, just execute pkill to identify and kill the server process, or perhaps just use the screen command to kill the appropriate screen session, then issue a command to start a new server process inside a screen session.

http://paste.ubuntu.com/7259805/

 

[Hippieshaman]

inotify can be set up to sense if a new crash log is created in the crash logs folder, then execute a script when such event happens. I don't have any experience with inotify yet as I too are looking to set up the exact same thing. You could look here though, as there seems to be a lot of ways to do this: http://superuser.com/questions/181517/how-to-execute-a-command-whenever-a-file-changes