OK, So it seems that Sphax texture pack was messing with it. I joined the server for like 30 seconds and then it crashed. I got back on removed the pack went into single player fine, went on the server and it has been fine. So maybe the server doesn't like Sphax? Or I might just be using too...