Speaking from experience, prison servers are not hard on hardware when setup properly. In fact, because of the limited exploration area performance is actually decent. The main problem with combining them with FTB is that there are so many mods that there are going to be loopholes that require entirely custom plugins and decompiling closed source mods to fix.