Just to note, that's not how ram actually works. Ideally you would be using 100% of your ram on load to be fully efficent, it's how your ram handles switching between programs and what it's called to do that matters (swap and the like) when it comes to performance. Also, Java is notoriously horrible with trash collection depending on what you are doing and what version you have (it's come a long way), so the more ram you have, the more trash there is to clean up, therefore more potentials for latency in performance. More ram isn't always the answer.
I'd recommend reading up on how ram works, and what java does to affect it. Also keep in mind that on the backend, how cores are used can make a difference. You can assign Java to utilize cores other than the one minecraft is running on for cleanup tasks. You can also have your server dedicate a core to minecraft if you like (IE, I'm running a dual hex core on Debian).
As for mods, as a server admin there are a couple required mods I run on every pack I used, including the pack I put maintain myself. Forge Essentials, Opis, Fastcraft. Really that's it lol. There have been several other mod's I've played with, but it always goes back to those. Forge Essentials is amazing for granular control over permissions and abilities of players, groups, and zones/dims. Opis is my favourite diagnostic tool thus far for assessing and finding where issues reside on the server causing lag (the thing can go all the way down to highlighting blocks for you). And fastcraft... well I honestly don't understand how the hell it works, but it does. Without fastcraft I've had issues with terrain generation and certain machines with "special" animations and particles causing tps and block lag, fast craft has greatly improved performance in those situations. It hasn't eliminated it by any means, but improved yes.
To give you an idea of what my server runs on:
OS: Debian 7 (I like the simplicity and customization of linux and the control I can have using CLI, really that's all I can say)
Proc: Dual Hex Core AMDs (12 physical cores, not a biggie for minecraft as it uses one core, but nice for daily routines and tasks I have going on on the backend and also for leveraging some of javas newer trash handling using multiple threads on different cores)
RAM: 32Gig on the node, 8 dedicated to production server, I use 4 on test and development servers.
Network: 1Tb thru with 20Tb monthly bandwidth allotment.
I am by no means any sort of expert on anything, these are just my opinions. There are some really smart and savvy people on here, so as they reply you will get an even better bigger picture, and I'll learn some things as well I'm sure
