With modded Minecraft, here's something that really grinds my gears and you should avoid at all costs. Don't ban practically every block and item that could maybe, possibly be used for griefing. It's a huge turn off, and why I personally avoid public servers. Also, no stores where you can buy in game items for real world money(using in game money is okay, as long as all the items on the store are craftable.) Put a player cap! Another huge issue with public modded servers is lag, because they have more players on then they can handle. Set limits for how many chunks each player can force load(this can be done through the Forge config.) Restrict the creation of mystcraft/RFTools dimensions. Use OPIS regularly to find chunks/blocks that are causing problems. Make judicious use of anti-griefing plugins/mods. Make sure either you or at least one staff member is on at all peak hours(and if possible, non-peak too.)
That would be all the functional stuff, to make sure everything runs smoothly. Now I'll bring up some stuff that is more of my opinion.
Avoid banning items. Hold the players responsible for using them the wrong way instead. Don't be afraid to punish players!
Make your rules clear and available(both at server spawn, and through a simple command such as /rules). Sometimes, a player may be breaking a rule and won't even know it!
If a player is causing lag, don't punish them! Work with them to try to figure out a better method of doing what they are doing.
Having a well made server spawn that makes use of the various mods is fantastic. But don't make it too big and fancy that a player can't find their way out! Consider using warp signs/commands to teleport them into the wild.
Ban griefers without mercy. Every single time I've tried giving a griefer a second chance, they did it again.
DO NOT BAN ITEMS FOR BEING "TOO OP"!! Yes, you are the server admin. But your job is not to dictate how your players play. Your job is to make sure they enjoy your server and to keep it running smoothly. If they want to use an item that you consider OP, that's their choice.
If you want to allow PVP, consider using a PVP toggle command, so players can choose whether or not they want to participate in PVP.