Forgive my interruption, but some people seem to be confusing lag for framerate drops - the two are very much different, and as a server owner it bugs me to no end when people scream "LAG!!" whenever they get large FPS drops. What's considered "Lag" is caused by the server not being able to handle the load, or network issues (anywhere along the client<--->server route, more accurately described as latency.) This typically manifests itself in disconnections, rubberbanding (moving and snapping back to where you were before), empty chunks, or blocks breaking but not dropping, only to reappear. Framerate drops, when your screen moves slowly or you can barely move or issue commands, are much more common and usually caused by a lot of entities/particles/etc. onscreen or in the locally loaded chunks at any given time.
Case in point on my server: Rapid chunk generation causes absolutely no problems with clientside speed, however, after a short time the server struggles to keep up and you get the lag effects I mentioned above. On the other hand, my old RoC workshop had a huge mess of LogisticsPipes before I realized I could do point-to-point links with golden pipes. The server didn't care but my client FPS dropped drastically; this is framerate drop.
To answer OP: Very few mods are prone to server lag, but any can eventually cause it. Rotarycraft is frequently cited but the only blocks that seem to cause noticeable lag are solar tower arrays, while Twilight Forest is rarely cited for lag, but if enough mobs spawn it can notably reduce server performance (4,000+ wild deer, anyone?) Your best bet if you have a brand-new server is to leave default+desired mods enabled, run it for a while in survival with players in creative mode doing anything they can think of, and do frequent performance monitoring of the OS/server thread. That'll give you a better idea if you need to disable mods or can handle what your modpack of choice will throw at you.
As for griefing, almost any mod can be used for griefing purposes to some degree or another. BC quarries, Thaumcraft warded blocks, TE Pyrotheum, RoC borers, Stevecart drills, Bees with End effect, Mystcraft portals stripping gear, the list goes on. I can't tell you how to administer your server, but your best bet is to clearly communicate that griefing will not be tolerated, and come down hard on those who do (and you can prove it!) Since you've chosen to use bukkit/mcpc/whatever it's called nowadays, there's a large amount of monitoring plugins to choose from.