I've never seen a way to track RAM Usage per mod, but there are multiple ways to profile mods to check their tick performance and CPU Usage.
TickThreading by nallar has some awesome commands such as /profile and /dump to check how certain blocks and mods are performing. It also makes Minecraft Mutlithreaded, which is excellent for FTB. Worth having.
MCPC+ comes with /timings, which will print some information that you can paste into this website to get information on what plugins are doing. You have to enable this in bukkit.yml
WarmRoast by sk89q is a web-based CPU Sampler that you can use to get even more specific readings about plugins/mods using the CPU. Not only can you see the mod, but you can look into which parts of it.
Alternatively to WarmRoast, you can get a JVM Snapshot as described
here.
Let me know if you need help with any of that.