Do you have anything else running in the background? One common fault I'm finding with the current generation of PC users is that they have a ton of background programs open and that they love to alt-tab into other stuff. This is a BIG NO NO.
Doesn't matter how good your PC is, doing that sort of stuff will give your PC headaches, as games, etc, are not designed to be used in that way.
Ensure nothing else is running so that all your resources are available to your game.
I have a 2 core, 2.4hgz processor, 3 gig ram, 512 meg graphics card, use a 64x texture pack, view distance on Normal+16, and Fancy graphics plus Optifine. I'll get FPS bouncing around between 5 and 40 FPS, after a while, I start getting lag spikes, and sometimes graphic corruption (espeically GUI's) at that point I have to reload MC.
My personal belief is that some of these mods are not written as optimised as they could be (or an issue with how mods are loaded). After all, the Minecraft "mod" is 7 meg and contains a ton of stuff and the entire core system, and that loads very quickly and gives me 90+ FPS. Start chucking in a few mods, all of which are less then 2 meg each, and FPS drops by about 1-5 per mod and loading time (for me) can reach up to 2+ minutes.