It should go without saying to turn down many of the Optifine settings. If you are on a sluggish computer, it can be painfully slow to cycle through some of them (like mipmap). Fast graphics, Open GL, and fog are mandatory on many machines, as is normal or less view.
But this isn't vanilla Minecraft and future MC versions will hopefully improve performance - of vanilla Minecraft. Keep in mind that the many, many combined mods authored by people of various skill levels can create some rather complicated code, creating all sorts of stress on your machine.
As stated, Xy crystals can be a big culprit. Maps also cause your system to need to do more. Perhaps Rei's has a smaller footprint than Voxel? I know Rei's allows for multi-threading. Btw, set mutl-thread chunk loading in OptiFine but make sure to turn OFF the graphics card's control of that. (Google java 3D threading I think.) If you don't need the Voxel features, perhaps get rid of all of those mods or start with the DW20 pack.
Make sure you allocate enough memory to java - you may have to tweak the perm space too because again, with such a huge variety of mods, there are tons of classes and whatnot. I really can't find any good information on how much is really needed - it is not as simple as "never too much".
The big problem is that once you load your GPU and your CPU, with a modded Minecraft, then playing Minecraft only makes it worse. Once things begin to flow thorough your automation system and mobs start gathering in your traps and pens, the load grows and grows. It is why I gave up on modded MC (and MC in general) when it was still in beta - and back then we didn't have any fancy loaders or packs.
So, besides Xy, one thing that might help is to take out more mods and use less of the features. And, if you use the Extra Biomes (or some other biome mod), disable any terrain that generates huge trees. I've read that the marsh can also be laggy. (I'm still experimenting with BoP myself, and have a much better computer than I used to.)
