Are those quarts crystals generated in all biomes? I haven't had any issue in my current world, and i have mined out a fair share of those crystals.
only in one of maybe ten worlds generated have i had this lag issue.. (I've had some really bad luck with my spawns recently.. tonnes of ocean, endless thundra/alpine spawns and one really awful one in a wasteland that seemed to go on forever.
The spawn with the issue was quite similar to the one the mindcrackers have on their server. a large fairly flat savanna biome.
My current one had me in a hilly rainforest. I've set up base just outside the rainforest, in a green meadows biome. I sometimes get lagspikes when moving south, but I know that is an issue with the redwood forest over that way,(it's a very big one) and i intend to stay out of range of those giant trees most of the time. except for when i want to harvest some of them for their interesting log texture. (i really like the way you can have a wooden border that even turns corners.) the spikes from the forest are not even half as bad as those i had in the treeless savanna biome tho, so I know for sure that it has nothing to do with trees. Not even the giant Ygdrasil style oak trees give me much more than a slight hickup when they get loaded into the world, nothing more.
Going trough the mod's list there really aren't that many suspects. there is the Voxel stuff. music does cause lag when it loads, so disable that. it doesn't add anything of gameplay value, so disabling it is an easy decision.
there is xycraft, but I've seen tonnes of it's ores without lagging. but i tend to turn particles to low anyway. fire tends to lag me pretty hard even in vanilla. especially in the nether.
I have a strong suspicion that it might be thaumcraft making all that lag. it adds special spawners and stuff, and a lot of the items added uses special particle effects as well.
I'm beginning to think that there might be some kind of thaumcraft dungeon or something making all that lag.