Weird Lag with Mystcraft / RFTools Dimensions

[Flexico]

I've googled this many times and haven't found anyone else who has the same problem as me, so I figured I'd make my own page about it. I demonstrated what happens in this video:

I suspect there's some other mod conflicting with something, but I have no idea what.

 

[Golrith]

If it's any consolation, I get the same thing in my modpack in the overworld. While I have heavy custom world gen, it's very random. Modded MC at it's finest!

 

[Flexico]

But I see videos of people playing it ALL THE TIME and it's fine for them! I don't get it!