Thanks for everyone's feedback
I have removed Reis Minimap, everything voxel related, liteloader and basically anything that wasn't necessary to login to the server I play.
I cannot get away with removing thaumcraft because I am playing / experiencing these issues on a SMP server - therefore I wouldn't be able to join. I can tell you though that everything is 50x worse on SP - but SP has always been laggy for me. (at least until the world settles)
I have "made up" for the lag by lowering view distance and using optifine and tweaking some settings to get better fps. The fps definitely still dips when looking at these "spots", but since I have higher FPS overall, it's less noticeable.
I have zero idea what most of the mods do, but apparently there's some sort of invisible aura thing? That's interesting, and would make sense. I'll have to grab me some of those goggles or whatever to investigate.