it's not the CPU. Same problem here.
Kinda looks like: if you have enough memory, the leak isn't noticeable (or takes sooo much longer to notice). I'm on 8 GB and it's pretty fast noticeable.
I can't pinpoint where it's at, I've updated several mods to try and fix it. It kinda looks like AE...