I have found one of my biggest problems after I got garbage collection sorted a bit smoother was CPU heat. Granted, I play on an optimized potato, but the game would start off fine, then chug as my CPU fought for cool air. A good blow out and clean up along with replacing a busted fan has helped It would also make sense that letting it sit and idle would allow it to cool off again. Maybe I'm barking up the wrong tree, but it's something else to consider if nothing else has worked thus far.
I had a similar situation that your post brought back to mind, except mine was a failed cooling fan on a video card. As the card got warmer things became slower.