Out of ideas, the lag just won't stop.

Ok so a bit of history...

I still use the same computer today as I did about 5 years ago, technically almost, it's been upgraded. Back then I played FTB Ultimate and had a local server with 5 friends playing, and I also had a Stargate Mod added manually. To set up the server back then was fairly simple and I managed to get things working smoothly.
Today I'm trying out FTB Revelation and things don't go so smooth as they used to...

The Issue:
Today I'm constantly getting freezes, periodically for seconds, stuttering and low FPS. And that's even when I'm alone on the server.
((edit This problem is only occurring to me, client-side... Nobody else on my server gets any freezes but they do experience ping spikes during my freezes.)

What Has Changed:
The only change is "some" hardware and ofc software...
FX6350 (same as 5 years ago)
16GB of DDR3 RAM (5 years ago I had 8GB)
Radeon RX570 (5 years ago I had HD7870)
Samsung 840 EVO 250GB (same as 5 years ago)
1080p 60Hz screen (same as 5 years ago)

The Setup:
I've set the CPU affinity 2/4 (6-core) so the server has 2 cores and the client has 4, and everything else is shared across all cores. When I have these massive spikes I see no problems happening on the 2 cores my server has, it stays cool and doesn't fluctuate in performance unless someone moves very quickly or switches dimensions...
The 4 cores the client has is somewhat unpredictable. Just idling 3 of the 4 cores fluctuates between 30-70% load up and down all the time, the 4th core stays somewhat idle. But when I move, build, looking at recipes or fighting monsters all cores are working between 40-70% load.
But, when my spikes of lags and freezes occurs the load reaches up to 90-100% but only on 2 or 3 cores, and the period of lag/freeze does not match with the CPU load... The graph is showing me a much shorter load-duration than what I experience in the game.
Furthermore about the 2 server cores - in the console, I sometimes get the message that the server can't keep up, I don't understand why because the 2 server cores never tops above 40% load, and nothing odd is happening in the game when that message pops up.
I do get a lot of "Waiting for Server" messages in the game when the lag occurs, but Task Manager's performance monitoring doesn't show any hickups.
I also set the affixes in the FTB Launcher, properly I think, I have dedicated 4GB to the client. Previously I just set it to 6GB with the slider without affixes but with the affixes of 4GB things just got a "little" better... I can't remember the lines but if someone asks I could look it up (currently playing).

Questions:
If my game suffers from bad performance hardware shouldn't CPU load reach 100% on all cores at least when I get these lags/freezes? All I ever get is a tiny tiny micro-spike in CPU performance and this causes 10's of seconds of freeze lag in the game.
Why do I get "Waiting for Server" flashing in the game when the server barely process anything, does the server need more RAM allocated? I feel that the FTB team should've put up the proper affixes in their server files for this specific mod pack...


Ty for any help