I haven't played Monster but it has a large amount of mods. Yogscraft complete requires about 18-20GB of RAM for 128x textures. It has over 250 mods. Your RAM usage is correct, Minecraft was never designed for this. I thought my 16GB was overkill but I had to upgrade for that pack.
While, you don't need to allocate that much RAM, it'll need that much to cache textures or Minecraft will crash. You really should never need to allocate more than 5GB to minecraft in the launcher, if it's still crashing at that point you likely just need more physical RAM. Check your RAM commit in the task manager, it's likely peaking.
The reason why you were likely fine before is because that's how it works with new worlds. It always gets harder and harder to run, and RAM requirements skyrocket as you start placing down more and more unique blocks. 8GB of physical RAM just isn't enough for such a pack + textures.
