The RAM allocation you set in the launcher is not, I repeat, NOT the RAM allocation to Minecraft as a whole. It's the allocation of heap space, which is one out of 4-5 different memory spaces that the JVM uses to work with internally. And then there's the amount of memory the JVM uses to run itself, which is another matter entirely.
Your Windows task manager sums up all these different memory pools into one big number. So when you notice that it shows you as using 2 GB of memory in the task manager, that's a combination of heap space and eden space and young generation and permanent generation and JVM memory and god knows what else. Heap space is probably around a quarter of that total amount... maybe a third at most, unless your force Java to assign a giant heap. In that case you game will assign far more memory than it can ever use, and it will also slow itself down with trying to manage that huge space.
Most of the time you run well with 1 GB heap space assigned. If you encounter issues, up the slider to 1.5 GB. Under no circumstance should you ever put it above 2 GB because increasing the slider above what you need just lowers performance and there is no known configuration of mods that actually requires that much heap space. 99 out of 100 "out of memory" errors that modded Minecraft throws are actually the result of bugs that would happen no matter what you have assigned, or permgen space related, which is an entirely different memory space and not affected by the slider in the launcher at all.