It's the End that will take the most punishment but not a lot of people are going to be affected by that, so you should be OK.
I always thought if you have mod related items in chests etc. and you delete the mod, opening the chest will result in a crash. But I could be wrong.
Most mods will delete their blocks and items when you try to load them in - or, specifically, Minecraft will see an invalid block/item ID in the save data, assume it's corrupt, and replace it with a blank value (i.e. air).