I'd personally like a mod that makes monsters smarter and better at killing you. One where mobs will actively try to enter your castles to try to kill you.
Creepers could actively try to blow up holes to get to you. I hate how you can just dig a 3 block hole, place a block above you and stay safe all night. I want creepers to really be dangerous and making huge forts actually useful other that "it looks cool."
Mobs can jump on a spider's back to climb up your castle walls. That way 3 block walls that are 20 blocks thick won't stop the creeper menace. Of course, spiders will no longer take fall damage.
Skeletons will gain a new tool: the zipline arrow. Skeletons will actively travel to high ground and shoot ziplines over your walls for mobs to glide down their paths.
Zombies will tunnel under your walls to try and get you from underneath. Or if your walls are thin, dig through them. They will mine at 2/3 of a player's speed and have the equivalent of iron mining speed by default. However, they mine obsidian at diamond mining speed so that obsidian doesn't become overpowered.
Overall, the basic idea would to make monsters actually pose a threat after you have made a house. However changes are configurable and there is an option to disable all changes after the enderdragon is killed (that way there's an extra incentive to do so)
I don't really have too good an idea of how memory intensive this would be for a computer, but I want to get this idea out there nevertheless.
Since you posted elsewhere asking for analysis on this, here, have some. I should start by saying my computer isn't very powerful, and I don't want nastier mobs that do this sort of stuff in my game. But not all people are like that, which is why modpacks like Blood'n'Bones exist. So.
I hate creepers. I hate how they ruin the landscape, and I hate how they ruin my stuff when they explode. Creepers are the most iconic, and most belovedly-hated mobs in the game. Minecraft would be very different without them. Creepers that can know where you are without seeing you, and actively hunt you down by exploding things is a bad idea. People get very attached to their creative work, and people want to be able to build a shelter without being afraid it's going to be taken away just because they were inside it at the time. Not to mention the trouble of building a shelter in five to ten minutes (your first day's work) that
isn't a handful of dirt blocks or a hole in the ground. And then there are caves, where it's dark until you light them up... Nasty.
That said, having mobs hunt you by more than sight is not too bad an idea. I've seen mods where if you get hurt, your character starts bleeding and leaving a trail of blood that mobs can follow. Why not add scent and/or hearing to the list of senses mobs have? Then you'd have a reason to sneak when near a monster, and if you leave a trail of blood they might follow you to your home. And then there are windows. Why would a mob not see through a window? Suddenly Direwolf20 never adding windows to his walls makes sense.
Blocking spiders is easy: have an overhang on your defensive wall, or eaves on your house. Spiders can't climb past overhangs. I think mobs deliberately riding spiders would be a fun improvement to the vanilla game, but not all spiders would want to be ridden. Also, maybe spiders could have a feather falling effect, rather than eliminating fall damage entirely.
Giving some zombies picks to go with those occasional shovels might be nice. Then you could have zombies with tools try to use them to get into your home. Not all behaviors need be cooperative, zombies swarm anyway. Those tools are iron, so iron mining level and 2/3 player speed (because rotten muscles) make sense. Giving a few zombies diamond picks would let them mine through obsidian, and little or no need to nerf the speed.
Skeletons with ziplines are silly when your home has a roof, especially a roof with a water trap on it (just put a water bucket in a dispenser, but on a square roof it should wash those mobs off nicely). However, what if fire arrows had a chance of lighting a fire where they land? What about arrows that do explosive damage? Giving skeletons a higher chance of having dangerous bows would bring the danger level up when you fight them, particularly in grasslands and forest biomes where burnable things are at hand (nearly every player will be building with stone and obsidian anyway with super-hardcore mobs like this). Also, farms with wooden fences become an issue. That said, skeletons with ziplines is still a way for mobs to get inside your protective walls. Could the player make ziplines as well, then?
Honestly, instead of mobs suddenly getting easier after you kill the dragon, why not have the dragon egg calm/confuse them or prevent spawns in a moderate radius (maybe it changes most mob spawns to endermen)? And/or maybe a beacon (which requires a nether star) has a new option to stop mob spawns nearby. Since the AoE of a beacon's effect is determined by the size of its base, suddenly you have a mechanic which requires danger, dedication, and lots of resource gathering.
All that aside, ultimately, your idea boils down to having mobs know where you are at all times so that survival is desperate and frantic the instant you spawn, which is a way to discourage players who aren't prepared for that kind of experience. Instead, I'd like to suggest the ability for mobs to learn where you are over time. Already in the game is that the longer you spend in a chunk, the more mobs spawn and the tougher they get. So let's add that the more often you leave blood trails, the more likely they focus on the paths you use around your home. Eventually, every night will leave dozens of mobs gathered around your doors, trying to get through. If you use a piston door or a drawbridge to put tougher blocks in their way, they'll try to dig their way through because that's where the blood trail ends. Secret entrances will get discovered, and eventually you have to fly in or teleport through if you don't want to have your home broken into and invaded.
And that's why you have to kill the ender dragon and the wither.