The Deadly World mod makes enemies break torches, glowstone, and other light sources so more of their brethren can spawn. It also adds lava veins and silverfish veins throughout the underground, as well as other trap spawners which makes exploration dangerous.
The Special Mobs mod adds creative variants of all the mobs that add flavor without creating too extreme a power distribution. A few examples are dirt creepers that explode into a ball of dirt, jumping zombies that just want a hug, and mother spiders that give birth to a small swarm of baby spiders when they die.
However, if you do want overpowered mobs, give Infernal Mobs a try, it makes random diablo-style rare mini-bosses. They gain extra abilities, like causing wither, shoot fireballs, extra toughness, and can sometimes have more HP than the Wither or Ender Dragon.