Is this the invasion mod you were talking about?
http://www.minecraftforum.net/forums/mapping-and-modding/minecraft-mods/1278438-1-6-2-forge-invasion-mod-v0-12-0-so-you-think
Special Mobs doesn't affect the rate of mob spawning as far as I know; it does however add some super hard and annoying...