Thats the only thing I can think of too, wisps do hurt pretty bad though, but inside they lose their advantage of range. Ive already ventured inside and torched it up. I usually play Hmode so its old hat.2 types? I can only think of the wisps, other then the spawners.
I've spent hours in those, never seen anything except zombies, creepers, skeletons and wisps. Creepers only when first approaching, once dead they don't tend to respawn that close if you're inside.