Has to be square/rectangular, continuous(no gaps), then break one of the fences and put the quarry there and right click.
It could also be the Extra Trees fences messing up, I know I had a problem with them in Infinity. Maybe cheat in some vanilla fences and try?