If you're checking by moving back towards the island, endermen will start spawning because you're closer. When you're sufficiently far away you'll just have to trust that they're not spawning on the island.I checked it, and on Normal render, the island completely disappears, and then I go another 50 blocksish and I go back and there are still baddies sitting there. The end portal is right by them too, would that load any chunks or anything?
EDIT; I start at 130, and the very end of the straight line out is 378, so I'm almost 250 blocks away.
At any rate, even if some did spawn on the island, your farm should still be very efficient, so I wouldn't worry about it much.