Yeah usually I find the cause for this is my golems wandering off to get an item droped from a random mob that died in the daylight and then falling down a hole or getting stuck.
the best way to fix this is to specify what items your golems can pick up so, eq. if it's working on a wheat farm...