Uhm... yes golems don't really like advanced pathing. They like to go from point a to b and not through stairs c. Point b may be a lot higher, as long as it's basicly a straight line.
Thing is though, the golem has to go up essentially the same kind of stairs to get to the crops above as it does to get to the crops below, so there's no actual straight line involved. It has to go to the stairs, go up around a corner or so, go to the crop on that floor, and go back the way it came to its home block, so it should be able to go down, too.
Also, the golem has been harvesting the problem floor now that I put him right on it, so I know it's not the plant that's the issue.