Well, you are right about that. If there is an open sapling/seed slot in the arboretum/farm, random junk can get in there. Seems like it wasn't always that way, but I dunno.
Anyway, your farms/arboretums will never use more seeds/saplings than they create, so it's not a problem as long as...