Instead of placing an ender chest/relay directly on your quarry, place a diamond pipe, connected on one side to your chest/relay, and on an other to a void pipe.
Put the diamond pipe to output what you want to void into the void pipe, and don't configure anything else.
I really can't understand when that setup would be a problem, it's easier to do it directly at the source, because it makes more room tor the rest of the stuff, and you don't need to stress your sorting machines anywhere as much.
If you still want to keep some of it, just put chests/barrels and extend the path to the void pipe, but gravel/sand/cobble/dirt can be automatically made from other sources anyway.