If you want to go grab Logistics Pipes, the Quicksort Module and Quicksort Mk II module would be about perfect for your needs, AND is your sorting system all in one. Pulls out stacks at a time, so it could easily keep up. The Mk. II version pulls them out faster.
I have an initial diamond pipe coming out of my quarry to split off ores to be refined from the rest of the stuff. The enderchest the ores side goes to is connected to one directly on top of my Pulverizer, which will automatically pull stuff out. It then gets processed by the pulverizer, induction smelter, and powered furnace (for the bonus ores) and outputs into the pipe that leads straight to my metals chest. Cobble, dirt, sand, and gravel all get routed to my recyclers. The rest get sorted appropriately.
Puts a lot less stress on the system that way, since the vast majority of the quarry's output (cobble, dirt, sand, and gravel) don't even hit the sorting system. They go directly to recycling.
Yes, sorting to recycler at the source is one way, just not the way I was attempting to figure it out. With RP this isn't an issue... I'll try tubestuff buffer before logipipes.