In Forestry 2.3, 5% is tough. Even with 3x Mutators w/uranium and 6x Soul Frames, I might get batch after batch of nothing but hybrids of the original species and not a single pure or hybrid with the new.. I just keep at it until I get enough of the new species. Ideally, I'll wait for a pure princess and drone, but at low % mutation rates, I'll try with 1 pure and a half, i.e. 3 out of 4 active/inactive with the new species.
I also will Isolator the halves and sometimes get the new species via serum.
Radioactive and Quantum each took several hours of mostly shuffling bees and serum vials between machines and the alveary.