Some things I would like to point out here:
* The materials which are most useful will depend on what other mods you have installed. For example, Signalum and Enderium are only available with ExtraTiC. Pink Slime is from Minefactory Reloaded. Without these mods, these options will not be available.
* Signalum is far and away the best for bow/crossbow limbs, if available. If not, Pink Slime is superior to any 'vanilla' option. Otherwise, I'd suggest regular slime for speed.
* There is no reason to not Fiery Bowstring. Sure, you can use enchanted cloth, but it is effectively valueless because you don't need many modifiers on your Bow, and it decreases arrow speed, which cannot be effectively compensated for. Use a paper tough binding and thaumium stock on your crossbow for a total of 5 modifiers anyway.
* Crossbow Bolt weight actually aids in armor penetration, so having light bolts might assist them in traveling further, but heavy bolts will be more useful against armored opponents, particularly if you have Mekanism installed and they're wearing things like Glowstone armor. Therefore, thaumium rod is actually useful for both the modifier and the extra weight, when compared to pink slime. Arrows, on the other hand, don't have this advantage, so lighter arrows are better.
* Technically, blue slime leaf fletching has Jagged whereas plain ol' leaf fletching has Stonebound, which means the former SHOULD do more damage when the quiver is low. However, neither of these modifiers appear to impact damage output, and leaf fletching has over twice the durability, giving you far more shots per repair.
* Bolts and Arrows only start off with two modifiers, making damage stacking somewhat more difficult without a significant investment. Mind you, it is actually worth the investment now that the stack CAN be repaired instead of having to be replaced like they used to require, but it is still an investment.
* Damage modifiers got nerfed hardcore, they're almost not worth bothering with anymore. So if you are going to use modifiers, use ones that carry utility. For example, knockback to keep opponents back. Remember in the original Diablo, how a bow with the 'Bear' prefix was so godlike because opponents never got to you? Yea, it's kinda like that. Go ahead and toss Fortune on it as well for extra drops.
My new crossbow looks something like this:
Signalum crossbow limb, fiery bowstring, paper tough binding, thaumium stock. Dishes out the damage, and rate of fire has already been maxed out with no modifiers. From there, just toss on Fluxed, and you are good to go. It's just a shame that Fluxed won't affix to ammo.
To answer the question in the OP: there is no advantage to high draw speed, however it is typically paired with a high arrow speed, which is what you are looking for. With ExtraTiC, however, you don't need to make that sacrifice.