It could be done like in forestry, where the fancier wood types are all gated by having to go through the genetics game. No boss fights, and still, you have to put time and effort into getting to them.
I know not everyone likes the genetics game, but that is beside the point. The point is, boss fights are not the only way to accomplish what you want.
Speaking of things that good old Forestry does better....
One of the things that irks me about modded MC is pointless tiering.
This is where you use a tool or block, for the sole purpose of being able to get a more powerful version of that tool, repeatedly.
The worst example of this, is
fortunately not in MC at all, but it is the 2D game Terraria. That game has over two dozen of tiers of pick-axe, where each tier is strictly superior than the last one with no downsides. Basically, each tier is a bootstrap-tier.
Unfortunately, a lot of minecraft mod makers seem to be under the very wrong impression that making everything in tiers is the way to make a great game.
For example; Lets compare the IC2 jetpack with the simply-jetpacks mod.
There is only one jetpack in IC2*, it is fast enough to be comfortably used in all stages of the game's progress and is balanced by the fact that the player needs to be careful how, when and how often they use it.
Simply jetpacks has 4 jetpacks, ordered in tiers, and each higher tier is more expense to make and superior in every other way than the last. Once you have a higher tier, the lower tier one is basically junk that waste your computer memory and icon-space in NEI/JEI just by being registered at all.
The lower tier ones are deliberately crippled in speed just to make the mid tiers more useful in comparison. The higher tier ones are effectively OP to justify their end-game-ness and super high cost.
Even vanilla minecraft isn't that bad (although by no means perfect) The only real bootstrap tier is wood, and that is unavoidable because you need it to mine your first stone. And while most people would probably want to upgrade to a diamond pick-axe quickly, there are cases where you would want to revert back to a stone pick. For example, in the nether, where the speed of a diamond pick can be dangerous.
I wish modders would learn from the way forestry does tiers. The Bee-house, apiary, and alveary, are all uniquely useful in their own way, such that even if you have the higher tier, there are still uses for the lower tiers. There is a sense of progress, but never a sense of being restricted until you get to the end. And no useless items you would only use until you get something better. You can even get the workhorse-tier right away if you trade with the villager, and yet, this is not OP.
One could argue, that I'm getting tired, or tiers...
notes:
*(not counting the fuel based jetpack vs electric, but those aren't tiers)