I've heard a lot of RP2 and IC2. These are great mods that can work quite well as stand-alones, but there's one I haven't seen, and honestly, I'm a bit surprised...
Thermal Expansion.
Granted, getting yourself bootstrapped into automation may be a bit tricksy, since you'll need to hit the nether before you can get energy conduits, and there's no item pipes at all (and the tesseract is decidedly end-game content for a mostly-vanilla play), but it provides quite a bit of automation even without those pipes, thanks to how the machines interact with one another (with proper side configuration).
Steve's Carts is another 'can theoretically do it all' kind of mod, although it's fairly crippled in the automation department by a lack of pipes/tubes. But it can do anything from forestry-esque automated farms, auto-mining like a tunnel bore... quite a versatile little mod.
Computercraft is another one that can really explode if you are good with lua, being able to fully automate just about any project with turtles. Everything from branch mining to sorting systems to farming.
But if I really wanted a challenge? If I really wanted to use just a single mod and ignore everything else? TerraFirmaCraft. It beats BTW with an ugly stick, both in terms of complexity and in being internally consistent with the tech tree. It's explicitly not compatible with other mods, simply because it changes so many fundamental concepts of the game world that the other mods just don't have any points of commonality to interact with. Which is a much better reason than BTW's, in my personal opinion. But the mechanics it introduces are in-depth, flavorful, and quite interesting once you get the hang of them. Stone knapping, for example, becomes your stone-tier tool mechanic. Smithing requires more skill than throwing stuff on the crafting table. Heck, getting your ores requires a lot of effort. Making charcoal isn't easy either. It is a glorious and wonderful insane challenge which anyone who considers Minecraft in general 'too easy' or 'op' should check out.