My problem is not with difficultly level but with the speed in which I get to endgame content. I have conversations with people on my server all the time about how they don't know what do next... I made it a point to slow way down, not use quarries, try to focus on one mod at a time, not use solars for power, and not wear quantum or power suits. Personally, the game has gotten so much better. Sure I still collect stupid amounts of resources but most of the time it is things I will never use. I am not interested in the most efficient builds anymore but focus on the builds I think are the most fun. Again, as angelnc mentioned, its all about setting goals and limitations for yourself.
TerraFirmaCraft sounds like right up your alley. It really throttles the speed at which you can get to endgame content, it makes said endgame content entertaining and challenging, even when you have end-tier tools and materials, and it has a lot of verisimilitude and internal consistency built in.
Other than that, here's some challenges to try out for yourself:
1) Plain Jane Tools. Never augment, enchant, or otherwise enhance your tools. This includes upgrades and TiCo modifiers. If you can build it, you can use it. If it requires another tool to build, no-go.
2) Home Owner's Association Member. Your home must be 'up to code'. This includes using proper building material (such as stone brick blocks instead of cobblestone), All piping and wiring must be 'up to code' and not visible. Must have a neatly trimmed and manicured lawn (i.e. no creeper or 'civilization' holes) which is not cluttered with excessive things (such as torches, basically if you can hide it aesthetically, you can have it). Must have a proper roof with overhang and gutters. Must be decorated tastefully. Must have clearly designated rooms for your industrial machines, power generation, storage and autocrafting, living, dining, restroom facilities, and 'greenhouse' rooms which are separate. Must not have a mining operation entrance in your actual household, it has to be in a seperate facility, which must be in an enclosed building and otherwise comply with all building code and regulations. Other restrictions may apply depending on how restrictive you feel you want your HOA to be.
3) No powered equipment. We don't need no steenkin' power tools or powered armor 'round here. This pick was good enough for my grandfather back in the days of Beta, and it's good enough for me!
4) Never Ending Story. Do not complete any End Portals.
5) Whether wilt thou wither? You are prohibited from building an arena are for fighting the wither. Nor may you fight the Wither from within your home or on any property you have claimed as yours, they are against your city's pet ordinance. You may take advantage of naturally spawned terrain (nether fortress, village, abandoned mine, or dungeon) as long as you have not done any significant improvements to the property. You most certainly are prohibited from using an MFFS system to contain the wither to fight in. Furthermore, you MUST fight the wither every time you collect three wither skulls. You may never have more than two wither skulls in storage at any given time, unless you have stored them at base and are returning to collect them to build the wither.
6) Forcefields are for wimps! Never use any item from the MFFS mod.
7) E does not equal MC squared! Do not use anything from Applied Energistics. Fortunately, Iron Chests mod is still permitted.
