Unless you wilfully avoiding it, using the stone will accelerate you. Honestly the /obsidian/, /pearls/ /iorn/ and /gold/ recipies make the most difference, not the diamonds. 4 gold becomes a lot of gold, iorn and obsidion. These become early enchantment tables, early chunk loaders, early TE...