There is a little cariminite available in the top 2-3 floors of the tower; however, it can also be crafted from towerwood and borer essence, if I remember correctly. While the towerwood is very difficult to harvest, it's not impossible to do and it will drop itself; borer essence can be obtained by breaking infested towerwood and releasing the silverfish-like mob inside.Ohh ... I like that. It also means you have to actually go inside the tower, and not just derp-pile up.
Although ... I'll have to check where you get it. I thought it was found all over the tower, or at least past the first locked/4-key door.
I don't remember the recipe off the top of my head, but it is shown in item frames at the top of the tower (which, incidentally, are the only source of cariminite and its crafting components other than griefing the tower itself.)