You have to download and use the map. You may want to remove the floating candles that are not over the infusion altar, as doing so upped my framerate significantly.
(It was, in fact, at least in part the candles. It was also partially Journeymap, and the paintings and customNPCs are probably...