Yeah, but it's the contrast that makes it. No, it doesn't fit aesthetically, but that's the point. True, sandstone looks all nice and everything, but then you spend forty-five minutes trying to find it after you've gone out looking for resources. So my desert houses now have that nice smooth sandstone base, with brick block walls and wood log pillars on the corner.
As a forester's home? No. As the home of a stone mason who uses charcoal to heat his home? Absolutely. Or just a charcoal maker who's successful enough to have had a house built out of stone.
Of course just changing it up so the bricks and wood logs alternate back and forth would have solved the problem instantly.