The punishment is that you are getting less value for your time, i.e. your ores. That, for some people, me included, leads to the described bad feeling. As I said, the game is not forcing you to double ores, but it is rewarding that behavior with increased efficiency. Which is, in my opinion...