And you've hit the only concern I have about nuclear power...
As seen currently in Iran and N. Korea, they are using nuclear power as an excuse to get fissionables for 'power generation'. Nevermind the fact that Iran is a member of OPEC and practically swims in oil compared to the rest of the world... nope, it's gotta be nuclear power. Yanno, for the kids. And the puppies. Right. Pay no attention to that enrichment facility. It's only there to... umm... help our nuclear power goals be achieved. Yea.
It doesn't need to be able to go Critical and cause an explosion to be used as a weapon of terror. Simply broadcasting nuclear materials in a 'dirty bomb' is quite enough to be used as a weapon of terror. Because, as I mentioned previously, that's about the only thing a nuclear weapon is good for anymore. They don't need to enrich it to weapons-grade for it to be used as a weapon. Even if a dirty bomb doesn't actually do all that much to background radiation realistically, depending on how heavily populated the area is, the stampede and panic could kill thousands or even millions.
Yes, virtually any power source can be weaponized, so this isn't exactly a unique argument, however nuclear problems tend to hang around a very long time, and cleanup is a PITA, so this puts it in a special category of 'hell no' in my mind.
Having said that, the nuclear material is already out there. Using it to power homes rather than blow craters in the ecology would at least safely get rid of the stuff. Eventually. We might make some kind of dent in the stash in... oh, about a thousand years or so. I see fission as a stepping-stone energy source, a temporary stopgap that we CAN use NOW. Also useful later on in 'battery pack' stashes. Setups like
this one can be used to power devices with a lot of power draw over the one to ten year time frame, which will have uses even after we've largely moved on to better energy sources.