A long time ago I had a set up where I had made some little dirt 'pods' out of a peat/sand/vermiculite (and probably other stuff) mix. It was supposed to be under a gravel cap, and contained in a peat pot. I'm not exactly graceful, and vermiculite isn't the neatest substance to try and work with, so I ended up with plenty all over the tank.
I don't think there would be any harm to fish.
I'm skeptical about the bit with adding minerals to the tank causing diatom blooms. I don't think anything we do/add will significantly increase the amount of dissolved silica, and even so, I'm also doubtful that dissolved silica is the limiting factor for diatom growth.
I think the biggest concern would be cosmetic. Until you do a gravel vacuuming, you'll have little floaty shiny bits scattered around your tank.