Sorry about your toe there, you do that while breaking up the granite? Haha. Granite is more than likely going to be fine. Granite consists of predominantly silica. It may have a little bit of Ca, and some other minerals that might leach a little bit, but should not be to a toxic level, nor should it cause too much of a parameter issue. I think it would look pretty good as an iwagumi stone! What you could do - give them a bath in a muriatic acid:water solution. I don't think you're going to see much, if any, reaction, but this just eats away all the superficial stuff.
As far as using it as a sub...I don't see why not if you can get it fine enough. I'd stick with something like Black Diamond Blasting Sand(BDBS) or Pool Filter Sand(PFS) though and skip adding anything else. Water column dosing is really all that's needed, I have no problems growing carpets of DHG and HC in my PFS on my 10 gallon.