To me this is your tank working to balance itself out. With consistent care (waterchanges, ferts, scrubbing algea away), the diatoms should stop growing and the tank should reach a balance. Diatoms normally become a nuisance within the first 2-3 months of a new tank, then they die back. If I recall correctly, quartz/silicates can cause diatoms algae as well. However, I doubt the rocks you placed in the tank a really causing you major issues.
As you increase plant mass in the tank, more ferts are needed. I would not be afraid of nitrates, your tap water may have some, but it's certainly not going to be enough for a thriving tank.
I agree, very unlikely rocks are causing a problem, especially with the lack of CO2. CO2 causes the water to be more acidic, which causes the rocks to leach more and can cause parameter issues with GH, and KH if there's CO3 present. More likely to cause algae problems is the wood - wood breaks down releasing organics into the water column which algae will thrive on. I believe we just have a case of a new tank settling, as Vinster stated. Diatoms are unattractive, but they sort themselves out.
I would like to mention that letting your water sit for 3-4 days after treating with Prime is not the best idea. More and more municipalities are using chloramine as a disinfecting agent as opposed to just chlorine. Chloramine is chlorine and ammonia bound together. With Prime, it will immediately break that bond, neutralize the chlorine which can then dissipate, but it simply "holds" ammonia. Prime is only effective for roughly 24 hours, meaning that if your city uses chloramine, that ammonia will be free ammonia in the water by the time you're adding it to your tank. This could be part of the issue with your fish. It's also possible that a die off of the beneficial bacteria could be causing small ammonia spikes. If you were trying to achieve total removal of 2 ppm ammonia in 24 hours with cycling, that would be equivalent to a pretty big bio-load, which is not present in your tank. As a result, the bacteria colony can die back, which will result in quick traces of ammonia hitting the water column.
I would recommend temperature matching your tap water to tank water as close as possible, adding Prime, giving a stir and immediately adding to your tank. There is no reason to "age" your water. Prime acts immediately. Also, check your city's water quality report to see if they use chloramines. If they do, this could potentially explain your issue with fish death.