They are slow release so even if a small amount of the granules came out of the gravel it shouldn't be fatal, as long as your filter is working. You will just have a bit more nitrates for some time.
The danger is when people overuse them, putting in way more than they need.
They are reasonably safe if you just use common sense. Don't put in more than you absolutely need and remember to poke small holes in the capsules so that they don't float up at some point.
So that means you really should only be using them for swords or tiger lotuses or other plants that feed mostly from the roots. For these plants they can be used more heavily, say 1 or 2 for each plant.
They can be used sparingly for any other plants that do feed somewhat from the roots, but for these you should space them pretty far apart.