My thoughts is if your fluorite is building up excess detritus is you have poor circulation patterns/gas distribution in your tank. That or your feeding way to much. Fluorite has a good pore size for letting oxygenated micro-currents into substrate, when that happens good bacteria will literally eat every scrap of organic matter that dissolves/settles into substrate. Gravel will actually clean itself unless your grossly overfeeding. Tanks with discus etc where your feeding a lot of high protein food during rearing they may fall behind, nothing can keep up with that amount of noxious matter except a stringent water change and cleaning regimen. If you got a few tetra your trying to maintain and this becomes a problem you need to look at how much your feeding or if plant leaves are adding to compost pile to quickly from poor plant health.
Got whole tank picture of this tank?