I may be in the minority... I have pea gravel and a mix of little river stones in my main 30 gal long, and I deep clean the gravel once or twice a year. I've found the plants root better if I do that - I went 8 months without a deep clean and actually saw some plants suffering .
At the point that I go to deep clean the gravel, however, what comes up is less "mulm" and more... dirt? It's black like soil and it doesn't stink like fish poop, but has a scent more like wet garden soil. it's pretty interesting... I think it's a good idea to remove it so that fresher mulm and such can settle in and nourish the roots more, rather than leaving the old stuff there to pile up after it's served its purpose.