Debris of this sort is the same as other waste that gathers and has no more nor less danger. It's just that you can see it!
The point that it becomes a danger is when /if your filtering and cleaning begin to run short. It does add to the load the good bacteria has to process or the filter and your cleaning have to remove. In theory, the bacteria builds up to process more as more of it collects, so no problem. When it might become a problem is in some situation where you have a sudden increase in the load for the bacteria, maybe from a dead fish getting lost somewhere? Or maybe due to some error in cleaning, the bacteria is knocked way down.
When either of these happen, having a large amount of debris left around can push things over the edge so that a large amount of ammonia is left not processed.
Something like driving too fast? You can do it all the time and it is not a problem but it does eat up a certain amount of the "slack" we like to keep. All is well until the unexpected happens and then the high speed makes it lots worse than if we had kept our speed under control.
Not a problem to have excess debris laying around but it can be a real problem if you come home and find the filter has not been running.