SW enthusiasts who don't change water often because it's expensive shouldn't be in the hobby. I do more waterchanges on my SW tanks than my planted. I have a skimmer on 1 of 3, the other two I keep stable by changing 25% every two weeks. One of the reasons for this is many of the mineral elements get depleted in the SW, (which is never evaporated seawater, short reason is that elements that precipitate out during evaporation process do not reconstitute properly when re-hydrated.) So, even if you top-off and/or skim to keep nitrates low and DOCs manageable, you lose alkalinity, calcium, strontium, iodine, etc as the inhabitants use them up. Some "reefers" compensate for this by dosing lost minerals and not changing water, but this usually results in an algae farm instead of a nice aquarium.
The same principles apply to freshwater. Certain elements in the water will get used up, others left there. There is no real "magic" way to process out all the waste, other than waterchanges. In some ways, a planted tank is in a better position to go long-term without waterchanges, because the plants process fish waste for food, unlike most SW creatures, (other than algae. *sigh*) So, though you are losing important trace elements, it is easier to dose them or survive long-term without adding them in a planted tank.
That's not to say slacking on waterchanges is recommended. Besides the organic waste component, there is the matter of the hormones building up in the water. While still theoretical in some aspects, there is growing evidence that stunting is in part affected by growth inhibiting substances produced by the fish.
There are always people who will "experiment" with aquaria... not doing waterchanges for months on end, seems to happen as often with reefers as with aquatic gardeners. Reefing sometimes utilizes Deep Sand Beds, which performs the function of turning harmful nitrate levels into nitrogen gas, which then escapes the aquarium. It is a matter of some debate, many feel that this gives a false sense of security to the reef aquarist. ("I don't have nitrates, and my pH is good, I don't need a waterchange...") Eventually, the DOCs that are not converted, build up to a level that the system fails, often catastrophically.
I believe the same thing can happen in freshwater tanks, though not as dramatically. Planned closed-loop systems are usually much more stable, because research is done and attention given to balance. But the average aquarist will make neither time nor effort to properly maintain a recirculating system.
All that to say this: IMO, most of the "hidden toxins" are really an accumulation of non-harmful elements to unsafe levels, combined with a depletion of necessary elements... as others have said. LOL!
(please ignore the talkative woman behind the curtain.
As an interesting note regarding depleted elements... there is a woman in the local aquatic club. She brings loads of plants every month to the meeting for auction. Her plants are healthy, and don't melt when going into a new tank. She has no algae issues. All of her tanks are low tech. She has minimal lighting... in all tanks 1wpg or less, and diffused daylight into the rooms with the tanks. She doesn't use heaters. She has plain old gravel substrate. Yet she is growing moderate light plants, easily, and in good form.
What she does for her tanks: every single week, without fail, she changes 30% of the water. That's it. I suspect there is enough trace mineral and nutrients in the water to feed the plants, and her consistency keeps them happy and healthy in conditions where you would not expect them to grow.