My point was that we don't have to go all bleeding-heart-animal-rights-nut to understand that we are doing harm. Stress is a neurological response that can be observed. "Suffering" is an subjective emotional response to stress. Two people can undergo the same stress. One might "suffer;" the other might not.
Stress suppresses the immune system and leads to disease and premature death. Once you understand that, the question you have to ask is, are you okay with that, and why?
I generally agree with you here, except in a lot of cases you have to be very familiar with the species of fish to know whether it is suffering or not. I had my first betta in a half-gallon "tank" without a heater. I didn't know better at the time, and I only changed his water when it looked dirty. The fish seemed fine to me (except for twice when he had some sort of fungus thing, which I treated), and he lived for over 3 years. Once I knew how wrong I was in the way I had taken care of my betta, I got another one and kept it temporarily in a 1 gallon tank. He was much more active and actually showed a personality I never knew a fish could have. Then I realized my previous betta survived but probably wasn't happy. He also could have lived longer as a betta's lifespan can be up to 5 or even 7 years.
When I moved my new betta to a 5 gallon tank, he seemed even happier, he was able to spread out his fins and swim fast.
So you have to know the signs of a stressed or sick fish. Every species is different and when I had my first betta I didn't know that bettas are active fish with personalities, so I thought he was normal just kind of drifting around in his tiny tank.
I forgive people who don't know any better because I was there. I'm not going to freak out and act like a member of PETA. I just try to educate people according to what I've learned and my own experience. I do get upset if people continue to keep a fish in bad conditions after I've tried to tell them the way they could care for their fish to make them happier, but I can't really do much more than that.