I used to keep saltwater tanks, and I tried to get everything aquacultured/tank-raised. And live rock came from Tampa Bay Saltwater, where they're required to replace as much rock as they took out, and let it sit for a given number of years so it could re-populate with corals and fish.
The primary problem with the marine trade is that we don't actually know how much of this stuff is being removed. We know what *arrives here alive* -- that's what gets recorded. But we don't know how much dies before they even get it into a box, or how much dies in between middle men, or how much is dead on arrival.... And the practices that people use to get the fish and coral out of the water are highly destructive to the environment. Coral isn't just gently plucked off a reef -- it's usually hammered or even dynamited. Fish are often netted (and anything that isn't marketable is just chucked aside to die), stunned with electricity (which zaps everything around), or even poisoned. It just isn't cool.
And in response to a previous comment about tourism -- yes, it's destructive. But it's not destructive in the way you think it is. Direct damage to corals from people is pretty small in the big scheme of things. The real damage comes from the carbon dioxide emissions flying to and from your destination, even if you're going to Canada versus the Caribbean (ocean acidification and rising sea surface temperatures), tearing down forests to build hotels (erosion/sedimentation, wastewater treatment), flying in food to feed tourists (more CO2, and waste in general), etc.
Certainly everything we do has an impact, some things greater than others. Keeping tanks has its impact, even if we buy all tank-raised species, because we're burning fossil fuels to run lights, powerheads, filters, etc., and constantly changing/topping off water, etc. Everything is a trade-off.