Number of factors; but big one is cost and difficulty. I don't think there's another hobby out there that requires this much investment in terms of money, knowledge and time. Also it's on your mind 24/7 as a lot can go wrong and sometimes it can be catastrophic.
As others mentioned, it can also be skill-level. How many times have you envisioned an aquascape and realized your reality pales in comparison to your vision.
Any other hobby, you can quit and come back with relative ease [fishing, golfing, etc.] In this hobby, it often forces you to sell everything.