This hobby can be cheap if you know where to look. My dad recently picked up a basically new 29g tank (can't even tell it was used), hood, two filters, gravel, and heater for $35. That's a pretty darn good deal. Fill with water and fire it up!
It just takes a little time and effort to search for the deals. The ability to haggle is nice to have as well.