I am just getting started in this hobby, and I am working with a VERY tight budget. I bought used tanks from Craigslist (along with a new 7.5 gallon nano tank that seemed like a good deal, and came with all of the necessary equipment). While it is tempting to throw down a bunch of money right from the beginning, the key to keeping your spending under control is a bit of patience, and knowing which items you can skimp on and which items you need to splurge on.
I have wasted money unnecessarily already, only to do further research and discover a cheaper and better way to do things. As far as tanks go, I've done the right thing by buying used. I have had to put in extra work re-sealing the silicone and what not, but the savings were tremendous.
As far as filtration goes, I've spent a lot of money on high-end canister and HOB filters, only to discover that I should have just gone with inexpensive sponge filters (and I already had air pumps that came with my used tanks from Craigslist).
For substrate, I've gone with Caribsea Eco Complete and Flourite mixed together, and had great results with my plants so far. I have had a few melting episodes with some, but I scour the shelves at Petsmart for marked-down plants, and expect that some won't make it. I've also ordered some online, but found that the bulb-type plants are the way to go if you plan to get them this way. Drsfoster&smith's website is a great place to buy certain items, such as glass canopies, substrate, and driftwood, but Ebay is better for heaters, filters, lighting and other such equipment. Definitely shop around!
I bought an expensive Aqueon water changing system, and have found it to be rather inconvenient to use, so I'm not sure I recommend it. I have gone to simply siphoning water out with tubing into a bucket & refilling from a bucket.
Online shopping is something of a way of life for me, and I am sure that it may actually be an addiction in my case, but NEVER buy anything online without first searching for a coupon code. Set a budget for yourself and shop around until you can get everything you need to fall within your budget.
One thing I can say about this hobby is that is provides a great deal of satisfaction and it has been worth every penny!