The sellers here are a good place to start for plants--not uncommon to find free /free for cost of shipping selections of clippings too. Great way to try new plants at a low cost. The other place I typically hit is Aquabid.com.
Substrate doesn't have to be expensive, everyone has their own favorite. If I'm only doing heavy rooted plants like crypts and swords and/or mounted plants like anubia and java fern, I rather like plain old aquarium gravel--in part because I've got so much of it stored from earlier tanks! If I'm going to be doing stems, I reach for Flourite. Dirtied tanks (usually capped with sand) can be wonderful--but frankly, too much of a hassle given my habits of setting up tanks on a whim and frequent re-scaping.
The one place I wouldn't scrimp on is the heater. There are plenty of reasonably priced heaters out there that will do great, but don't lowball it. A heater that just stops working isn't a catastrophe usually--the tank cools slowly enough you have a chance to catch it. A heater that malfunctions ON however can easily cook your fish and plants long before you spot the problem.