If you are planning on doing a fairly typical paludarium, I don't think a HOB filter will work for most set ups. They can only draw water an inch or two up (maybe more, but not enough to work on a 1/2 full tank).
Better would be to just get a small powerhead or fountain pump, and build a filter into your waterfall.
For substrate, You'll probably want two different versions, your submerged substrate (which would be any of the typical aquarium substrates), and the terrestrial substrate.
What you plan on using for the terrestrial portion is probably going to depend on the plants and the conditions. Do a search for something called "Atlanta Botanical Garden Mix" (ABG mix), a lot of the dendroboard folk use that and it seems to work pretty well from what I've read (haven't used it myself...).
Anyways, I'd recommend going with a bigger tank if you have the room for it. 10 gallons are already fairly small, and when you stick backdrop in there, and then have it only 1/2 full, there isn't much water or room in there.