Hello and welcome to TPT.
First, hardscape- since you mentioned that you're a newbie and that you would like to save some $, I would suggest simple. Either none, or one piece. One piece of driftwood or one piece of rock. Placed off centre.
The betta you want has fragile fins so nothing rough. Could you find a nice big piece of rock maybe the size of two or three fists? In the neighborhood?
If you decide on wood, I would recommend Mopani I because it is smoother for the betta. Will need to be prepped though, so search the forum for prepping wood.
Plants- the swords can get big but there are smaller specimens available through Tropica. Even then one would dominate half the tank. I have one in a Mini M, see here. Crypts would be more to scale and would be fine in low light.
Mosses would be fine too, I recommend spiky over Christmas, I find the form of spiky is nicer. Weeping is good too. Java moss is fine, but if you want compact growth you'll need to provide good light. The moss can be glued to the wood or rock. In this case, weeping moss attached to the upper portion of the wood/rock is pretty nice.
A small tank is harder to scape, so even if it's hard to choose, I would suggest sticking to less plant species. So, maybe a moss, a rosette plant (sword or crypt eg), and a group of stem plants. Then plant multiples of each depending on size.
For lighting, you have waaaay more choices than I do here, so I'll leave that to my US counterparts to suggest on, same with filtration.
However, for dead quiet filtration, you may want to consider an internal filter or a sponge filter fitted to a power head (vs air pump).
Hopefully, this helps you to get started.