Too many different kinds of rock.
Use only one kind of rock, and make them look more grouped, connected.
Think of it like this:
A big crust of rock has pushed its way up through the earth.
It has been eroded from wind and water over the years, but the rock has not broken apart. Whatever grain or layers there are should all point the same way. Whatever breakage has happened (leaving tall pointy parts) all happened in the same direction, leaving the main mass as it was. It is this main mass, partially buried in soil, that is usually the goal of a planted aquarium with rocks.
So...
1) put a bed of substrate down that will support the rocks. Begin hills with deeper substrate, valleys with less. But this is more of a bottom cushion, just a hint of 'scaping.
2) Line up the rocks so all the grain or layers or breaks point the same way. Place some (tallest) toward the back. Place some near those tall ones, but in front. Most often these are done in 2 groups, similar but not the same size.
3) Put some up higher, some lower. Add substrate as needed to hold the rocks up.
4) Add more substrate, making the part against the glass front low and level. Make distinct valleys.