That won't work because the dirt and sand will eventually settle into the spaces you mentioned and the gravel will be on top or at least show through. I haven't tried it myself but maybe you could try to prevent this by covering the gravel with a fine mesh before you add the dirt and sand. Good luck.