The tank seems to be coming along nicely. It looks much better with the sand cap, and some hardscape to attach the fern to will help as well.
Have you looked into dosing any fertilizers? With CO2 in the tank and no ferts, it could cause your plants to grow improperly due to an balance of nutrients.
You didn't need to pull the java fern completely out of the ground (though that [alone] won't hurt it any). Only the rhizome needs to be above the surface, which is the thicker, green root-looking thing that all the little black/brown roots come off of. Honestly, though, it's easier to simply tie it to something than trying to plant *just* the roots.