Is the soil you're using potting soil? I think it would be fine regardless, especially if it's only a couple days. Longer, you might start worrying about mold or bugs getting in your dirt (Bugs sometimes happen regardless even with dirt straight out of the bag... little gnat-type things, IDK what they are. The water will drown them--no worries).
