They'll be fine. Once bacteria is growing in your substrate and filters, it will grow fast. Your tank is probably already cycled, the substrate is just still leeching. (those are two different things)
So id take it out now if you don't like it, you'll have to take it out sometime!
Good points! I will.