Perlite should be harmless.
It might work it's way to the surface eventually (like empty osmocote balls or sticks from MGOCPM, etc.), but it might also get waterlogged, I don't know.
If it does surface, it should be fairly easy to remove with a vacuum or something.
Or, if you don't want to put it in, soaking the soil, stirring it up, and pouring off the floaties should help, I think...