I'd recommend at least a moderate rinse; not necessary to have it run clear but definitely worth knocking the worst of the dust off.
First time I used it I didn't rinse and, as it cleared up in a couple days, I didn't understand why folks worried about rinsing it. Sure, it kicked up some clouding when I moved big plants around, but it settled, so I dealt with it.
Last month I decided to pull about 1/5 of the fluorite out of my main tank in order to put in sandy areas. NOTHING compared to moving a few plants and driftwood. Mud city. Daily filter cleaning, having to toss filter pads that should have lasted months as they got worn out being rinsed every day, big water changes, totally not worth saving a couple hours on set up by skipping rinsing.