Are you being paranoid? Maybe. Are you doing anything wrong? Absolutely not.
It probably adds a negligible amount of time to your process and gives you peace of mind.
Personally, when I do a full filter cleaning I rinse it all in tank water. But sometimes I want to rinse the pads alone and if I'm not doing a water change I just do under the tap. I've never had a problem.