Do you normally clean all of your filter media in dechlorinated tap water? Or just like the filter pads? My guess would be, either way, that even if there is a larger die-off than normal, it's not going to be really noticeable in your tank in anyway, especially if you're only cleaning out the sponges. If you're worried though, you could just use a small amount of water directly out of the tank and then top off your tank. :thumbsup:
I've never heard of something as hardy as bacteria being killed by a temperature fluctuation, especially to a lower temperature. I think the worst that would happen is their metabolism may slow a bit for the small amount of time you have them out of the tank.
Here's an interesting google cache of an article with a lot of information about nitrogen handling bacteria.