You use hot water to fill? The problem is that the hot water tank frequently had harder water/mineral buildup compared to the cold water line. May not be an issue, but I don't fill with hot water at all.
Btw, it's not residue in the softener tank, it's how it works. There is an ion exchange, leaving sodium (not "salt") in the water as it removes the hard water minerals.