Oh, maybe we have different ones. Mine came in two parts and you are supposed to snap them all together.
I think you are right. I checked one of the spares and it's not obvious that it can be taken apart and used without the holder. Where did you get your lens?