So, is it harmful (to fish or plants) if I dont remove them ?
Yes and no. Depends how much film: this film blocks light from getting to your plants. It's helpful to remove as much as you can, so your plants will get all the beneficial light that you provide them. As for fish, there again it depends. Most likely not, but then you have to look at why you are getting so much of a buildup. Your "good" bacteria break down this protein and converts it to nitrate. The more work these good bacteria have to do, the more O2 they have to use. These higher nitrate levels may not be suitable for some species. You see this alot in salt water tanks. So all in all, I would say you are ok. But hey, I'm no expert.
I heard black mollys are good to remove this film also.