I think it really depends on your light. I have been twisting mine ever for about 2 months now and the o-ring is fine. Like i said there has been at least one CPFer who has had the o-ring destroyed by the action. Taking the o-ring away is perfectly ok but i am guessing it would allow leaking? I wouldn't try to take off the o-rings if i were you. Too high of a chance to snap them in the process. My personal stand is that even if the light twists fine without lube, i will apply a little lube for ease of mind.
One more thing, IIRC, i was told that silicone grease is not a good choice for use with fenix o-rings. Someone more knowledgeable about this might wanna chime in.