Could you be confusing lithium primaries and unprotected lithium-ion rechargeables? If you overdischarge the latter, you'll damage the cells. Primary lithiums don't sag much compared to alkalines, for example.
As far as harming the light itself... :thinking: ...only if the batteries are giving higher voltage than the circuit (or bulb) can handle.