Warm is fine. Hot isn't. Too hot to touch can be very bad. The P1 on a primary can get fairly warm, but I don't think it can get too hot, even if it's just sitting around (ie, not being held in your hand, which sinks heat away). A 3.7V lithium ion can get too hot fairly quickly. It's a misnomer that LEDs generate no heat (we know better, but the masses don't), and brighter LEDs generate more heat the brighter they get. A Lux V light like a Surefire L4 can get quite hot under normal operating levels.
For the most part, unless it's extremely hot, the LED is probably fine. This varries from light to light of course, but if the body of the light gets warm, that's a good thing, since it means heat is being drawn away from the LED. That's why most lights have aluminum bodies, because aluminum has good thermal characterstics for flashlights. There are definitely some lights that can get too hot for their own good, but the P1 on a lithium primary isn't one of them.