It depends... If it is regulated, it is probably safe. If merely resistored or DD, maybe not.
And/or If it has decent heat dissipation probably OK... If not, maybe not.
And without a regulating circuit, the Vf bin and the value of the resistor becomes a factor.
It will PROBABLY work ok... I doubt it would 'instaflash' unless you are real unlucky. If you try it, just monitor the heat, and if the base of the LED gets uncomfortably warm, go back to alkalines.