It depends on the flashlight. If it has a quality mechanical switch that physically disconnects them from the circuit it should be fine.
Kicking around in a drawer is fine if there is no loose metal to cause a short.
Here is the thing, they are batteries, the put out power. Alone they are fine. If you toss them in a drawer full of junk, and they get jostled around in such a way that they touch something metal, that touches something else metal, that touches the other end of the battery and shorts it, then yeah you have a problem.
But if you put them all nice in a storage box, or positioned carefully, basically common sense stuff to avoid the chance of a short, they could sit there a very very long time without incident.
Chances are if they have been somewhere for a long time without a problem, they will continue to not have a problem unless you change something.
Often the true hazard of batteries comes from their continued use after abuse.