This is a very bad idea. A dead CR123 is not inert at all! Neither is it a convenient conductor of electricity, between the + and - there's a lot of volatile gunk that does not like battery chargers.
If you do this, the cell will heat up and most likely explode, disgorging extremely nasty chemicals in the process, most notably hydrofluoric acid(HF). HF penetrates skin quite easily and unnoticeably, even more so if you happen to breathe the fumes from the explosion.
Once inside your body, HF will destroy whatever tissue it encounters and the pain will become quite severe when it starts reacting with the calcium in your bones. This may not manifest until hours after exposure, which may be too late for treatment, possibly necessitating amputation. As little as 2% body exposure can be fatal, depending on concentration. 3 parts per million is considered the safety threshold for HF. This really is the stuff of nightmares.
Use an inert spacer, machined out of a solid block of metal.