I wouldn't carry them loose, without taking some of the precautions mentioned above.
Here's something funny that happened to me quite recently, that sortof applies (albeit, with 9 volt batteries, which are probably more dangerous):
I was doing the annual* battery changes of the smoke detectors in the house -- there are 9 of the miserable things, and we have high ceilings. After changing each one, I was carefully taking each leftover battery back to kitchen before doing the next one. But once I started doing the upstairs ones, I got impatiient, and just shoved a leftover battery in my pocket, and started on the next one.
Naturally, this one choose to be difficult; instead of having the smoke detector disconnect from the mounting, the mounting itself came loose from the ceiling. So I replace the battery, climb back up on the ladder and start screwing the mount back in, when suddenly I start to feel a sharp pain in my leg, where I'm leaning against the ladder. After a few seconds, it gets even more painful, and I stick my hand into my pocket absent mindedly to see what's going. Ouch! Ouch! I frantically scramble down the ladder (in the back of my mind, I'm thinking the 2 "dead" batteries are leaking acid or something), and once I'm down try to get them out of my pocket (ouch! ouch!).
Eventually I evert the pocket and the two "dead" batteries fall onto the carpet and skitter under a bed, where I have to fish them out.
Turns out the almost dead 9 volt batteries shorted out in my pocket, and darn near burned their way out. It took them quite awhile to cool down again, too. I saved the other 7 batteries for use in my PAL lights, but those two I threw away...
*I don't really change the batteries once a year; the detectors are primarily powered by the power mains, and the batteries are just a backup. As soon as the battery gets low, the detector starts chirping (very loudly), so I change all of them as soon as the first one starts to chirp. Which indicates that most of the batteries should have been fairly depleted.
BTW, you don't want to know what it sounded like when a water leak shorted out one of the smoke detectors -- all 9 of them were shrieking like unholy terrors [they're wired together], there was no way to figure which one was having a problem, and it happened in the middle of the night. There were 5 of them within twenty feet of where I sleep -- gads, I hate those things...