You should worry, somewhat.
An excerpt from an article July 2004
"Lithium batteries may represent the ultimate hazardous material, especially when shipped in bulk as cargo, with the potential to breach all defenses should they catch fire. That is the principal finding of a June report of...