I'd second poking around the interwebz. There are good articles and posts on it.
I like winter activities, so I use headlamps in the cold, for very long timeframes (maybe 3-5 hours outside consecutively), so I've read a lot about this, and tried some different options.
I know some brands have Li-ion powerbanks that have a warming element wrapped around the cells, and while that DOES eat the capacity, it does work (Electric cars do this, too, apparently).
Another EXCELLENT option is to get a headlamp that can use an external powerbank, and keep that under a hat, in your jacket, etc. I tend to keep my spare Eneloops in the cold, but keeping some Li-ion cells inside a jacket pocket might make a big difference.
One of the more interesting points is when you get into different battery chemistry performance in the cold. Lithium primaries tend to be better than Li-ion rechargeable in low temperatures, but NiMH batteries get...interesting.
Eneloops (the best NiMH) store very well in the cold, and they'll retain their stored energy even better.
USAGE in the cold is really interesting. LOTS of people deep freeze cells, and measure the voltage and whatnot, and show Eneloops are so-so. BUT, one of the more interesting tests you can sometimes find (and usage experience substantiates it) is that Eneloops seem to quickly return to operating temperature very rapidly once you start using them. Panasonic rates them for usage down to -20*C (which is about -4*F) without any issues. Below that, I tend to not spend too long outside continuously (as that can get dangerous if something goes wrong).
Long story short, I didn't know a lot years back, but I had Eneloops, and bought some headlamps that took AA and AAA cells. I never had any issues. As I got more into flashlights, I started trying out Li-ion cells, and those tended to not perform super well in the cold. I went back to Eneloops, and pretty much just use them in the cold. My experience is similar to a lot of cold-weather folks' experience; they seem to work just fine in the cold. I live in the Midwest, so I'm typically in the Plant Hardiness Zones of 5A and 5b. I might be outside with a headlamp on for several hours at a time.