Buried trash doesn't always stay buried. Critters will dig up things that you or I cannot smell or otherwise detect.
I recall a camping trip to a national park where the previous visitors declined to use the provided facilities, and instead buried their feces all around the campsite. By the time we arrived, animals had dug up many of the potty holes, and the result was not only unpleasant, it was unsanitary. Our trip to the wilderness (planned for months, etc.) was ruined because someone was too lazy or dainty to use the port-a-john. This is just one example.
Aesthetics aside, some trash -- like wrappers and food waste -- can be burned in the campfire. Some trash -- like bottles and plastic -- can be recycled or (if it's in keeping with local practice) thrown in the local landfill; depleted alkaline cells may fit in this category, but then again they may not. If unsure, I would carry them out.
TMK, Lithium primaries and secondaries contain heavy metals that should not be allowed to leech into anyone's water supply. In the Mongolia/Siberia examples from the OP, the locals probably drink from wells, or -- in remote locations -- maybe even directly from rivers and streams. An ethical guest visiting such locations will err on the side of respecting the health of the locals, meeting or exceeding local trash disposal practices, and protecting the the natural resources that draw visitors in the first place (and thereby help provide a tourism-related income for the locals). Does one or two CR123 cells make a measurable difference in this context? Maybe not,... but you have to see yourself as a part of the larger group of visitors, each with their own trash pile that might seem insignificant until you add it all together over time.
In remote locations, the locals may not have the means to properly dispose of everything that tourists bring with them. Personally, in that case, I would carry back to civilization anything that could conceivably have a health impact if improperly disposed. And honestly, a few spent CR123s and alkalines weigh almost nothing, and they are not stinky or unsanitary to carry home. It really seems like a no-brainer.