1. the backpacking breed does not see lights as important as we do,
2. they normally not hike the night and thus small, cheap, long running lights is what they buy.
I am a backpacker and I consider lights as being very important. I have friends who take backpacking seriously and they carry decent lights. What they don't do is buy a new light every month because the newest one is brighter or longer lasting. Most l.e.d. flashlights give you decent runtime with adequate brightness. One of my friends hikes the John Muir Trail annually and still uses the same original Petzl Tikka Plus to this day. It has survived years of use and it's trustworthy. Durability is more important to a backpacker than anything else. I don't care how efficient a light is , how bright it is, or how much runtime it has. If it can't survive the hard use of backpacking, it will never go in my pack. Older lights have a proven track record and when people buy a new light, they buy what they're familiar with and what they trust. They don't want to fiddle with a new U.I. or have to put the batteries in a different way in the dark. Keep in mind that just because backpackers don't buy the brightest lights like you, doesn't mean they don't consider them as important as you do. They simply have a different set of priorities when it comes to lights. Durability comes first, then runtime, then brightness. They still buy quality lights, just not the brightest lights.
I do go on nighthikes, constantly. When backpacking, I stay close to camp at night so I don't get lost, but I still use a light at night for setting up camp, cooking, cleanup, bearbagging, etc. In camp chores require less light and more runtime, but I still take a thrower light for spotting bears and finding trails.
As for headlights not being upgraded soon enough, I agree that they usually aren't as bright as flashlights. However, for most situations they don't need to be. In warm weather, they attract bugs to your face. In colder weather, when your breath gets fogged up, you can get blinded by it. It makes sense to have a headlight that isn't too bright. By the way, there are way more multilevel headlights than flashlights even today.
Instead of us complaining that sporting goods stores don't have the best, up to date lights, we should be glad that we have better lights available today and decent lights are in cheaper stores. Cree, Rebel, and K2 lights are flooding the market. Princeton Tec has finally upgraded their headlights along with Petzl, and Black Diamond. Surefire lights are now sold at REI and Lowes. I just bought a rebranded Fenix L1T 2.0 with a Leatherman Blast at Costco for $50 (the light alone + shipping would have cost me more online). Energizer, Duracell, and Coleman now have decent offerings at Walmart and Target in the $20-$30 range. I can get a brighter light for $20 than I bought for $60 5 years ago (CC Expedition). Why can't we be happy with the progress being made instead of complaining that it's not good enough?