My review of the Impala's lights is based on the objective criterion that it is not possible to see the side of the road immediately in front of the car; rather the beam seems to gradually fade in well forward of the car
Here's a sticking point. That's not an objective evaluation (or criterion) at all, it's a subjective impression: you are judging the headlamps based on what it feels (you say "seems") you can't see. In fact, we (human beans) aren't able to make accurate judgments of how well (or how badly) we can see, or how well (or how poorly) a headlamp performs. It's just not something we're equipped to do; our subjective judgments are very often, very far out of step with the objective reality. Plus, the number-one factor that drives subjective "ratings" of headlamps is foreground light, that is the amount of light on the surface of the road close to the vehicle. But foreground light has zero-to-negative effect on the actual safety performance of the headlamp (i.e., how well it actually lets the driver see what they need to see to drive safely). This same specific disjunction between reality and perception is what causes so many people to misuse their fog lamps as always-on auxiliary lights: the extra foreground light creates the impression (but not the reality) of improved seeing and safety. There are many, many other "we think we see X, and we think Y makes us safe, but neither is the case" specifics involved with headlight beams. And photographs are equally useless for other reasons. The only way headlamps can really accurately be assessed and compared is on a photogoniometer. There's some info and examples
here.
The performance of the Impala installation has bothered several people enough to register complaints at NHTSA
I don't doubt that -- but while I don't want to sound like I'm just dismissing the idea that the current Malibu headlamp might have annoying deficiencies in its performance, I do want to point out that people file complaints at NHTSA all the time for all kinds of things that aren't really the way they seem.
including one accident where the driver left the road for a ditch because he or she could not see.
That might be what happened...and it might not. This hits on why some aspects of vehicle lighting systems are slackly regulated in America: our regulatory structure requires positive cost/benefit for any mandatory aspect of a safety standard. Risk analysis or simulation isn't allowed, it has to be actual cost/benefit analysis based on actual crash data. That's easy to do for crashworthiness standards (seat belts, for example) and for post-crash survivability standards (fuel system integrity, for example) but it's very difficult to do for crash-avoidance standards (lighting being the prime example) because effective lighting creates "unCrashes", that is, crashes that don't happen. You can't assess the costs of a crash that didn't happen, so as a result it's hard to legally justify stringent lighting standards in our system. To be clear, I am definitely not defending this sorry state of affairs, just describing it.
Back to your comment about the couldn't-see/went-in-a-ditch crash: maybe better headlamps would have prevented this crash, and maybe not. We cannot rely on the driver's conclusion. The driver simply isn't equipped or in a position to make that conclusion reliably.
Regarding subjective versus objective evaluation, I wonder whether the standards applied by objective static tests might miss important aspects of performance that are immediately apparent upon use in real-world conditions.
That's mostly a reasonable question to ask, but the answer is no. That's not the problem; each and every test point in the headlamp standards is well backed by good science. The problem is that the minimum allowable headlamp performance is not necessarily adequate. Regulators don't tend to advance the bottom end of the standard nearly as often as the market advances the top end of the performance available to drivers. A headlamp that gives performance that was considered legal in 1975 is still considered legal on a new car today, even though it is in most respects hugely outperformed by newer headlamp designs and technologies.
More importantly, headlamp performance is not a single "black box" that can be rated or ranked on a scale. Headlamp beams are specified and evaluated in terms of minimum and/or maximum allowable intensities at a range of angles up, down, left, and right relative to the headlamp's axis. Each test point or zone has an allowable intensity range. Because of the way our human visual system works, a headlamp that produces "X" amount of light straight ahead of the car in the foreground and "Y" amount of light at wide lateral angles can dependably create the impression of perfectly adequate light at wide lateral angles, while a headlamp producing "2X" amount of light straight ahead of the car in the foreground and the same "Y" amount of wide-angles light as the first headlamp can seem to leave the wide angles totally unlit.
I say this was a "mostly" reasonable question because people who don't understand what they're talking about (headlamps or anything else) often resort to the "real world" argument when faced with facts that don't line up with their opinions. That is "Fine, whatever, I don't care about your so-called 'objective' tests; I KNOW WHAT I CAN SEE IN THE REAL WORLD!!! I DRIVE WITH MY EYES, NOT WITH A PHOTO-WHATEVER-METER IN THE LAB!!!". That argument is without merit (and isn't worth quarrelling over; those who advance it are not interested in learning). I don't think you're putting forth that line of argument, I'm just bringing it up because you used that "real world" phrase.
As an aside I owned a Dodge Shadow many years ago and was struck by the poor quality of lighting on that car
Yes, you're right about that. Chrysler installed cheap, nasty, minimal headlamps and severely under-rated wiring on almost everything they built with replaceable-bulb headlamps through the early 2000s. Remember, the Federal standard does not require "good" headlamps, it requires
legally compliant ones. Legally there is no difference between a headlamp that just barely meets the minimum requirements and one that is near the maximum allowable output. (It might interest you to know that the lousy headlamps on your Shadow were a result of Ford Motor Company's cheapness-centered headlamp system specifications they submitted to NHTSA in 1983, which NHTSA rubber-stamped and wrote into law as the basis for all American-market replaceable-bulb headlights).
I'm struck by the extent to which inertia, for good or ill, strengths and weaknesses, drives design, features and performance in the car industry.
H'm. I'm not sure how this applies here. How do you mean?