it is very expensive to design and build a light that successfully utilizes all light from the source and does not overwhelm opposing traffic
This hasn't been true for decades. In the past, when only parabolic reflectors were available, the only way to create a sharp beam cutoff was to block off the lower portion of the reflector from "seeing" the light source. With the advent of non-parabolic complex-shape reflectors about 25 years ago (actually first developed in the early 1970s, but not widely commercialized at that time), it became not just possible but rather easy to use the entire reflector/lens area and still have a beam with a sharply-defined cutoff. Likewise, when projector-type headlamps evolved from simple ellipsoidal reflectors to complex-shape reflectors, the light losses due to the cutoff shield shrank considerably. As computers and software used to design optics have grown more powerful, this has all grown easier, not harder. And LEDs have made it easier still, as there are no 2nd- and higher-order reflections (as off bulb walls in the past) creating difficult-to-control stray light that would tend to cause glare above the cutoff.
Low-beam reflector headlamps have to block all light exiting the lens directly from the bulb as this is too bright a source.
You don't seem to properly understand the function of the bulb shield. It's not primarily to block glare directed towards other drivers, but rather to block upward stray light that would tend to create backscatter/self-glare in fog, snow, and rain. Whether a bulb shield is needed depends on the architecture of the specific lamp in question, but bulb shields generally do not significantly reduce the efficiency of the lamp.
the issue remains with bicycling to produce a system that is compatible with being on the road and providing enough light to be safe.
Numerous manufacturers are doing a very good job of it, and the products just keep getting better. The main reasons why there are poor-performing lamps on the market is that people buy them, and no regulation prevents them from doing so.
getting mfg's to even consider complying with DOT (NHTSA) rules is a very long shot due to the costs involved.
What specific NHTSA rules do you have in mind for manufacturers to comply with?
Yes, Phillips has done somethig, but they are a HUGE international company involved in ALL aspects of lighting and they own the automotive lighting realm.
No, they don't. They're just one of many companies involved in automotive lighting. They have a good chunk of market share in automotive light sources, but they don't manufacture automotive headlamps.
A golden savior might come in the form of optics designed for automotive use
I don't agree that an automotive type of headlamp would necessarily be optimal for bicycle usage.
we really need to use effective lights.
Quite true!
I do suggest they run TWO rear lamps, one solid (brightest), one blinkie, because having a solid lamp provides better depth perception and tracking
That is a very good recommendation. I agree.
The big, central problem not (yet) mentioned in this thread is that everyone's got their own ideas about what kinds of lights work to make a bicycle effectively conspicuous in nighttime traffic. And everyone's sure their idea is right because they can make what they consider a good case based on "common sense" to argue in favor of blinking lights, flashing lights, pulsing lights, moving lights, lights mounted here, lights mounted there, lights of this color, lights of that color, lights in a triangle pattern, etc. There's not a scrap of science behind any of this, just hoary anecdotes that amount to nothing useful.
We are far away from being able to specify anything usefully, whether it's a wholesale adoption of the German StVZO specifications or devising an all-new spec; what's
needed is basic research on what bicyclists need and how best to specify and provide it. That research has not yet been done, and I don't hear much interest in it among the community of researchers and scientists qualified to do it. They've only just (finally) gotten around to looking at those questions with respect to motorcyclists, who have -- like bicyclists -- had inadequate and non-optimal lighting for many years.