why are there no optics that spread the light instead of focusing it?
While there a lot of retrofits using highly focused emitters as you rightfully gripe about, the few getting a lot of good press like the Ecosmart we've been talking about use a diffusion dome. While this limits the over-all amount of diffusion because of it's small size, it's roughly on par with a frosted light bulb in terms of softness.
I've spent a lot of time working out how to fix the 'specular' nature of LEDs for better aethestics, and the physics are rather interesting.
Basically, you have three ways to diffuse the light of a point light source like an LED. You can either (1) diffuse it via frosted plastic (or micro faceted materials). Or, (2) You can reflect/bounce the light off a high albedo surface such as a white ceiling or panel. Or (3) you can use lots of LEDs spread out to simulate a larger light source such as LED strips.
Retrofit bulbs are going to be forced to shine through diffusion plastic given their restricted size, but the good thing is this is cheap to do and high tech materials with high transmission rates are being developed as we speak. This is the same stuff being used for small diffusers that clip over LED optics.
LED strips are obviously all over the place, but have some practical / aethestic limits such as needing it placed far enough away so you don't get annoyed with a dozen shadows of your pencil when you do a crossword puzzle. Or, combine a mild diffuser over the strip.
Reflection based light sources are perhaps the most intriguing given they have a huge amount aethestic potential even though the suffer the greatest efficiency loss. This is what I've been working on, and like the fact you've been thinking about as well. Shows I'm not off my rocker. If you think about it, a simple white lamp shade over a standard incan does a pretty good job generating nice room light, and that's kind of my standard (minus the crappy yellow bulb).
McGizmo's house has some cool design ideas, but over-all my impression of it is a lot of accent light that doesn't produce enough direct light. Not everybody wants to paint their walls white to bounce LEDs. So, I've tried to figure out how to localize very diffuse light, or direct it in a way similiar to a nice lamp shade, but optimized for LED.
One idea I'll throw at you for a DIY project is a floor standing reflection lamp. Build a frame of wood, or wrought iron (doesn't matter) about 5-6' high and maybe 10" square. At the top, sit a 12" piece of ceramic tile with the brightest, glossiest surface you can find. Underneath the tile pointed up sit a big heat sink with about 1,000 lumens of either Cree or Bridgelux emitters shooting up at the tile. Build a narrow frame around the sink to hide the emitters and the sink. I built a mock-up of this and it produces a huge sphere of down firing soft light that's more local than ceiling bounce and hence a lot brighter.
Lotsa ways to play with reflection and get stellar light from LEDs, but you aren't going to find it in retrofits, and commercial fixtures are fighting to have the best efficiency specs and not most pleasing light. So, be creative and DIY. LEDs don't have to be ugly.