Hi Get-Lit,
You probably already know this, but if you want maximum throw (technically candela or candlepower) then the only thing that matters is surface brightness (technically called luminance). Lumens and arc size are irrelevant (although arc size and surface brightness are closely related, for a given number of lumens smaller arc sizes will have higher surface brightnesses). So when looking at the specs for lamps you should focus on finding one with the highest possible surface brightness. Surface brightness / luminance is usually measured in cd/cm^2.
When it comes to high surface brightness the classic choice is xenon short-arc as you already know. However it is very old tech and suffers from very low efficiency (around 30lm/W in the 500W range, 40lm/W in the multi-kilowatt range) and short life, but it is also the only HID technology which doesn't take any time to warm up and it also produces the best quality light which is probably why it is still so popular today. The Osram XBO lamps have surface brightnesses ranging from around 40 000 cd/cm^2 in the 500W range to around 100 000cd/cm^2 in the multi-kilowatt range.
As you already know another HID technology with high surface brightness is mercury short-arc which is used in the Maxablaster. The Osram HBO lamps are really designed to produce intense UV radiation, not visible light and don't even have a particularly high surface brightness. Most aren't any better than the XBO line of lamps and in many cases are actually worse. The HBO 500 W/2 has a surface brightness of only 30 000cd/cm^2 vs. 40 000cd/cm^2 with the XBO 500 W/H OFR.
The one exception to this is the HBO 103/W2 lamp used in the Maxablaster. It has a much shorter arc and a much higher surface brightness than the rest of Osram's HBO lamps. It has a surface brightness of 170 000cd/cm^2! This is what allows the Maxablaster to exceed the Spectrolab Nightsun in throw with a much smaller reflector.
Getting back to xenon short-arc lamps, Osram makes a lamp which is much more suitable than the XBO 500 W/H OFR. The XBO 500W/RC OFR is a new (I think) xenon "ultra" short-arc lamp with a much smaller arc and much higher surface brightness than the XBO 500 W/H OFR. Its surface brightness is an incredible 260 000cd/cm^2, 6.5 times that of the XBO 500 W/H OFR! This is even higher that the HBO 103/W2 used in the Maxablaster. The luminance spec is only the average surface brightness, so the peak surface brightness will be even higher still! It's possible that the peak surface brightness is more than double that of the HBO 103 W/2, which means that in the Maxablaster it could produce over 100 million candlepower!
Back again to different HID technologies. Xenon short-arc's main markets are cinema projection and searchlights which need high surface brightness but don't need particularly high efficiency or long life. Over the last 15 years or so a new market has emerged which needs a lamp with high surface brightness, high efficiency and long life: portable (compact) consumer projectors. They need high efficiency because of their small physical size which is unable to dissipate much heat, they need long life because they are a consumer product and they need high surface brightness because of the small size of their optics. In fact because of their small optical system they need a light source with a surface brightness even higher than that needed by cinema projectors! So a new light technology with higher surface brightness, higher efficiency and longer life than xenon short-arc was needed.
The first such technology is short-arc metal-halide. It's around 60 - 70lm/W, or twice that of xenon short-arc and the HBO 103 W/2 used in the Maxablaster. Surface brightness can come close to or even exceed that of xenon short-arc. The Ushio EmArc you mentioned above is an example of short-arc metal-halide. Osram also make a short-arc metal-halide lamp for projection called the VIP R 273/45. It has a surface brightness of 100 000cd/cm^2 and an efficiency of 63lm/W. However the life of short-arc metal-halide is still rather poor and the surface brightness still isn't high enough, so an even better technology was developed by Philips: Ultra High Pressure Mercury (UHP).
UHP lamps have roughly the same efficiency as short-arc metal halide lamps, but have an even higher surface brightness and longer life. 200 000cd/cm^2 seems typical. UHP lamps are mercury discharge lamps, similar to Osram HBO mercury short-arc lamps, but are still very different and have significantly better performance characteristics than the Osram HBO lamps. I don't know the technical reasons for this, but it's probably due to the pressure. As the name suggests UHP lamps are ultra-high pressure, probably significantly higher than the Osram HBO lamps. The HBO 103 W/2 used in the Maxablaster has performance characteristics much closer to UHP lamps than the rest of the HBO product line, it probably has a higher pressure than the rest of the line and may be designed for projector use. However despite having a similar surface brightness to UHP lamps it is still less than half as efficient and has a much shorter life. Osram actually make their own UHP equivalent lamp called the P-VIP which has a surface brightness over 200 000cd/cm^2 although they have only ever briefly mentioned it in their literature, I can't find a datasheet anywhere on their site.
There is an excellent article on UHP technology here:
http://www.iop.org/EJ/abstract/0022-3727/38/17/R01/. One interesting technology is a dichroic reflector coating on the inside of one hemisphere of the arc capsule to reflect light back into the arc allowing much smaller reflectors to be used. They say it increases output by 20 - 30%. It also mentions an experimental UHP lamp with a 0.3mm arc-gap and an even more incredible surface brightness of over 600 000cd/cm^2. In the Maxablaster this would produce over 150 million candlepower, more than the AN/VSS-1 searchlight on overdrive! Combined with the dichroic coating and a higher reflectivity reflector (such as silver) over 200 million candlepower should be possible from the Maxablaster! This is my dream light, although I doubt anyone would be able to obtain this experimental lamp from Philips (someone prove me wrong please!) As far as I know UHP lamps are pretty much standard on projectors these days, although some lower end and older projectors still use short-arc metal-halide.
One final technology is xenon "ultra" short-arc which isn't really a new technology, just standard xenon with a much shorter arc. These can meet and even exceed the surface brightness of UHP lamps (except the experimental UHP lamp mentioned above), but they still suffer from the normal problems associated with xenon: low efficiency and short life. In fact due to the smaller arcs life and efficiency is even less than with conventional xenon short-arc lamps. The advantage of these lamps over UHP is that they are instant on, have better light quality (better picture quality) and are more environmentally friendly (no mercury). The target market for these lamps appears to be high end rear projection televisions and larger high end home theater projectors where the low efficiency isn't as much as a concern due to their larger size.
The Osram XBO 500W/RC OFR I mentioned earlier is an example of such a lamp, although the best example is the PerkinElmer "Cermax" line of lamps. The interesting thing about these lamps is that the arc remains the same length from 125 - 800W so while at 125W they are nothing special by 800W their surface brightness is very high. However they have an integral elliptical reflector which means they aren't very suitable for mega-spotlights (although the $6900 Megaray uses the 175W capsule quite successfully). PerkinElmer also make a 5kW "Sapphire" ultra short-arc xenon lamp for large format cinema projectors, although I've only seen it mentioned in an old press release.
I've ended up rambling on a bit (more like a lot), but to answer your question: for absolute maximum throw the best commercially available lamp in the 500W power range is the Osram XBO 500W/RC OFR. However since you "only" want to throw a few miles you may be better off using a more efficient short-arc metal-halide lamp such as the Ushio EmArc, or a UHP lamp. A 250W UHP lamp produces more lumens than both the XBO 500W/RC OFR and the XBO 500 W/H OFR while obviously drawing a lot less power and easily out-throwing the XBO 500 W/H OFR. Alternatively if you wanted a LOT of light you could use the 600W Ushio EmArc which should still have "sufficient" throw in a large 9 inch reflector to reach a few miles, but will produce more than 3 times the lumens of the XBO 500 W/H OFR while only drawing slightly more power.