Halogen lamps use incandescent filaments to produce light. Halogen cycle keeps glass bulb clear. HID uses an electric arc as does Xenon short arc. Difference is in the gas. HID uses a mix of metallic salts in their gaseous states to produce lots of light. The Xenon short arc uses .. er .. Xenon gas! It is much less efficient but has the advantage of quickly reaching its maximum output at turn-on. HID takes a while to raech full output as the metallic salt that make them efficient are solid at room temperature and must be heated to vapourise them. As electric arcs can have negative resistances, some sort of control gear (ballast) is needed for the last two types of lamps. Light source size is smallest in the Xenon Short arc, is larger for the other two types where the sizes are similar. Hope its quite clear?
Xenon can refer to a type of incandescent and a type of HID so it can be confusing. Here's some +/- of Halogen/Xenon Incandescent and Xenon/HID sources.
Halogen/Xenon are cheap. A bulb can be a few dollars. The filament is very fragile when hot. Run off anywhere from 1.5 to 220V, don't care about AC/DC. Full power right within milliseconds. Get hot. Maybe 30 Lumen/watt on a really good day with a tailwind.
HID/Xenon are expensive. A system runs about a hundred dollars, typically more. Require a ballast and igniter. Need very specific power source. No filament to break. Slow to warm up, even automotive systems take a minute or two to reach full intensity. Least efficient put out about 50 Lumen/watt, most put out ~90 lumens per watt.
More detail: Quartz Halogen is a light bulb where the envelope is quartz and the gas inside is a halogen (typically a blend of iodine, etc) This lets you run the filament hotter without it disintegrating. Next step up is Krypton fill, lets you run a smaller wire hotter, with the best incandescent using Xenon fill (Surefire, Streamlight, etc use Xenon bulbs)
Arc / HID / Xenon HID covered in detail by Phyhsuts.
The xenon - krypton - halogen filled (tungsten hot-wire) lamps all have similar performance....3,200 K color and 20-25 lumens per watt
The metal-halide HID lamps (which also use small amount of xenon) are the common HID lamps used on newer car headlamps, a few high-end hand-held lights and industrial-commercial lighting....very effiecient at 80+ lumens per watt and 4,500 - 6,500 K color
The "pure" xenon short-arc lamps are used in stage lighting (although more and more use specialized metal-halide HID now) theatre projectors and other optical path applications that need a very compact light source that can be easily focused. typically 25-40 lumens per watt at 6,000 K
Xenon is also used in long-arc lamps..the common flash bulb and graphic arts illumination (pulsed xenon)
The people that market cheap "xenon plasma" (actually blue tinted halogen) bulbs for kids that want that cool blue headlamp look add to the confusion
Actually 3200K lamps tend to have very short lives, even as halogen/krypton/xenon, and only short lived (typically a few hundred hours), or very high powered ones get past 20 luments/watt. The upper limit for long life lamps (1000+ hours) is about 2950K. For example a GE 100 watt HIR Lamp (the HIR's are the highest efficiency) is 20 lumens per watt. There are halogens at 3200K and 20+ lumens per watt, but they tend to be 150-300 hour life.
Kryton and Xenon are added to effectively slow down the boiling off of the tungsten filament to the walls of the lamp, and as inert gases will not react with the tungsten. The darkening you see on many light bults at end of life is in fact tunstent that has been boiled off the filament and deposited on the inside of the envelope. Halogens react the Tungsten vapor to form compounds so they halogens effectively scrub the tugnsten off the lamp envelope as long as it is hot enough. In the vicinity of the filament, the temperature are high enough for the tungsten to disassociate with the halogen, and it gets redeposited on the filament, incrasing lamp life, keeping the lamp envelope transparent contributes to high lamp output throughout the life of the lamp.
By contrast even small HID lamps are very efficient. The 10 watt Welch-Alyn is 50 lumens per watt, and by the time you get to 175 watts, most HID's are running at right around 100 lumens per watt, a 1000 watts are tyipcally about 110 lumens per watt. These devices do have strong negative resistance characteristics (one they flash over to light, the resisitance in the arc is very low, so they are ballasted to prevent runway.
The most efficient lamps of Low Pressure sodum SOX lamps, although the colour rendition is best described as SUX's...
They have essentially monochromatic output in yellow, 200+ lumens per watt is pretty common in a 200+ watt SOX lamps.
High pressure sodium (a sort of pink/gold output) tends to be a little better than HID, a 1000 watt HPS can deliver about 140 lumens per watt, better colour rendition than SOX (not hard), but nowhere near as good as the best HID lighting can provide.
The atmosphere of tungsten-halogen lamps comprise an inert gas (xenon, krypton, argon, or nitrogen, or a mixture) with about 0.1% to 1.0% of a halogen vapor added. The halogen vapor may be pure iodine or a compound of iodine and bromine.
Tungsten-halogen lamps run at about 7 or 8 atmospheres of pressure as well as a hot bulb temperature to maintain the halogen-cycle which cleans evaporated tungsten off of the bulb and re-deposits it on the filament.
I don’t think there is anything special about a bulb marketed as “high pressure” since all the halogen bulbs are high pressure.
It is marketing hype IMHO