I know I'm a bit late to the game, but just saw this thread and thought I'd put in my 2 cents.
The circuit referenced by Norm in post#2 (look at post #1 in that thread) should work fine over the entire range of battery voltage that might be expected. It would be easy and inexpensive to build, even for a noob.
In this circuit the '431 is operating like a comparator with internal voltage reference, so it's similar to the more complicated circuit posted later in that thread.
For your lower voltage, you want to use a single LED and a 1.25V version of the '431, and adjust the resistor values. I'll explain how to do that.
Here's how it works: when the reference pin (pin 1 in the circuit of post #1) is less than 1.25V, the cathode current will be low. The cathode current flows from the cathode (pin 3) to the anode (pin 2). When the cathode current is low, current will flow through R1 to the LED, lighting it up.
When the voltage at the ref pin is above 1.25V, the device will pull the cathode voltage down to 1.25V. This is not enough to light the LED.
Both the cathode current and the LED current are controlled by the series resistor (R1 in the schematic). You want to select R1 to limit the current to a safe level for both the LED and the 431. The maximum current for the '431 depends on which part you select, so be careful.
I like the TLVH431QLPR part for noob builds. It's in a package that's easy to work with, allows for a maximum current of 80 mA, and the whole project can be done for less than $1.00 in parts and can fit on a piece of perf board the size of your thumbnail. However, there are many other choices available if you know what you are looking for.
The minimum value of R1 is determined by the maximum LED current, or the maximum '431 current, whichever is less (probably the LED), or a lower current if you choose.
Lets say you have a red LED with 1.7V forward voltage and 20mA maximum current. The formula for R1 is R1 = (Vbatt-Vled)/current. Plugging in our numbers, we get (3.6V-1.7V)/0.020A = 95 ohms. This would be the minimum value, so we might select a value of 100 ohms. For a current of 5mA, we'd want 380 ohms. So you have a big range to play in, and there's no need for precision.
You will also need to select the value of R3 to get the cutoff voltage you want. The formula is R3 = 1.25*R2/(V-1.25), where V is the desired cutoff voltage. If you want 3.6V, then R3 = 1.25 * 4700 /(3.6-1.25) = 2500 ohms.
If you want a two-level indicator, just build two circuits and set the voltages different!
(More than) A few words on accuracy:
The TLVH431QLPR has an accuracy of 1.5%. If you select 1% resistors for R2 and R3, you'd expect an overall accuracy of about 2% (I won't go into how I got that). At 0.9V/cell, this gives you a range of 0.88 - 0.92V/cell where the LED comes on. I'd say that's probably good enough. If you want better than that, 0.5% accuracy on the '431 only costs another $0.20 or so, getting you to 1.5% overall. Changing to 0.1% resistors adds around $2.00 and gets you to 0.67% overall. If it were me, building one or two units, I'd be pretty tempted to throw a couple bucks at it and improve the accuracy. Or I might put a in multi-turn trimpot instead of more accurate parts, and make it as accurate as my ability to adjust the pot.