Simple low power regulation can be achived with a single NPN transistor and a voltage reference, referred to as an emitter follower circuit.
The voltage at the emitter of the transistor will be the same as the voltage applied to the base (minus about 0.7V)
The wikipedia entry for "Linear regulator" has a good example (scroll down to "simple series regulator") where R2 is your load, you can also replace the zener diode with another resistor if your precision does not need to be high.
I have built similar circuits, and have had good results up to a couple hundred miliamps, after that the regulation deteriorates, the transistor starts to complain,and it would be better to use a LM317 or other more complex (reliable) circuit