You can get "cheap" multimeters that RESOLVE to the 1mv level. That has absolutely nothing to do with the accuracy, i.e. whether that 4.23V is really 4.23 or 4.33. Even cheap multimeters claim accuracy below 1% or below. On a Li-Ion, 1% is 0.04V, so your meter could be off by that much. Some of the cheaper ones claim 0.5% or 0.02V .... and some even claim better, but unless you are paying good money, you are not going to get that. If you are just planning to use infrequently, try to find something with 0.5% accuracy, likely on a 20V scale. That should be fine.
One issue to be careful of with cheap meters is they tend to fail prematurely with bad readings...have had it happen to three now... I think likely internal corrosion on the switches. To that end, you are better off spending a little more money on an autoranging meter. There is less to go wrong.
Semiman