All of my primary CR123 cells get used in pairs, and I like to keep pairs together so that I have sets that are at the exact same level of charge. So, for these cells my marking system is 1A, 1B, 2A, 2B.... so that I can just use the A and B of each number together.
For rechargeable cells I simply number each (i.e. 1, 2, 3...).
I use a slightly different system, since some of my NiMH cells are used in groups of 2, some 3, and some 4. I put 2A on both cells in a group of 2, 2B on another group of 2, etc. -- groups of 3 are labeled 3A, 3B etc., so I can tell how many are in the group. I usually use this only to keep groups together, but on those occasions where I need to distinguish the individual cells in a group, they get an additional number, for example 3B-1, 3B-2, 3B-3 for the cells in the 3 cell group 3B.
A Sharpie has worked fine for me, although it does need sprucing up after some time if the cell gets a lot of handling. Markings from the "fine point" (actually the coarsest readily available tip) markers is more durable than from the finer ones. Alcohol removes the markings if needed.
c_c