Too many 5 star lights?

[Traglite]

Flashlight reviews does a great job, but with all the new cree lights popping up, are we seeing too may 5 star ratings? Maybe the scale needs to change a bit. I'd light to see multiple ratings for one light with categories like throw, beam quality, build/machining, runtime, durability, value, etc. All these ratings would then average out for the final score. I'd like to see decimal points too, not 1/2 stars. Just my thoughts.

 

[bridgman]

It probably is time to reset the bar and adjust all the ratings down a bit to reflect the advances over the last few years, although I don't even want to *think* about how much work it would be to do that properly.

In the specific case of FlashlightReviews, I think Quickbeam is trying real hard to "retire" from the reviewing business for a while (and probably would if not for all the cool new lights coming out ) so it seems unlikely that you'll see FR re-design the rating system any time soon.

 

[Lobo]

I initially thought so too when looking at the front page at FLR. But if you take in to consideration that he is semiretired, and only reviews the most popular(best) lights, it's not so surprising that all of them got so high grades, since all of them have been great so far! Maybe he should revise the scale in the future, but I think it's too early now.

 

[prof]

Quickbeam did revise and rerate a bunch of lights just a few months ago, before he "semi-retired." One of the things that makes FLR a great site is that he does update the reviews occasionally.

When Quickbeam announced his semi-retirement, several people volunteered to either help out or start a new review page. However the big benefit of FLR (to me, anyway) is that all the opinions come from one person--makes it easier to compare. I don't know what happened to the volunteers, but you could try to get a group of people together and start your own review page. Seems like there are a few people working on ideas similar to yours, but they've all run into the same thing that Quickbeam did--lack of time.

Also, if my memory is not too addled from coffee and lack of sleep, it seems like someone commented on the number of 4 and 5-star lights a year or so ago. Quickbeam posted a response, and basically explained that he was focusing on high-end lights due to a lack of time and because most of his readers (us) were more interested in good lights than in garbage lights. Thus, lots of high ratings.

You should probably search for and read his post, as I'm trying to remember a post from a year or so ago and would hate to misquote him.

If you're really interested, try emailing quickbeam--I believe he posts his address on FLR. He's always been very helpful and friendly when I've emailed him. Seems like a good guy.

 

[Lee1959]

Quickbeam has kept his criteria and ratings pretty constant and like has been posted is semi-retired. His rating is an indicator of how he views the lights by a specific set of guages, and not a direct ranking against all other lights. It is not intended, I believe, to tell you that light A ranks 2 spots higher than light B, simply that they are both good, or bad lights.

I personally like how it is set up now, as it is simple and easy, similar to A, B, C, D, E in school because it is not a competition of lights simply a direct "grade". You can tell at a glance and do not have to go and see ok, what constiutes a ranking of 7.85 out of 10?

He has a direct comparison of lights on a separate page, which may be more what you are wanting?

 

[nerdgineer]

For me, FLR's payoff was the availability of a set of common, standardized quantative measurements for each light, especially throw, output, beam pattern, and run time. These (along with subjective comments about quality and feel) let me make my own "star" judgements about each light (normalized against current prices) so his old star system was just fine.

I'll miss additional QB reviews because I'd lose the common framework of measurements, but others will step in, and QB more than deserves his R&R&R.

 

[aceo07]

If he is only reviewing what is expected to be the best lights, then there's a good chance they'll get high marks.

I hope he gets a NovaTac and Surefire in the future to review. More 5 stars hopefully.

 

[Lobo]

For me, FLR's payoff was the availability of a set of common, standardized quantative measurements for each light, especially throw, output, beam pattern, and run time. These (along with subjective comments about quality and feel) let me make my own "star" judgements about each light (normalized against current prices) so his old star system was just fine.

Couldnt agree more with you. The strength of his reviews are that they followed a standard, which makes it very easy to compare different lights, in almost all aspects.

 

[Outdoors Fanatic]

Yeah, I'd like to see new categories and maybe a 10 stars rating. Fit/finish/easy-of-operation/reliability should be as important as brightness/features/runtime currently is to FR.

 

[lightbug]

When people read the reviews, they have to factor in the TIME the review was done upon. Obvious you can't compare something made in 2004 to something made in 2007. A 5 star rating in year 2004 probably is only worth 4 stars rating in 2006 and 3.5 stars in year 2007.

When you factor in the time effects, I think Quckbeam's ratings are very fair.

 

[Illum]

I personally would preferr to see two separate classification systems

one luxeon based the other cree based...from this standpoint all premium quality cree lights, if reviewed based on the luxeon classification then the ratings would be useless

though it may be too much to ask...but since CREE lights mark a beginning of the flashlight evolution, it may be ideal...but thats just me

 

[LED_Thrift]

When people read the reviews, they have to factor in the TIME the review was done upon. Obvious you can't compare something made in 2004 to something made in 2007. A 5 star rating in year 2004 probably is only worth 4 stars rating in 2006 and 3.5 stars in year 2007.

When you factor in the time effects, I think Quckbeam's ratings are very fair.

GREAT point!

 

[cratz2]

I don't know... I understand what some of you guys are saying, but FLR is more than just a number review. There are subjective aspects to many lights and he posts runtimes, throw, and total output in addition to the pure number. The review number is a subjective rating provided in addition to the verbage of the review. I think they are meant to be compared to similar lights... 5mm vs 5mm lights, Lux III vs Lux III lights, Cree/SSC vs Cree/SSC lights.

For example, take the reflectored Inova X1 and the M1 Hunter. 4 1/2 stars for the X1, 5 stars for the M1. Out of all the available lights, surely no one would end up with these as the two finalists as the perfect light for a given use. But compared to the Dorcy 1xAAA and the Gerber Infinity Ultra, the X1 gets 4 1/2 stars. The M1 on the other hand, is being compared to the Huntlight FT01XSE Cree or maybe the Fenix L2DCE.

 

[Vikas Sontakke]

Just like how some volume knob goes to "11", QB should not hesitate to accord higher than 5 ratings, if warranted. As the technology marches forward, you would start seeing 6 star and 7 star lights.

- Vikas

 

[who]

IMHO, we should take prices into consideration. For example, a $50 5* Fenix is not equal in absolute terms to a 5* Surefire, but it deserves a 5* rating anyway.

 

[adirondackdestroyer]

When people read the reviews, they have to factor in the TIME the review was done upon. Obvious you can't compare something made in 2004 to something made in 2007. A 5 star rating in year 2004 probably is only worth 4 stars rating in 2006 and 3.5 stars in year 2007.

When you factor in the time effects, I think Quckbeam's ratings are very fair.

Great point! Time absolutely has to be factored in. Mostly because only two years ago a 60 lumen light was considered exceptional if it ran for 1 hour, now their are 60 lumen lights that run for 5 hours, and 100+ lumen lights that have 1 hour runtime on a single 123 cell. It is really the progression of technology that makes this change possible.

I also think that Cree lights should be compared to other Cree lights and not really Luxeon. Mainly because their is no comparison between the later two.

 

[nerdgineer]

...For example, a $50 5* Fenix is not equal in absolute terms to a 5* Surefire, but it deserves a 5* rating anyway.

But FLR's numeric data does allow you to compare them directly in exactly absolute terms. For example, the 5* Fenix L1P-CE has a max output of 7850 units vs. the 5* Surefire's U2's max output of 5500 (i.e. 42% MORE than U2) and that's a fact - at least for the FLR samples. For other considerations, you have to read up and decide for yourself, but having all those measurements side by side is soooo useful.

It would have been really interesting to see all those 8W and 14W Kaidomain and DX Lux and Cree lights compared with FLR gold standard measurements; but as I said, QB deserves his retirement and our gratitude for the huge insights he has given us already.

 

[Art Vandelay]

There are lots of good points at this link. http://games.slashdot.org/article.pl?sid=06/08/08/1316204 It's about video game reviews, but many of the same issues apply. Is it "fair" to compare the games (lights) of a few years ago to the games (lights) of today. Games (lights) that were cutting edge at the time got top scores, now those games (lights) still "have" those ratings. Will Pong have 5 stars forever?

I read the whole review to get the information I want. Some things will be more important to be than to the reviewer, and somethings will be less important to me than the reviewer. Maybe I want a light that will fit in my front pants pocket, but the reviewer does not care about that. Maybe it's a huge deal to the reviewer that a light have HIII anodizing, but I don't care that much about it.

Some flashlight makers are using the most current technology, and others are resting on their laurels. The problem is not with the new leaders, but with the old companies who are dragging their heals.

 

[BentHeadTX]

You have to take cost into consideration

If some company came out with a variable output single current regulated Cree AAA light with HA-III, double o-ring seals, aluminum OP reflector, gold plated brass contacts, potted head with GID O-rings under the lens and it costs $30. Alas, it does not take 3.7v li-ion cells and a reverse clickie.

Say Surefire comes out with the same thing but added 3.7v capability, a regular clickie and a crenelated bezel and it cost $150. How would you rank the two lights? Would the first light get 4 stars and the Surefire get 5?

FR does it right and you must take cost into consideration. My L1D CE does not put out the lumens of a FF3 but it is soooo close. The FF3 beats it for quality and beam quality. How to rank? Cost, UI, size and battery required make them different classes.

I can see FR as a usability rating that does take cost into consideration. Many folks will take a 4.5 star Surefire over a 5 star Fenix any day because they know what class of light they both fall into.

A section at work ordered Surefire L1's but the hazmat batteries caused a problem. I recommended Lumapower M3 transformer lights so they can use their existing stock of CR123A batteries if they want, they can run 1AA if they want or 2AA since some folks prefer a larger form factor. Ever try to get 8 people to agree on flashlights? Kudos to those crazy enough to actually put their thoughts into writing for the world to judge. :thumbsup:

 

[TORCH_BOY]

5 stars, I have to admit just about all the new Cree lights are really good,
you don't see many pea green emitters any more, More light for less power