Why are still so many serious flashlight manufacturers cheating about the lumen flux?

[Barbarin]

Thanks to my job I do have access to many integrating spheres and photogoniometers. Real, accurate, expensive and calibrated ones. To give you an idea, during the last 24 months I bought more than ten complete labs for one of my customers.

Most of the time I use this equipments for "boring" light sources like tubes, downlights, streetlights... things like that. Sometimes I use them for my own flashlights, the ones I make as a hobbyist, but lets say that I never used them for flashlights I have bought. But lately I had some spare time, and I have found that many respected brands are still more than optimistic when it comes to declare their lumen output. Why?

I can understand the market pressure is high. I could say that no one of the well known reviewers have a serious integrating sphere, but they do their best with "lumen box" devices. A real integrating sphere with the software, sensor.. will range from 4000US$ to 10000US$. The smallest photogoniometer will be well over that price tag. And you will need some room at home, even a whole small room.

*This is the minimum equipment you need to test seriously a small light source.

What is the problem with "lumen boxes"? I think the problem is the calibration. If you think that 45 lm are 75 lm, all your measurements will be wrong. But reviewers, with their best will. get a light source and some numbers that they believe they real, BUT in fact as many companies are cheating is difficult to find a calibrated light source. A small 50 watt calibrated bulb will cost you at least 150 US$, and you will need a very accurate power supply to feed it with the exact current that is supposed to give you that lm and TºK.

Now lets take a look at one of my results. A well known, well respected flashlight brand. I would say among to top ten. And honestly, is a very good light, with excelente finish on everything and seems to be made really thinking about quality... BUT what happens when it comes to declared values? This one is supposed to be almost 900 lm on turbo mode. With a single 18650 and 2020 mA on the tailcap seems quite difficult to believe. Here is the result, of a 30 minutes test.

I'm not saying they are not greatly made lights, but it is just disappointing to see that people who is doing their very best on every aspect of the light will be not serious when it comes to declare the results. And probably they will keep on doing this as long as the reviewers won't be equipped with all that expensive stuff.

Regards,

Javier

PD: EVERY TEST HERE PUBLISHED IS DONE WITH FULLY CHARGED 18650 PANASONIC BATTERIES, AND STARTS AT T=0.

update, 31-III-2017. Two 10 minutes runtimes charts:

This one, another well known manufacturer, with quality made products, again fails when it comes to declared performance. They are announcing 1200 lm. Current at the tailcap on that "turbo mode" is a ridiculous 4415 mA. From a single 18650 .

On this one I will tell the name of the manufacturer and model, because what they claim is real. Is a Sunwayman C20C. A small light I own and usually is with me. Tailcap current is 1300 mA. Also turbo mode.

 

[vadimax]

I am sorry, and why not to disclose the make of the light? Kind of: "I possess wisdom, but I don't share with you".

 

[Barbarin]

I am sorry, and why not to disclose the make of the light? Kind of: "I possess wisdom, but I don't share with you".

I have a powerful reason, and it is because disclosing it won't be fair. I don't know if the unitI have is a faulty one, and before disclosing anything I should contact the customer service to have a proper explanation from them. Furthermore, that runtime is from a single product from a single manufacturer, but there are out there many which I do believe are making the same optimistics declarations. Again, it won't be fair to expose just one.

Ultimately, what I want to do is to get them to be more accurate about what they say. But anyway, as long as I keep on doing tests I will post them, with EVERY information. The information I'm handling now is just a few cases...

 

[Pöbel]

I'm sure lumens might be off by 10 or 20% in the real world. But with 900 vs 345, there has to be a problem somewhere. As the manufacturers report ANSI, this is too big of a difference. Maybe the battery, maybe not turbo mode..

 

[rayman]

Really interesting finding you have there. And I think you have a really valid point that it is hard to compare flashlights by different manufactures just by the specifiations because of the various views on the luminous flux. For my part I think I also have a wrong view regarding what I think a flashlight with a spefic amount of lumens will look like in reality.

I would really like to see some measurements from you with real flashlights.

 

[vadimax]

Well, some Fenix lights claim to be 900 lm at max. FD30/41 for example. Bull's eye?

P.S.: Why I constantly get my post rejected because of being too short (the line above is "too short")?

 

[bykfixer]

Thanks to the lumen wars and the propaganda machines I'm not the least bit surprised to see that is the case.

Part may be that the makers themselves rely on inferior equipment as well. Cost and all that jazz. Part is because people buy the stuff thinking "well brand BR549 would never lie about such a thing" and perhaps part is where said brands take their actual measurements from.

Not posting the brand was in good taste imo. Good thread.

 

[subwoofer]

... I could say that no one of the well known reviewers have a serious integrating sphere, but they do their best with "lumen box" devices....

What is the problem with "lumen boxes"? I think the problem is the calibration. If you think that 45 lm are 75 lm, all your measurements will be wrong. ...

Now lets take a look at one of my results. A well known, well respected flashlight brand. I would say among to top ten. And honestly, is a very good light, with excelente finish on everything and seems to be made really thinking about quality... BUT what happens when it comes to declared values? This one is supposed to be almost 900 lm on turbo mode. With a single 18650 and 2020 mA on the tailcap seems quite difficult to believe. Here is the result, of a 30 minutes test.

I'm not saying they are not greatly made lights, but it is just disappointing to see that people who is doing their very best on every aspect of the light will be not serious when it comes to declare the results. And probably they will keep on doing this as long as the reviewers won't be equipped with all that expensive stuff.

@Barbarin great thread.

You are certainly right, as almost all reviewers are amateur (in the real sense of the word) so are not going to invest in $1000s to make measurements that can be estimated to a reasonable accuracy for $10s.

In a review thread I've just been talking about exactly this subject due to other reviewers finding quite different results to mine.

The output testing is an interesting subject, and with so many home-made measuring methods we do get a lot of variation in quoted figures.

One of my concerns is in attempting to calibrate these devices. I have spent a lot of time measuring a large number of lights from different manufacturers on each available mode, and comparing it to the manufacturer's specifications. These were all plotted and the trend line added. This showed the exceptions to the trend and gave a good factor to use from the integrating sphere output voltage.

Maximum output is where there is the most variation, so these values should be used with caution.

...
Once I had a trend, and identified some lights with regulated consistent outputs, I also tried an addition method, where I start with only one light, then add in the next, and then a third to see if the response of my sphere was linear or not. It turns out that within the 5000lm capacity it is linear.

After finding these factors, I then set aside a group of reference lights with stable outputs, so that over time I can see if there is any drift in readings. I bring these out of storage to check the IS calibration is still correct. These are not used at all except for this calibration.

...

That said, despite my best efforts, my own readings might be completely wrong, but are consistent and have been a method of providing like for like comparison between manufacturers.

You did hit the nail on the head when talking about calibration, but this also needs to include the response of the sensor at different levels. If the response is not linear, but you calibrate to a fixed point, you can get more and more skew of the real reading.

Also what is very important when recording the output trace is to capture that vital 0s-30s at the end of which the ANSI reading is taken (or in very rare cases, more than 30s as some lights keep going up). If you miss off that first 30s, or use a cell which is not fully charged, you might actually miss the rated value completely.

Classic example here, so if in this case you turned the light on, fiddled with this, and fiddled with that, then set the measurement going, you would get an output reading of half the rated value.

Accurate measurement is a science to itself and is the reason the professional equipment is so expensive.

I frequently find under-performing lights, and they can vary up to 20% (or more if there is an actual fault). Interestingly, despite the manufacturer saying the output can vary by up to 20%, it is never 20% more, only 20% less. My figures are never 'adjusted' to please the manufacturers, and are reported exactly as found.

Many of the manufacturers I have tested for say they have their own certified and calibrated IS and check the output of their lights; some have sent me photos to prove it. When I find an under-performer, the same IS also proves another similar light from the same manufacturer is reaching its specified output, and all output levels are as expected, they still blame my test equipment for the low reading. I always batch test, so have many other equivalent results for comparison.

Until there is a central independent testing facility, manufacturers will prepare their own figures and publish them. Just look at car manufacturers' fuel consumption figures.

The more of us there are out there to challenge them the more likely they will keep it real.

 

[reppans]

I agree with you and this has been a pet peeve of mine for a while, and it's not only output, but also runtime specs - "in accordance with ANSI FL1...." is a JOKE (from certain/many manufacturers). That said, I've only found cheap Sipik clones and 3AAA hardware store lights to be half or less than spec'd output at maximum (before stepdowns). I'm personally a low-lumen efficiency guy and primarily test for the lower outputs and runtimes in terms of lumen-hours - there I regularly find massive exaggeration (by multiples) from certain 'respected' manufacturers. Many manufacturers are quite honest/conservative though - they get my repeat purchases.

I use a homemade lightbox (BLF special) that I feel is reasonably accurate, but as you say it's all in the calibration. ti-force is the only reviewer I've seen claim lumen accuracy although he's not reviewed a light in years.

For those of you who may not be aware, I have my own calibrated homemade integrating sphere for measuring lumen output of different lights. My sphere has been calibrated using lights that were measured in a professional lab sphere, so my lumen results are very accurate....

I calibrate to two of his reviews and on the step-down/lower modes which are less battery condition dependent - IME, max and moonlight tend to have the greatest sample-to-sample variation. My third trusted calibration triangulation point is HDS - Henry calibrates each light individually. On that calibration, I've found the quality US-domicile manufacturers - and not surprisingly, those also with the best warranties and customer service - to be the most truthful.

I understand the desire to not disclose manufacturers - without a relatively large collection of manufacturers, and even sample of lights from each manufacturer, then your informed comments might be unfairly punitive. How about commenting on the lumen scales our primary CPF reviewers - you must have several matching lights that will average out pretty well, we can then extrapolate your adjustment factor across that reviewer's data base. Alternatively, and even better, how about you offer to test a sample of our reviewers' own calibration lights so that everyone is on the same, and accurate, page. Consider it a donation to the community .

 

[lightfooted]

Barbarin,
While I appreciate your attempt to be fair, this post is virtually meaningless. It's a manufacturer YOU consider to be in the top ten...but would others think the same? Is it top ten by price? Volume of lights made? Variety of colors made?

By giving us no information you simply moved the burden of proof to yourself. You say you don't want to name them without checking with them because it could be a defective unit or something similar but why would you think that is the case?

Also, the graph you used doesn't show anything of value other than the drop in output and how that drop progressed over time. It doesn't show us where it started at nor where it finished. The numbers at the bottom are useful but perhaps you should just post those next time rather than the entire runtime graph as the drop over time was clearly not the concern. I feel it's similar to measuring a 2x4 by placing it next to a sheet of paper with marks along the edge. We can see the marks and how many make up the side we are measuring but we don't know how those marks translate to the real world or our own understanding of distance.

 

[Woods Walker]

I'm sure lumens might be off by 10 or 20% in the real world. But with 900 vs 345, there has to be a problem somewhere. As the manufacturers report ANSI, this is too big of a difference. Maybe the battery, maybe not turbo mode..

Agreed. That discrepancy is too large but without knowing the battery or light we really can't work out the issue.

 

[Woods Walker]

On a side note sometimes things work differently in the real world. I read tests with lights having fans blown on them etc etc etc to simulate......activity. Once tried to compare a light with a fan blown on it to say a 8 mile trail run in snow. The one outside didn't really get warm. Inside it got hot and no the fan didn't matter. Wasn't the same not even close. The output didn't seem to adjust to my eyes as the thermo regulation did not kick in. Can't carry a big blue ball and lab where I actually use the gear.



Though I find such information valuable and wish all manufactures printed run time charts I will leave that work to others. I would rather really use the stuff then tell people how it actually worked in the field if doing a review unless for some reason the output was noticeably lacking aka overstated run times or a vast discrepancy in output say 900 vs 345 but than again I wouldn't need a 1,000,000 worth of lab to know something was up. A simple ceiling bounce test compared to lights of know quantity will show that in about 2 seconds. That said I agree with the OP in that too many manufactures use industry standards and funky math to overstate things IMHO.

 

[ssanasisredna]

Many of the manufacturers I have tested for say they have their own certified and calibrated IS and check the output of their lights; some have sent me photos to prove it. When I find an under-performer, the same IS also proves another similar light from the same manufacturer is reaching its specified output, and all output levels are as expected, they still blame my test equipment for the low reading. I always batch test, so have many other equivalent results for comparison.

- Its a fair assumption that most do not know how to properly use their sphere even if calibrated
- At best only a small sampling of the MFG run would go into the sphere (if at all)
- They may .... may have an inline tester to verify output
- At best the LED bin is a 10% spread, though for cost, could be larger
- Most won't be controlling current better than 3-5% ... given a MFG spread of at least 13-15%, but if two LED bins, more like 23-25%

False reporting will not stop till someone gets enough data and files a class-action ... which could only be done against a large supplier with financial holdings in U.S. or Europe. Going to something like lighting where a certified lab is needed at least verifies the potential of the design (not the MFG result). Till then, thanks to Self Built and others for putting in the work.

 

[Barbarin]

Barbarin,
While I appreciate your attempt to be fair, this post is virtually meaningless. It's a manufacturer YOU consider to be in the top ten...but would others think the same? Is it top ten by price? Volume of lights made? Variety of colors made?

By giving us no information you simply moved the burden of proof to yourself. You say you don't want to name them without checking with them because it could be a defective unit or something similar but why would you think that is the case?

Also, the graph you used doesn't show anything of value other than the drop in output and how that drop progressed over time. It doesn't show us where it started at nor where it finished. The numbers at the bottom are useful but perhaps you should just post those next time rather than the entire runtime graph as the drop over time was clearly not the concern. I feel it's similar to measuring a 2x4 by placing it next to a sheet of paper with marks along the edge. We can see the marks and how many make up the side we are measuring but we don't know how those marks translate to the real world or our own understanding of distance.

Well, you are right, I just didn't posted some information which I thought, wrongly, that should be obvious. The test starts at T=0, with freshly charged batteries. I always use same batteries -Panasonic 18650 3400 mAh- and same charger, and always check the voltage, which is always 4,215 V with the charger I use.

Other than that, I do consider that it is one of the top ten brands because they are very well known among the community, the number of results you can find on Google is very similar to other brands no one will say that they are not among the best, and it's distribution is global. I have my reasons to place them on that position, and trust me, after so many years on the market I can say I know what I'm talking about. You think it is subjective? I don't, but of course feel free to keep on reading or not.

I just said I don't know if it is a defective unit, and that is the reason I won't disclose the brand. But after being responsible of manufacturing tens of thousands of lights every month, I can see that on the flashlight industry there are some claims that are difficult to believe, that is the reason I made some random test and will share the results with you.

Let me tell you a very simple case which happened to me. I bought some weeks ago a few Cree XQE, I made an MCPCB and drove them at 20% of their maximum current. Eff lm/watt= 65,30. Absolutely dissapointing. At the same time I bought some Refond 2835 double chip. Eff lm/watt= 179,70. I know positively that big brands use their flashlight manufacturing customers to get rid of their poorest LEDs, with too low CRI and low efficiency, units that would be discarded even for street lighting in many countries. In the real world is not easy to get those top notch LEDs, and even if you think you are buying them, you are not. So, let's say I do design and promote a flashlight with some performance in mind, based on numbers published by Cree, and then, when everything is done and the money invested, I can't get those LEDs -because the good ones have been bought by any other company which is making 2000 street lights a day, and every street light has 140 pcs of Cree XQE.... Do you think I do have any other chance but to cross my fingers? Things are not so clear.

For the rest of you, this one is being quite an interesting post. Tomorrow I will post two other reports from two well known manufacturers.

Regards,

Javier

 

[Barbarin]

- Its a fair assumption that most do not know how to properly use their sphere even if calibrated
- At best only a small sampling of the MFG run would go into the sphere (if at all)
- They may .... may have an inline tester to verify output
- At best the LED bin is a 10% spread, though for cost, could be larger
- Most won't be controlling current better than 3-5% ... given a MFG spread of at least 13-15%, but if two LED bins, more like 23-25%

False reporting will not stop till someone gets enough data and files a class-action ... which could only be done against a large supplier with financial holdings in U.S. or Europe. Going to something like lighting where a certified lab is needed at least verifies the potential of the design (not the MFG result). Till then, thanks to Self Built and others for putting in the work.

I just want to make very clear that I do really appreciate what the reviewers are doing. They put on this a lot of time and effort, and they give us very interesting comparative figures.

 

[Kitchen Panda]

Unless it's enforced by law, any numbers you see on a consumer product are marketing, not data. You may have seen that "5 horsepower" compressor that plugs into a a 120 volt outlet on a 16 gauge cord set. Some among us will remember the "Peak Music Power" advertising for home audio equipment, or the inflated horsepower advertising for car engines. One of my pet peeves is canned ham, where the advertised weight on the package includes the 20% of water that's added. You've got to take the stand mixer manufacturer to court to get him to stop putting "1.5 HP" on the mixer nameplate and "300 watts" on the real rating sticker underneath.

And in a world where the first thing most consumers do with a flashlight is look right into the reflector to see if it's working at all, the difference between 900 lumens and 300 lumens is unimportant. Almost no-one can tell the difference, it's not like putting the baloney on the kitchen scale to see how much of his thumb the butcher has sold you.

Bill

 

[Barbarin]

Unless it's enforced by law, any numbers you see on a consumer product are marketing, not data. You may have seen that "5 horsepower" compressor that plugs into a a 120 volt outlet on a 16 gauge cord set. Some among us will remember the "Peak Music Power" advertising for home audio equipment, or the inflated horsepower advertising for car engines. One of my pet peeves is canned ham, where the advertised weight on the package includes the 20% of water that's added. You've got to take the stand mixer manufacturer to court to get him to stop putting "1.5 HP" on the mixer nameplate and "300 watts" on the real rating sticker underneath.

And in a world where the first thing most consumers do with a flashlight is look right into the reflector to see if it's working at all, the difference between 900 lumens and 300 lumens is unimportant. Almost no-one can tell the difference, it's not like putting the baloney on the kitchen scale to see how much of his thumb the butcher has sold you.

Bill

I'm not familiar with other countries regulations, but for sure in Spain and in the rest of EEC, what you see in printed on the packaging, or what you can read on the instructions, has a contractual value, it is not just pure marketing. When it comes to lighting devices it is regulated even the size of the characters and the technical information to be given.

 

[jso902]

My old JL 75 watt was better than many 1000 watt speakers. As was my 1000watt Honda generator was better than many harbor freight generators. A lot of times, you get what you pay for

 

[Barbarin]

Just updated the original post with two new analysis.
Sadly I won't be able to post more for a while, as I'll be on trip.

Javier

 

[bykfixer]

Nice update. Again, tastefully done imo.