How to convert Selfbuilt's Lightbox values to Lumens

[selfbuilt]

Everyone always asks me, when will I get myself a properly calibrated Integrating Sphere (IS) for lumen measures? The short answer is when someone pays for one for me.

But the real question is, can the relative output values of my home-made milk carton lightbox be converted to lumen estimates? First off, a few points about my lightbox.

I generally modeled my lightbox design on Quickbeam's (aka Doug Pribis) flashlightreviews.com site. Sadly, this excellent resource is no longer active (and the domain has been taken over by someone who has pirated Doug's original treasure-trove and added other unconfirmed and pirated material ). But at least the background info still seems to be there, untouched.

My lightbox differs in one important way - I've reversed the light and sensor placement, to facilitate runtimes. In my case, the flashlight enters the flat bottom of the milk carton, and the sensor is located on the side of the carton near the base. The bezel of the flashlight thus serves as its own baffle, preventing any light from shining directly on the sensor.

As Doug noted, this is hardly a perfect integrating sphere. But it doesn’t have to be – as long as you realize the results are simply relative output values, you can still draw meaningful comparisons between lights. But can you convert my milk-carton output values to estimated lumens by some sort of conversion factor?

To begin to answer that question, you would need calibrated IS data for the lights tested in the lightbox. Well, if I had a calibrated IS, I would be using it. But there are a number of other people – and manufacturers – who do have them. Although not as good as actually testing my specific samples in their ISs, can we tell something meaningful from lights we have in common?

Below is a graph showing how lights in my lightbox correlate to the reported IS values by three members here (MrGman, ti-force, and bigchelis) and three manufacturers (Fenix, 4Sevens and Novatac). Each data point represents one output mode of a given light we have in common. I have matched the reported batteries and time post-activation for the lights in question (if multiple time points were available, I picked the last one we both had in common). There are about 150 unique data points in the total set, representing over 40 lights.



If my lightbox were a calibrated IS, you would expect to see a perfect linear relationship as shown by the diagonal line. Obviously, it isn’t.

But the relationship does look somewhat linear, just not 1:1. Is it? Let’s improve the scale ranges and see:



Ok, clearly there is not a simple linear relationship of my lightbox to any of the IS results (although it isn't that far off, either). Still, this means that you cannot simply multiply my readings by a specific number to get a really good lumen estimate (i.e. the classic “y = mx + b” linear relationship with a slope and y-intercept wouldn’t hold here).

But even though the relationship between my box and the reported IS values is not linear, it is most certainly not random. Rather, it is consistently curvilinear. In fact, to my eye, it looks like a simple power relationship (i.e. y = a * x^b).

Before the modern age of computers, it was certainly a complex problem to try and fit non-linear data. But nowadays, you can do all sorts of comparison modeling of non-linear systems with statistical validation.

I’ve spent some time running analyzes of this data set, and I can't quite get one good power relationship that fits the whole range of 0.1 to 800 lumens perfectly. However, I have found two power fits that work well – one for < 20 on my lightbox relative output scale, and one for > 25. Here is how they look:





As you can see, these non-linear curves fit the data set very well. Correlation coefficients (r2) are reasonably good at 0.96 to 0.97. In fact, the fit is remarkably good considering we are talking about different light samples, run in 6 different ISs!

As a result, I think you can be fairly confident in converting my lightbox relative output values (ROVs) into estimated lumens using the following formulas:

For lightbox readings < 20 ROV, estimated Lumens = 0.56 * ROV^1.30

For lightbox readings > 25 ROV, estimate Lumens = 0.28 * ROV^1.48

For values in-between 20 and 25 ROV, I recommend averaging the two methods.

If you would rather not bother, not to worry – I plan to report the estimated lumen values in the summary tables of my reviews from now on. I will be doing this according the ANSI FL-1 standards (i.e. report estimated lumens at 3 mins into the run).

For those of you who have trouble visualizing non-linear data, below are some direct plots of my transformed lumen estimates against the original 6 different IS sources. If the conversions have worked well, the result should be perfectly linear.




My lumen estimates thus seem to correlate to the reported IS measures pretty linearly, wouldn’t you say?

Please note, there are a few caveats here. First off, I don’t have a lot of comparison data at the high-output range (i.e. >300 lumens). Thus, lumen estimates at the high end should be regarded with some degree of skepticism. As more data becomes available, I may need to revise these power relationships.

Another key point – a milk carton is not really a good integrating sphere! Not surprisingly, I've noticed in my lightbox that really strong throwers with narrow spillbeams typically report with lower values than similarly driven lights with wider spillbeams. Frankly, I’m surprised my values correlate so well with so many other sources. This may suggest that all lightboxes (including ISs) have some difficulty in integrating dedicated thrower lights. My point is that all attempts to compare overall output in lights with widely different beam patterns needs to be considered carefully (i.e. there may be systematic biases in all measures).

But at the end of the day, I think the analysis and correlation results tell a pretty compelling story - especially since they consist of the multiple output levels of over 40 lights taken from 6 different sources. You can thus feel fairly confident in converting my lightbox readings to something approximating lumens.

-----

P.S. A pet peeve of mine is the difference between precision and accuracy. Technically speaking, accuracy refers to the degree of conformity of a measure to the actual, true value. Precision is simply the degree of refinement with which a measure can be taken or stated.

In this case, I know my lightbox is precise to 3-4 significant figures (i.e. I can reliably get a value to that stated precision, on repeated testing). Of course, that says nothing about how accurate the result is! My "feel" for the range of lumen estimates out there tells me that we really shouldn't be reporting anything more than 2 sig figs for the output measures. Beyond that, I rather doubt any given IS is really all that accurate to the "true" lumen output.

Of course, scientific notation doesn't work well for text, so what I'm going to do is round my ~100-200 lumen estimates to the nearest half point past 2 sig figs (e.g. 145, 210, 160, 175, etc.), and keep the >200 lumen estimates to just 2 sig figs (e.g. 340, 470, 1100, etc.). That's really as far as I feel comfortable pushing my results.

---------------

UPDATE May 18, 2012: As you may have noticed, this original set of lightbox-lumen conversion estimates was based on lights <800 lumens or so. You might wonder, how do more recent higher output lights compare?

One issue here is that most of these high-power lights are actually too big to fit into my lightbox. Many of them have large multi-emitter heads, or massive reflectors for more throw.

So I have had to find an alternative to sticking them in my lightbox. Since the original goal of a milk carton lightbox was to simulate a ceiling bounce measure in a small room/closet, that is exactly what I have gone back to for output measures in high-powered lights.

I have performed a comparable analysis of ceiling bounce-to-lumen correlations for a number of high output lights, in much the same way as I have done here for lightbox-to-lumen. The results have allowed me to fairly accurately provide estimated lumens for high output lights.
In fact, I am actually surprised at how well this conversion continues to be consistent for new lights with properly-tested lumen measures (i.e., my method continues to appear to be accurate, within a low margin of error).

To keep the presentation of results simple, I am starting to present graphs of these high-output lights directly in estimated lumens. But I can easily work back from these ceiling bounce-to-lumen conversions to estimated lightbox relative values. As you will notice in all my reviews, I have always done ceiling bounce measures of outputs for all my lights. This gives me a large database to correlate lightbox readings to my ceiling bounce setup. So, I can easily represent my high-output lights on the same relative lightbox output scale, by converting back from the lumen estimates. This is very convenient when wanting to compare the output/runtime performance to older lights.

As always, I continue to monitor and verify my conversion methods, as well as the calibration level of my equipement (i.e., relative to my initial arbitrary reference standard). So far, things continue to remain on track with periodic monitoring and adjustment.

[selfbuilt]

Written by recDNA on 11-11-2010 09:21 AM GMT

I know that if the values are plugged into excel it will ceate a
function that defines the curve and measures the degree of accuracy that
the function defines the values. It would be a more complex function
but might define the values throughout without the need for separate
equations. Have you tried it?

Written by selfbuilt on 11-11-2010 09:47 AM GMT

I know that if the values are plugged into
excel it will ceate a function that defines the curve and measures the
degree of accuracy that the function defines the values. It would be a
more complex function but might define the values throughout without the
need for separate equations. Have you tried it?

Yes, Excel is quite good for that, but a bit limited compared to
dedicated non-linear curve fitting programs. FYI, although it's an
oldie now, I still use GraphPad Prizm for the power curve fitting you
see above. In addition to a lot more curve-fitting options, it also has
an extensive array of choices on how to determined the degree of
accuracy and fit (although the default Excel method is quite good, and I
what I prefer anyway). Yes, Excel is quite good for that, but a bit limited compared to
dedicated non-linear curve fitting programs. FYI, although it's an
oldie now, I still use GraphPad Prizm for the power curve fitting you
see above. In addition to a lot more curve-fitting options, it also has
an extensive array of choices on how to determined the degree of
accuracy and fit (although the default Excel method is quite good, and I
what I prefer anyway).

Using either program, I can get a power relationship or a multi-order
polynomial that fits the overall data with a higher correlation
coefficient (i.e. 0.98). But a quick visible inspection of the result
reveals the limitations:

• One common power curve underestimates at the high end, and overestimates at the low end of lumen outputs
• A 2nd or 3rd order polynomial drops to zero by ~2 lumens, so really can't be used at the low end <5 lumens or so.
• A 4th order overestimates at the low end, again <5 lumens or so.

Simply put, no one curve fits the whole range well. Of the various
methods, the multi-order polynomials actually seem the best for values
>5 lumens or so, but it would be difficult for anyone else to
reproduce (i.e. even for a 2nd order, there are 3 constants that need at
least 5 or 6 sig figs to get a good graph). And the polynomials are
all useless <5 lumens or so - none of them fit that data well.

I could do a power relationship for <5 lumens, and 2nd or 4th order
polynomial for >5 lumens, but that doesn't seem to do much better
than the "simpler" two power relationships shown above.

Written by MrGman on 11-11-2010 03:05 PM GMT

The math is definitely good and persuasive. Having computers with
software that does these formulae and finds the best fit is pretty
handy. Cannot argue with the findings.

However, the logic and the work is still questionable. Its still just a
rectangular box type milk carton. It took several years worth of other
people's data to find enough plot points to fit and make any sense out
of your data in terms of lumens. The ability to read the difference
between tight beam throwers versus flood lights is still highly
questionable as you yourself state above.

Here's the problem. My measurements taken with a real professional grade
Integration sphere system that I previously had access to at work were
accurate. My readings of various lights on 2 different home made sphere
systems have a certain degree of error which I readily admit, after all
they are not true IS systems with the right type of paint, and the right
type of baffle and sensor that gets calibrated every year with the
sphere back at the factory that made it all. So my home made readings
have what I will call first order errors, because they are 1 step away
from a professional and accurate system.

I helped bigChelis and then Ti-Force create sphere systems. They used
some of my lights for measurements in their systems and we came up with
best fit numbers. But it is really two steps away from taking readings
from the calibrated system at work. Since I had sold or traded away some
of my lights, plus they get older with run time on them. Can't say that
my reference sources are 100% but we do the best we can. We will call
Ti-Force's and big C's sysem as having second order errors.

The vendors like Fenix are now using the ANSI standard measurement which
is waiting for the light to be on for 3 full minutes before using the
numbers for the output. I didn't do that for all my lights. Don't know
because you didn't make it clear if you did that for your readings and
only used those readings from myself, Ti-Force or bigC that were at the 3
minute mark or not. But those readings then call into question battery
quality and cooling issues. I used to live in San Jose and taken
readings in my garage even though I waited till it cooled down when I
could sometimes were closer to 90F at the turn on. Point is there are
variations there.

So you collect readings with various first and second order errors and
deviations and you find a fit to the entire group and plot a line. But
you don't really say what the conversion accuracy of your new
calculation formula is for any single given light to that of what it
really is or really should be.

So for any new light that you would test that you don't already have
some one else's published data on, but simply using your formula, what
would you expect the accuracy to be? I would say anything better than
+/- 15% would be unrealistic, but you tell us.

Now the other issue that I have heartburn is, your one of these guys who
wants to do a lot of pseudoscientific testing (like me) but you're too
cheap to spend the money for even a 12 inch styrofoam sphere. Which cost
typically $28 max. You already have a light meter, and you buy and sell
and trade lights. You spend lots of time to do the calculations but not
invest in a more appropriate tool to do the job. No one paid me to go
out and by meters and spheres for home testing when I started this I
just did it for the pursuit of knowledge. No one paid Ti-Force, from
what he has told me. He did it for the pursuit of knowledge like me. I
know that bigChelis was out of work for more than half a year at the
time when he went out and bought his own light meter and first sphere
because he wanted to pursue knowledge as well. Yeah he had help paying
for the second 2 foot diameter sphere, but the amount of work he put
into all of his readings more than made up for it.

You take all of that work and turn around and say I can use it to
justify being cheap and show how I can get "lumens" readings from a milk
box. You don't publish any firm calculation to show what true accuracy
you can predict for any one given light, only that your model fits the
collective total of data with data points spread out on both sides of
your line. So if some one asked you to measure some new light that I
haven't measured with some tight beam optic on it, would you really have
any serious clue as to how good your readings are? Your work is still
what I would call third order error.

So I am not impressed. It is after all a milk box. the wrong shape tool
for the job. If you were a college student looking for a job at a
research firm and you came to me to potentially hire you and showed me
this as an example of your work, You would still be out walking the
streets, because you missed the bigger picture for not trying to use the
right tool and limit your errors in the first place for the price of a
couple movie tickets. I may get boo'ed for this but that's the way I see
it. to me this is a good example of an exercise in futility.

Written by zehnmm on 11-13-2010 12:00 PM GMT

Selfbuilt: Well done! I am a consultant statistician and it is
great to see statistical applications put to use like you have done.
Keep up the good work!

Regards,

Steve

Written by LED_Thrift on 11-13-2010 12:51 PM GMT

Cheers for Selfbuilt. This is great and very valuable work. You
are one of the very few, if not the only, CPF member who IMHO, ranks up
there with Quickbeam for providing useful, accurate and unbiased
information to our community. Thanks.

Written by recDNA on 11-13-2010 02:56 PM GMT

Yes, Excel is quite good for that, but a bit
limited compared to dedicated non-linear curve fitting programs. FYI,
although it's an oldie now, I still use GraphPad Prizm for the power
curve fitting you see above. In addition to a lot more curve-fitting
options, it also has an extensive array of choices on how to determined
the degree of accuracy and fit (although the default Excel method is
quite good, and I what I prefer anyway).

Using either program, I can get a power relationship or a multi-order
polynomial that fits the overall data with a higher correlation
coefficient (i.e. 0.98). But a quick visible inspection of the result
reveals the limitations:

• One common power curve underestimates at the high end, and overestimates at the low end of lumen outputs
• A 2nd or 3rd order polynomial drops to zero by ~2 lumens, so really can't be used at the low end <5 lumens or so.
• A 4th order overestimates at the low end, again <5 lumens or so.

Simply put, no one curve fits the whole range well. Of the various
methods, the multi-order polynomials actually seem the best for values
>5 lumens or so, but it would be difficult for anyone else to
reproduce (i.e. even for a 2nd order, there are 3 constants that need at
least 5 or 6 sig figs to get a good graph). And the polynomials are
all useless <5 lumens or so - none of them fit that data well.

I could do a power relationship for <5 lumens, and 2nd or 4th order
polynomial for >5 lumens, but that doesn't seem to do much better
than the "simpler" two power relationships shown above.

Excellent explanation. Thanks. When I think about it it makes
perfect sense that no one formula would express the values accurately
throughout the entire range. I use Excel for this purpose often and I
should think more about breaking up the values into smaller ranges. I
may get better predictive expressions with higher degrees of
correlation. I'm not dealing with electronics however some of the same
issues could be involved. The speed of chemical reactions also changes
as percentages of products and reactants change as well as the
availability of the enzymes involved. Excellent explanation. Thanks. When I think about it it makes
perfect sense that no one formula would express the values accurately
throughout the entire range. I use Excel for this purpose often and I
should think more about breaking up the values into smaller ranges. I
may get better predictive expressions with higher degrees of
correlation. I'm not dealing with electronics however some of the same
issues could be involved. The speed of chemical reactions also changes
as percentages of products and reactants change as well as the
availability of the enzymes involved.

[selfbuilt]

The main post has been updated with the final text.

The thread discussions have been fully restored from the search engine cache data (thank you tandem!).

Please carry on!

[RedForest UK]

Interesting, so roughly 195 OTF for the SC51 on an eneloop at startup..

[LeifUK]

Firstly your measurements given in reviews are very useful, even if not precise, because for example they tell us how the brightness varies throughout a run, and they give us a rough comparison between lights. Does anyone know what variation exists in the output of production lights such as Quarks? Secondly, is there a good degree of agreement between the IS results that you use as reference points? Clearly if they disagree with each other, that could indicate variation in the lights, or errors in the measurement techniques.

[mrlysle]

Selfbuilt, you're always the man to come too! Thanks for taking your time to explain these things, and to me your reviews are always impressive and concise. I been spending a little time on your slide rule site! It's fun! Pride yourself on being a very knowledgeable individual! Thanks again.

[selfbuilt]

Does anyone know what variation exists in the output of production lights such as Quarks?

An excellent question. Most of my tests are done on n=1 samples, so I have little way to gauge natural variation between samples. In the few cases where I have test 2 or more samples of the same light, I have seen variation up to level predicted by standard output binning (i.e. ~7% range for a Cree output bin). Of course, circuitry adds a few variables as well, but I would expect most samples manufactured at the same time should be within 10% of each other. As time goes by, that variation to early batch increases, as changes are made to the line (i.e. circuit tweaks, new emitter batches, etc).

Secondly, is there a good degree of agreement between the IS results that you use as reference points? Clearly if they disagree with each other, that could indicate variation in the lights, or errors in the measurement techniques.

Yes, you can see it somewhat in the graphs. Typically, the various sources consistently deviate from the average line (i.e. consistently above or below, although this is obscured somewhat by the variation between lights). On the whole, this suggests that there are systematic differences between the reporting sources. But as I have no way to ascertain which is the "truest" measurement source, the overall average of all of them provides a good overall estimate.

I been spending a little time on your slide rule site! It's fun!

A past hobby, before LED flashlights came into my life. Glad you enjoyed the site.

[selfbuilt]

Ah, I see someone has thoughtfully stickied this old thread that I recovered. Thanks mysterious mod!

Just to clarify, my intention here was never to recommend someone make a milk carton lightbox. But I have a large wealth of data collected over the years (as presented in all my reviews), and it is a worthwhile exercise to see if approximate lumen estimates can be extracted from all that data.

All my results were obtained and presented in a consistent manner. I continually monitor my lighbox readings (and tweak as necessary) to make sure they stay "calibrated" to the initial reported values. This internal calibration means a given ROV (relative output value) from my lightbox of 3 years ago should equal the same value published yesterday. You can therefore apply the same adjustment factor, with confidence that the ROVs have remained consistent.

So, you can look up the ROV values from the tables and graphs of my older reviews, and apply these formulas to give you a rough lumen estimate: (note: the "^" symbol means exponent, so it is ROV taken "to the power of" the number that follows)

For lightbox readings < 20 ROV, estimated Lumens = 0.56 * ROV^1.30

For lightbox readings > 25 ROV, estimate Lumens = 0.28 * ROV^1.48

For ROVs between 20 and 25, I suggest an average of the two results.

If you don't have a calculator that does exponents, simply cut and paste the appropriate formula above into the googe search bar (and replace "ROV" with the actual ROV value from my review), and google will return the correct result.

Again, a milk carton is not a true integrating sphere. Also, I make no claim as to the accuracy of the reported lumen estimate from the various integrating spheres on which this analysis is based. And of course there is a lot of variability between flashlight samples and testing regimens. But this analysis shows that my simple lightbox relative output values can at least be converted into something similar to the reported integrating spheres (at least for lights within the above measured output ranges). If you want to set out to estimate your own lumens, I suggest you check the various threads by those who have made integrating spheres.

[csshih]

Hey Selfbuilt - I actually recommend getting your hands on a HDS light - their lights are calibrated pretty well.. the last HDS 140 I got tested in BigC's sphere did 140.2 lumens, IIRC

[Bolster]

But at the end of the day, I think the analysis and correlation results tell a pretty compelling story - especially since they consist of the multiple output levels of over 40 lights taken from 6 different sources. You can thus feel fairly confident in converting my lightbox readings to something approximating lumens.

I agree. The r-squared of .96 and .97 say it all. There's a very good fit between Selfbuilt's estimates and manufacturers' reported lumens.

I have been busy re-inventing Selfbuilt's wheel over here with the Lumen Toob made of $7 of PVC elbows. I was not aware of this thread, which has a much superior dataset. Based on Selbuilt's r-squared (and mine, which is remarkably similar), I'm also of the opinion that the layperson can make reasonable lumen estimates without an integrated sphere. In the thread ref'd above, I was told such an exercise was "unprofessional and negligent," but the high r-squared disagrees. Build some reasonable sort of diffusion and collection mechanism (carton, PVC tube in my case), use a decent meter (a Gossen incident photo meter in my case) and you can get close. On my current dataset I'm estimating lumens within 10% on average of what various manufacturers are claiming.

[greenorange]

YES! I finally discovered this internet page! I??£¤ve been seeking just for this post for so extended!!

[selfbuilt]

UPDATE May 18, 2012: As you may have noticed, this original set of lightbox-lumen conversion estimates was based on lights <800 lumens or so. You might wonder, how do more recent higher output lights compare?

One issue here is that most of these high-power lights are actually too big to fit into my lightbox. Many of them have large multi-emitter heads, or massive reflectors for more throw.

So I have had to find an alternative to sticking them in my lightbox. Since the original goal of a milk carton lightbox was to simulate a ceiling bounce measure in a small room/closet, that is exactly what I have gone back to for output measures in high-powered lights.

I have performed a comparable analysis of ceiling bounce-to-lumen correlations for a number of high output lights, in much the same way as I have done here for lightbox-to-lumen. The results have allowed me to fairly accurately provide estimated lumens for high output lights. In fact, I am actually surprised at how well this conversion continues to be consistent for new lights with properly-tested lumen measures (i.e., my method continues to appear to be accurate, within a low margin of error).

To keep the presentation of results simple, I am starting to present graphs of these high-output lights directly in estimated lumens. But I can easily work back from these ceiling bounce-to-lumen conversions to estimated lightbox relative values. As you will notice in all my reviews, I have always done ceiling bounce measures of outputs for all my lights. This gives me a large database to correlate lightbox readings to my ceiling bounce setup. So, I can easily represent my high-output lights on the same relative lightbox output scale, by converting back from the lumen estimates. This is very convenient when wanting to compare the output/runtime performance to older lights.

As always, I continue to monitor and verify my conversion methods, as well as the calibration level of my equipement (i.e., relative to my initial arbitrary reference standard). So far, things continue to remain on track with periodic monitoring and adjustment.

[aoyiteled]

Often the practical conclusion is generally accurate, so as to ensure the quality of products, I am in favor of this sentence:As always, I continue to monitor and verify my conversion methods, as well as the calibration level of my equipement.

[selfbuilt]

For those who are interested, I've just announced a revision to how I do my beam intensity/distance measures in my reviews. Full discussion here:

Revised Selfbuilt beam intensity measures: new NIST-calibrated Extech EA31 lux-meter

I continue to use the lumen estimate calibration described in this thread for the lightbox data.

[Dplight21]

It is good thread. After reading this thread I know a lesson how to convert Selfbuilt's Lightbox values to Lumens.