LS4 Software/Firmware Concerns

Peter,

By now you've probably realized that the software/firmware in the LS4 is a significant investment. You probably consider Arc a flashlight company, but knowingly or unknowingly you've also become a software company...

I've made (a lot of) money, reverse engineering business critical embedded software for companies who didn't realize that they were in the software business until too late. Also led software and system development teams for 15 years. I've got a couple of concerns that you might want to think about...

1) If your current software guy gets hit by a bus tomorrow morning, can you (somebody other than the software guy) find, build and install the most recent stable version?

2) Can you find, build and install the next to last stable version? The current test / unstable version?

3) Who, other than your current software guy can work on the code?

If you (as a company/development organization) aren't already using some form of SW configuration management / source control, I would recommend looking into CVS, RCS or SCCS (all free, listed in decreasing capability/complexity).

If you are not already doing code walkthroughs or inspections:
Once you believe you have a "final" "stable" version, I would recommend that you invest the time to do a top-to-bottom code walk through (yes, it will take DAYS) with two or three software people you trust. This will serve two purposes.

#1 Any defects you catch in the walk through will not make it to the field, saving you BUNCHES of money...

#2 You will then have 3 or 4 people familiar with the code (the software guy plus the reviewers you trust) rather than just the software guy.

If, by looking at how the LS4 development has been impacted by the software, you realize how important the software is, it makes sense to do some risk avoidance/mitigation on that asset.

Experience dictates you will encounter one of four responses to discussing this with the software guy.

1) He goes ballistic, claiming that you don't trust him and are trying to micro-manage him. (Ya got trouble long term...)

2) He agrees that these are important and will get to them "real soon" or "after the code is done." (Standard answer)

3) Sees them as really good ideas and spends the day or two needed to get them up and running RIGHT NOW. (He's a keeper.)

4) Says "well... duh... Been doing that all along, here are the stats, wonder why it took you so long to ask." (a true diamond).

My $0.02US.

Free advice, take it or leave it as you see fit.

PluckyPleco

For 2 person teams and smaller, you should also consider Perforce (free) for revision control. Perforce is an excellent and extremely simple to use revision control system.

I have to agree wholeheartedly with PluckyPleco on some steps to take for validation and I would go one step further.

Place the complete buildable environment onto a CD (plus one backup). Copy the CD onto a clean machine and validate you can build from scratch.

Then place BOTH of those CDs in a Safe Deposit box (offsite). Do this rigorously and at regular intervals.

jbk

Just my $0.02 - I think you are taking this matter too seriously. If ARC had a Gov't contract for 2 million pieces, yes, he could afford the software development you describe. But, I hold in my hand an ARC LS with a serial # of 883. ARC has a small market now, probably even a smaller market with the higher priced LS4. I can live with a bug or two in the software, as can most people here. These are flashlights! We are losing touch with reality - smaller, brighter, microprocessor controlled, etc. He's not building aerospace guidance systems, just nice lights to walk the dog!
Rob
P.S. I don't mean to offend anyone, just my opinion.

[ QUOTE ]
But, I hold in my hand an ARC LS with a serial # of 883. ARC has a small market now, probably even a smaller market with the higher priced LS4.

[/ QUOTE ]
It doesn't really matter how big the market is. The reality ARC is making mission critical software. Can you live with a bug that renders your light unusable at a critical time (or any time)? If this software fails to meet reqirements over time the LS4 (and possibly ARC) will fail.

There is no free lunch when it comes to software construction, you pay now or you pay later. Either way, you pay.

-lc
PS: Not offended BTW <s>.

i'm certain that arc will do everything possible to make sure their software is bugproof, and i'm confident that i could rely on the ls4/5 if the situation required /ubbthreads/images/graemlins/smile.gif

[ QUOTE ]
Rothrandir said:
i'm certain that arc will do everything possible to make sure their software is bugproof, and i'm confident that i could rely on the ls4/5 if the situation required /ubbthreads/images/graemlins/smile.gif

[/ QUOTE ]

I don´t think the actions proposed regarding revision control and such is mostly about having a bug-free software to start with (dude - never trust a rev 1.0 anyway) - while certainly I fully agree with those proposals as Peter is going a long way to have a great product and the walk-throughs and such would certainly help this goal - but IMO the most important parts were about protecting PG and ARC from any accidents - being the single SW guy having his PC burn down or whatever else might happen to him - remember rule number one in adventure games - safe often safe early /ubbthreads/images/graemlins/grin.gif

Good thread

Klaus

PluckyPleco,
You actually pointed out right as the LS4 involves some software. If Peter want to look in to such practice he would have to go through contracts and agreements like non disclosure and etc for the contractors and beta testers.

If you look at things like an hightech portable music player or mobile phone companies it's actually a mix of both hardware and software. IMHO, LS4 is mainly hardware. It will still function without the software but lose all the cool functionality. I think it ends up with weighting. If the software weights more than the hardware then it's worth looking in to it. Compared to a mobile phone firmware, I think LS4 is a small scale project, Peter could also have sorted out all these problems in the beginning.

I am quite new in this field so don't take my words too seriously.

Vince.

Robk,

I'm a software guy with about 15 years experience, and I agree whole-heartedly with PluckyPleco.

The big expense he's recommending is in time, not in the purchase of version control software (all suggestions were for freeware). PP's also suggesting that all the knowledge of the code not reside in one person's head, using the old "what if the 'one guy' gets hit by a bus" justification, which I think is totally legitimate. A couple days time to prevent having a single point of failure sounds like a good thing to me.

The software for this light, as I understand it, is mission-critical; no software, no happy little photons leaping out the business end of the LS module. (I'm more than willing to admit I could be way off base here.) Having the latest version or three of this software available, stored offsite (even if it's on a CD tossed on Peter's kitchen table), is a cheap insurance policy. Even if the light will function without software, Peter won't be happy with the light (and therefore won't sell it) until it has the software he advertised it as having {thereby making the software mission-critical}. So if the latest version of the software is lost somehow, Peter is out some not-small sum of money. If Peter's software guy really is rendered incapable of further work, and there is no recent backup of the software, Peter is out some considerable sum of money, until he can replace the software guy, and then re-create the lost code updates.

Do PP's suggestions make more sense now?

[ QUOTE ]
robk said:
Just my $0.02 - I think you are taking this matter too seriously. If ARC had a Gov't contract for 2 million pieces, yes, he could afford the software development you describe.


[/ QUOTE ]
How expensive is "the software development I describe?"

I recommended 3 FREE tools. Granted there is a couple of hours of learning curve needed to set them up and figure out how they work. Let's say... 3 days? = 24 man hours.

Now, for the walk throughs... Let's say it takes an entire week. 5 days times 3 people. 120 man ours.

That brings us up to 144 man hours total.

Now, compare that to the 3 MONTHS of porting time alone.

The measures I mentioned are not very expensive compared to the total development cost. What makes them hard is the development culture change needed.

Everything I've seen from ARC is TOTALLY top-notch engineering. And they seem to have a drive to do things that way. Electrical design, mechanical design, machining, etc, etc.

Walk throughs are one of the few methods that bring that degree of engineering discipline to software development. If you're not familiar with high quality software methods (Medical, avionics, automobile industry, mission critical embedded, anybody who loses a LOT of money if the software needs to be recalled) and ARE developing embedded software it is worth considering them.

[ QUOTE ]

ARC has a small market now, probably even a smaller market with the higher priced LS4.


[/ QUOTE ]

Peter has stated that the software "upgrade" method is to return the lights back to ARC have them reflashed and then returned to the user.

As you point out, ARC is a small company operating on reasonably small margins. Having a recall to fix a late found software defect would wipe out their profits for the lights effected. (And significantly detract from additional development. Opportunity cost).

[ QUOTE ]

I can live with a bug or two in the software, as can most people here. These are flashlights! We are losing touch with reality - smaller, brighter, microprocessor controlled, etc. He's not building aerospace guidance systems, just nice lights to walk the dog!


[/ QUOTE ]

Peter said he was talking to "serious people" who wanted the lights for "tactical purposes" and favor LEDs because they don't pop under recoil forces.

Now I don't know about you, but I'm not allowed to take an HK with me when I walk MY dog... ;-)

Those "serious people" would probably shun the light if they knew there was a significant software component in the light. (Unless Peter could PROVE to them that the software process was something they could trust.)

(I believe Peter is already downplaying the software for exactly those reasons...)

Plec

[ QUOTE ]
koala said:
PluckyPleco,
You actually pointed out right as the LS4 involves some software. If Peter want to look in to such practice he would have to go through contracts and agreements like non disclosure and etc for the contractors and beta testers.


[/ QUOTE ]
These are standard industry practices. The beta testers must return the lights and are not allowed to publicly review them.
[ QUOTE ]

IMHO, LS4 is mainly hardware. It will still function without the software but lose all the cool functionality.


[/ QUOTE ]
It has been stated that the vast majority of the software is "under the hood" and only a small amount of it is "cool functionality." From the information available it is not at all clear that the light will still function without the software control.
[ QUOTE ]

I think it ends up with weighting. If the software weights more than the hardware then it's worth looking in to it


[/ QUOTE ]
Software doesn't weigh ANYTHING. ;-)
[ QUOTE ]

Compared to a mobile phone firmware, I think LS4 is a small scale project, Peter could also have sorted out all these problems in the beginning.


[/ QUOTE ]
Agreed. ARC, not being a software development house, MIGHT not have considered these issues. I'm sure that Peter, et al, will do everything in their power to produce a top-notch light. But if you haven't been involved in producing near zero-defect software, you might not be familiar with the techniques needed to do so...

Plec

Good comments all. I just want you to know I do read most of the posts here on the CPF.

I claim to be somewhat knowledgable about computer programming (perl and C++). I have looked at Henry's code and I am very confident in his work. We have back ups of course. One thing that is very important to me is comments in the code and good use of whitespace. If we have to bring in other engineers to complete the work or update the code, the comments will help in that regard.

Peter

[ QUOTE ]
Gransee said:I claim to be somewhat knowledgable about computer programming (perl and C++). I have looked at Henry's code and I am very confident in his work.

[/ QUOTE ]
It's your company, and your company produces magnificent works. If you say you've considered these issues and have them covered, I'm a happy camper.

Plec

I think Peter is smart enough to hire smart people and make smart decisions with them.

I too am a S/W Engineer with 15 years experience and I have yet to be unable to pick up someone else's code, be it embedded for uC or PC apps, modify it and build it. However, I do agree with all comments made. Backups and a version control systems are a must for large projects. After all, all the files that make up the project must be there.

Peter's right, comments (correct ones!!!) help out a lot along with tested backups. Informal release notes are also helpful for tracking bug fixes and tool versions from release to release.

Can't wait for LS4/5...

Software developer here, too. (22 years.)

But wait, it gets worse: Security software.

Nah, worse still: DoD-approved security software.

You don't wanna know what kinds of QC and source control systems are goin' on here. (And some of them I can't discuss!) Especially now, with our flagship product getting a total ground-up rewrite and with an eye toward future Common Criteria evaluation.

oO

Hmm... wonder if there's a high concentration of programmers on the whole board, or just on this thread? Most programmers I've met do tend to like their techno-toys, but I've not met many as much into flashlights as I am, until I got here to CPF.

Is this a sound off for software folks? I'm going on 11 years experience in various programming capacities /ubbthreads/images/graemlins/smile.gif

The kind of things that Plucky starts this thread with SHOULD be standard practice, I know all about them and document their use and also I have violated every rule there at one point or another...

I've been following closely whenever Peter has let slip any information about this mysterious guy he's got slinging bits for him. I don't know the details, but I'm happy.

I don't think it's arc's problem, but I have seen OVER and OVER companies that treat the IT as secondary to a business plan which can't exist without IT, and they always pay for it in the end. Arc has their future invested in this project. If it gets lost or misplaced it's going to be expensive in time and money and other resources to get it back. paying 1 or 2 other people to sit in for 2 or 3 days on a walk through of the code I think is an excellent idea. Getting those people up to speed in a hurry if necessary later on would be greatly helped by doing something like this.

On other projects I've had large companies actually license my source code from me so that if I get hit by a bus they can continue to use the product. For smart companies this kind of thing is par for the course. All your eggs in one basket and all that!

I probably the youngest among, 7 years of s/w dev, NEVER been in the industry but I picked up all the essentials in my uni courses and still doing it /ubbthreads/images/graemlins/grin.gif.

Somtimes, I read comments, I go haywire. But when I ignore the comments and read the codes I understand. Of course this is for poorly written comments. You should read some of the comments I've seen /ubbthreads/images/graemlins/ooo.gif /ubbthreads/images/graemlins/grin.gif.

I find comments and documentations useless in the begining of my studies but now I have learned the use of it. I end up not recognizing the codes without comments and it's a waste of time to rewrite or figure out what's going on. That's only for highlevel. Assembly language is totally different thing, worst to 'decode'.

It's very exciting to know that Peter also himself is also working on LS4 firware. Can't wait to see it's release.

Vince.