Next-generation optical storage: what's the lowdown?

[KevinL]

Thought I would ask our computer experts As per forum policy, no trolls and flame wars, of course. Keep it civil.

I am a photographer who is rapidly becoming buried in DVD's, who has had minor issues with network attached storage, and doesn't trust external Firewire/USB hard disks at all. This is leading me to look at next-generation optical storage solutions. Being able to back up 500GB to 10 or 20 disks is appealing to me.

Where are we, and where is this whole debacle going? It looks as if the industry has refused to learn from previous quagmires and insists on starting another format war.

So what are the currently available drives, storage capacities, and what is this all ultimately going to mean for us?

Also, a footnote.. I am not intending to buy at the moment. Drives are still hovering around the $500 mark for me. While not unreasonable, and I recall in the early days our CD-R's cost as much, I have decided to stick it out with my DVD-R 4.7GB for a while longer. Especially since I just finished backing up ALL my data to DVD-R and don't feel like embarking on a media migration exercise yet. Just curious as to where it's all headed.

 

[jtr1962]

Honestly, even though there's higher capacity disks available I think the 4.7GB DVD is it. Dual layer capable drives have been around quite a few years, but the dual layer DVDs are still quite pricey compared to their single layer cousins. Bluray and HDDVD, forget about it. That stuff is in the stratosphere, and likely to remain so for a while.

The way things are going I feel by the time the larger optical disk formats drop in price enough to be mainstream, if indeed they ever do, solid-state storage will make them superfluous. You can already get 4 GB flash drives for roughly $25 to $30. In a few years my guess is you'll have 128GB or 256GB drives available for $10 or less to compete with the Bluray or HDDVD formats. Prices per GB for solid state storage will undoubtedly continue to drop further as time goes on.

I see the hand writing on the wall. Solid state has already replaced floppies, LS-120s, Zip disks, Jazz drives, etc. As more and more software is being downloaded instead of bought of CDs/DVDs those uses of optical disks are becoming less important. Who knows, in a few years it'll probably be cheaper to distribute software on 1GB or 2GB read-only thumb drives, effectively making optical disks just about obsolete. In fact, solid state will eventually make all forms of spinning disk storage obsolete. However, I predict magnetic hard disks will be last to go. They're still hard to beat in terms of raw capacity and price per GB.

In short, my suggestion is to just use DVDs for backup now as inconvenient as they may be. By the time optical formats which make your life easier become cheap enough, solid-state will also be. I really don't plan to invest in higher-capacity optical drives or media until the price drops considerably, and they have one drive which supports both new formats. Hell might freeze over before that happens.

 

[paulr]

If you are really serious and rich, get an LTO tape drive. Otherwise use hard disks with multiple redundant drives. Optical discs are just too much hassle, juggling all the silly discs all over, if you have a lot of data.

 

[Fallingwater]

The people responsible for the distribution of next-gen optical discs seem to be all hung up on HD movies and such. HDDVD and Bluray are still prohibitively expensive for general-use data storage... which is why IMO nobody cares about the much-hyped "format war". HD works either way, and we aren't getting cheap burnable discs, so the attention is very low. Hopefully they'll learn their lesson, and release the next format in a cheap burnable version along with the HD whatever versions.

 

[KevinL]

Very good points raised here.. :twothumbs

Good point about solid state. I've observed solid state go from expensive to cheap enough to make hard disks out of. This may be the tipping point and the watershed where prices crash.

I never believed in tape. Worked computing for many years - hated tape from the get go. Watched everything from Travan to DAT to DLT fall apart when needed most. Tape has something personal against me. My defense-in-depth protocol at work calls for multiple layers of backup, most of which rely on network attached storage. Yes, I built a NAS for my home network, despite the fact that it can be a real pain in the @ss at times. (NAS are not problem free either, I will grant you that, but the number of times they have prevented my bacon from becoming an accompaniment to the omelette on the breakfast platter of data loss are way too numerous to count. Yes I also roll a 'custom' configuration based on what I developed for securing our corporate data.).

Agreed on the movie distribution thing. I could honestly care less about movies and games. I'm looking at a cheap mass storage solution for my own content that I create and own the exclusive rights to. (generally speaking, photos are copyrighted to their creator at moment of creation)

DVD-R's, inconvenient or otherwise, are a second layer of defense should the NAS fail.

I don't trust USB/Firewire devices. Personally, I have had both a USB and a Firewire disk fail without any plausible explanation (and that's coming from someone who's seen computers do truly paranormal things) as well as recommendations from expert colleagues who have had it happen MANY times, not just a couple.

 

[Fallingwater]

Hard drives should never be trusted alone.
I entrust my backups to two different hard drives. The chance that both will fail at the same moment is negligible.

 

[eluminator]

I don't trust USB/Firewire devices. Personally, I have had both a USB and a Firewire disk fail without any plausible explanation.

Did you check the hard drive temperature? Many external enclosures have no fan, and the hard drive will eventually get hot enough to fry eggs on it if you use it for an extended time. I had that happen to me so now I buy enclosures with good cooling. I was able to salvage that fried egg drive by reformatting. I then cut a hole in the top of the enclosure and installed a fan, and I still use it.

There are lots of "bridge" chips in use in external enclosures to convert USB or firewire to IDE. Most are too stupid to allow software to run hard drive diagnostics and read the drive temperature. Actually most hard drive diagnostics are too stupid to test these drives, but Western Digital's will.

But there is a better way to connect hard drives. A SATA drive in an external enclosure plugged into a SATA port on your computer behaves identically to an internal drive. So you can run diagnostics and read the drive temperature. If you can find a SATA enclosure with adequate cooling, you should be in business.

 

[gadget_lover]

The solid state drives are OK, but they are not suitable for long term storage. They have a relatively high error rate and corresponding failure rate. I've lost a card full of pictures twice in the last 10 years.

For archival, tape is not a bad solution. Just remember that "One is None" and make multiple copies. If you get a tape robot (even a small one with 8 - 15 tapes) the backup process becomes painless. You do need to mark your calendar with a reminder to regenerate the tapes every few years, and you need to pay attention the orientation when storing them too.

But back to the question.... With terabyte arrays becoming comonplace I see an increase in the need for large capacity removable storage. I bought a 1 TB drive from Costco last week, so we know that it's become commonplace. I'm hoping that they can come up with something better than blue-ray, but I'm not going to bet on it.

Daniel.

Daniel.

 

[Fallingwater]

I suspect the powers that be are actively hindering research, development and marketing of alternative high-density optical storage.
If they dropped all the bullshit the technology would probably soar. *sigh*

 

[myk]

I think next-gen optical is completely feasible at this point - still a little pricey but you're on the cutting edge, and the part that will really be getting cheaper is the media - so what if the drive drops $100 in the next 2 years, as long as the media gets to $2/50gb sooner than later

A Blu-Ray writer can be had for $400, and Dual-Layer 50GB Blu-ray discs can be had for $20, or 25GB discs for $8

With archival quality dvd-r's at close to $1 each for 4.7gb, it's really only $3 more (media wise) for the 25gb bd-r's



another thing to consider would be a relatively inexpensive RAID setup - I don't know your current hardware setup is but it would cost about $600 with 2TB of storage, of which 1TB would be wasted as a mirror of the first TB (4x500gb SATA drives with onboard RAID with an inexpensive mobo/case setup).

 

[DM51]

The solid state drives are OK, but they are not suitable for long term storage. They have a relatively high error rate and corresponding failure rate. I've lost a card full of pictures twice in the last 10 years.

Ouch. Oh dear, I thought solid state would be the way to go. Obviously not, unless they can improve on the reliability.

I lost a pics from an SD card recently too. I blamed it on magnets, but I now understand they have no effect on cards and therefore had nothing to do with it. Maybe it was a dodgy card like yours, although it has worked OK since.

Are Blu-Ray discs likely to be any more reliable for long-term storage than CD/DVD, which has a limited life?

 

[JimmyM]

I've used a couple NAS solutions in my home office for years with no issues. While I was an independent consultant, I was my own IT department. I've used primarily IDE and SATA RAID-5 arrays. I've lost a disk on 2 occasions. I just pulled the bad disk, dropped in another, and told the RAID software to rebuild the array. Never lost data due to drive failure. 500GB and now 1TB drives are available. You could simply get 2 1TB drives and mirror them (RAID-1). This decreases performance, so you could add 2 more drives and "stripe" them (RAID-0) making a RAID-0+1 or RAID-10 array. If you want BIG storage use RAID-5, RAID-6 if you're paranoid. Usually only the more expensive SCSI or SAS controllers offer RAID-6.
Unless you use off-site archiving, burnable optical media is just a giant PITA IMHO.
I can see by your origional post, that you seem to already have ruled out NAS and are looking for an optical solution that does not yet exist. I think you should reconsider NAS.

 

[gadget_lover]

The advantage of removable media (optical or tape) is that the data is safe from being erased by accident, intentional act or viruses.

I have DVD backups of my systems as well as hard disk based backups. One of my systems has a small 4 tape backup robot, so it captures daily changes on my main workstation.

Daniel

 

[eluminator]

Are Blu-Ray discs likely to be any more reliable for long-term storage than CD/DVD, which has a limited life?

The quality of CDs and DVDs can vary widely. So can the quality of various brands of burners. And all burners will go bad eventually, usually within a few years.

But a good quality DVD or CD with a good quality burn should last many years. Nobody knows how long but I wouldn't be surprised if they lasted 50 or more years.

For qood quality DVDs get Verbatim or Taiyo Yuden. Personally I always check the burn quality of my archival DVDs with Nero's CDspeed program.

By the way, I'm doing a long term reliability test on some DVDs I put in the trunk of my car 2 years ago. I'll get back to you after 100 years. In the meantime I've found that not only have the number of errors (correctable errors that is) not gone up, they have actually gone down a bit.

 

[DM51]

I'll get back to you after 100 years

Lol. Thanks for the info, and the tip on Verbatim.

 

[KevinL]

Hmm, definitely agreed on the Verbatim part. The Verbatim disks are made by Mitsubishi Chemicals and they are good stuff.. I have had CD-R's from the earliest days of my CD-R burning (literally 10 years ago) still doing fine. By the time another 10 years go by it will be more than time enough for another media migration. One of the problems of being a digital archivist is forward media migration to newer technologies. However, the good thing is that newer media tends to have larger capacity so we just consolidate.

Good news: I managed to fix my NAS, a new software image cured the ills, and it's sitting underneath my desk.

Bad news: I had another nearly-brand-new Firewire external disk giving problems, I refuse to use these things any longer except for moving non-critical data.

Thanks for the tips on solid state, perhaps I will steer clear of those.

 

[turbodog]

Ditto on the verbatim. I've got some 15+ year old disks that are fine.

Also, and this is the _best_ indicator of their quality, I received some recovered files off a hdd from drivesavers.com. They sent the files to me on a verbatim disk.

 

[eluminator]

Just a little info for those of you who haven't opened up an external hard drive. The actual hard drives in the external enclosures are the same drives used inside your computer. If you open up the enclosure, you can see for yourself.

Most drives used in external enclosures in the past were IDE drives. Maybe new ones might use SATA drives, I don't know. If you remove the drive from an external enclosure, you can put it in you computer and hook it up directly. This could help diagnose any problems and could help recover data, if the drive is still good.

For instance, in the case of a "firewire" external drive that has problems, it could be the drive itself, or it could be the bridge chip in the enclosure that converts firewire signals to IDE signals, or it could be the enclosure's power supply, or it could be the firewire hardware in your computer, or the firewire cable, or the firewire driver in your computer. Plugging the drive directly into an IDE port in your computer would help answer a lot of questions.

As I said in an earlier post, if the drive itself is bad, it probably overheated do to poor cooling in the enclosure. Of course if you dropped it on the floor, or even set it on your desk with a bang, that could explain it.

It seems to me the best external drive for modern computers is a SATA drive in a SATA enclosure without a bridge chip. In this case the drive is essentially plugged directly into the computer's SATA port, just like an internal drive. The only disadvantage is the SATA cable must be fairly short. Around 3 feet max, I guess.

 

[KevinL]

Just a little info for those of you who haven't opened up an external hard drive. The actual hard drives in the external enclosures are the same drives used inside your computer. If you open up the enclosure, you can see for yourself.

Most drives used in external enclosures in the past were IDE drives. Maybe new ones might use SATA drives, I don't know. If you remove the drive from an external enclosure, you can put it in you computer and hook it up directly. This could help diagnose any problems and could help recover data, if the drive is still good.

For instance, in the case of a "firewire" external drive that has problems, it could be the drive itself, or it could be the bridge chip in the enclosure that converts firewire signals to IDE signals, or it could be the enclosure's power supply, or it could be the firewire hardware in your computer, or the firewire cable, or the firewire driver in your computer. Plugging the drive directly into an IDE port in your computer would help answer a lot of questions.

As I said in an earlier post, if the drive itself is bad, it probably overheated do to poor cooling in the enclosure. Of course if you dropped it on the floor, or even set it on your desk with a bang, that could explain it.

It seems to me the best external drive for modern computers is a SATA drive in a SATA enclosure without a bridge chip. In this case the drive is essentially plugged directly into the computer's SATA port, just like an internal drive. The only disadvantage is the SATA cable must be fairly short. Around 3 feet max, I guess.

Possible. It's SOME weak link in the chain. I loaded this enclosure personally, stuck my own 500GB Western Digital into it. No idea what it is, and no desire to troubleshoot it.

The same 500GB is now directly plugged into an Intel ICH9R and not only does it work great, performance absolutely FLIES. Linear R/W throughput has gone from 30MB/sec to 85MB/sec. That's useable throughput too, not a theoretical number.. Gotta love it!

 

[eluminator]

The same 500GB is now directly plugged into an Intel ICH9R and not only does it work great, performance absolutely FLIES.

Not all firewire or USB to IDE bridge chips are created equal. I'd guess that is the problem, assuming the firewire hardware on the mobo is okay. I think some of the firewire bridge chips are user programmable. But like you, I don't want to mess with them.

Anyway, that's why I would only use SATA in the future. An external SATA performs identically to an internal SATA. No bridge to cause problems.