All Archive Data is Not Alike

Just because we call all the data users aren't changing on adaily basis archival, that doesn't mean it's all the same.  Depending on its source and usage patterns, differenttypes of archival data can be best served by different storage techniques andmedia.

The most common type of archival data is made up of emailmessages and files that have been put into the archive and may need to be accessedby users transparently. Realizingthat object access rates fall off dramatically as objects age, system administratorsset up policies to migrate objects over 30 or 60 days old to an archival storagesystem and replace them with some sort of pointer or stub so the user can stillaccess the data from its current location in their mailbox or home directory. Organizations with strict regulatory requirementslike SEC 17a-4 require that some data be retained, and will archive objects as they'recreated and/or published/finalized and then replace the original files withstubs as they reach the 30-60 day age.

Since users are accessing this data transparently thearchive system has to be fast enough to retrieve data without the user noticinga substantial delay. While the archivedoesn't need to be as fast as the primary storage, notice there's no random I/Oto files on the archive; it does have to be able to retrieve this kind of datawith response times in the one second range.

After some additional time -- SEC 17a-4, for example, says datamust be easily retrievable for 2 years but retained for 7 -- transparent accessbecomes less important and archive solutions may delete the stubs leaving thearchive's UI and indexing system as the primary access method. Since we've moved from real time access to aquery/response model, the archive solution can take some tens of seconds tominutes to return the objects requested without serious impact on userproductivity. This second tier ofarchive storage also fits the needs of data retained for potentiale-discovery.

Then there's the deep archive that's kept not 7 years but 70or 700. All footage shot for a TV showor movie, the as-built blueprints for each plane Boeing builds, the designs an architectural firm or a buildingsdepartment keeps on file, digital X-rays for patients that have been dischargedfrom the hospital and all the other data that's traditionally stored inwarehouses. Add in scanned images ofhistorical documents and such that fill miles of shelves in real, as opposed tobusiness record, archives. This data can be stored on a system that literallytakes minutes (for those digital X-Rays) to hours to retrieve.

Note this deep archive data is also typically made up oflarge objects and these objects are frequently retrieved together.  Boeingmay need to call up all the tail sections of 737s built over a 3-year period tosee which have the old style parts the FAA says need to be replaced or Lucasfilmmay want to call up all the Star Wars Episode XXXIV footage for a new Blu-Rayspecial edition.
While all this data could be stored on magnetic disk systems,with or without MAID, the power, space and periodic migration required couldmake another medium more attractive. More on this subject next week.