More on Disk Archives & Objects

11:10 AM -- In my last blog post, we talked about objects and what they mean in a disk archive. As we persuade IT departments to get off of their addiction to primary storage, it is more important that we explore the dirty little details that affect disk archives. Our first entry on object count really begs the question: When do objects become a problem?

As always, the answer is it depends on the content-addressable storage architecture. But some vendors put the limitation at around 80 million to 150 million objects. I know, wow; who will need more than 150 million objects?

Probably YOU in a few years -- maybe even now. For example, assume the best case scenario that you start to see a performance hit at 150 million objects. Let's assume that the average file size is 250k. That means 4 files per MB, 4,000 per Gbyte and 4 million per Tbyte. So 150 million objects or files would equal about 37.5 Tbytes of archive capacity.

Depending on the average file size with some of the CAS archives, you actually could run out of object count space prior to running out of capacity.

For an archive, 37.5 Tbyte is not very much when we are talking about storage that is supposed to hold years and years of information. As we talk to end users about archive design, more and more frequently we are planning for petabytes of information that needs to be stored. The real gotcha is whether or not 250k is a valid number for average file size.For example, if I look at my email I have the annoying 2.2-Mbyte emails sent to me by vendors and then I have thousands of 16k or smaller emails. Now, most emails systems do not store these as separate files. But there are a few archive products that extract Exchange emails and store them as individual files. If you are in that boat, beware, you could hit performance issues real fast, like at 3.7 Tbytes. Other email archive applications pack a bunch of email messages into an SQL database; 5,000 per messages per database is a common setting. This number is also typically user definable, so in an object-based archive you may want to increase the number of message per database to optimize performance.

So what are the solutions to object overhead? The first method is to increase the object count with processing power by load balancing the object count across nodes. Permabit Technology Corp. , for example, distributes the object overhead across nodes supporting somewhere between 60 million to 128 million objects per node, so in a fully populated system they claim support of 180 billion objects. Meaning that the average file size would have to be less than 32k prior for them to suffer running out of object space prior to running out of capacity.

Companies like Nexsan Technologies Inc. , take a different approach, using a series of virtual CAS archives federated into one large archive, which they claim eliminates object count altogether. Doing so also allows for segregation of data by virtual CAS, yet still provides a single view for the administrators without the performance concerns around object count. With object count eliminated, you have the freedom to store individual records or messages as opposed to packing them into a whole database.

Regardless of the method that the archive vendors use, make sure you understand the impact of their architectures. The archive tier is going to be an important storage decision in the data center, and it is important to understand that this is about more than just going out and buying a bunch of cheap disks.

George Crump is founder of Storage Switzerland , which provides strategic consulting and analysis to storage users, suppliers, and integrators. Prior to Storage Switzerland, he was CTO at one of the nation's largest integrators.6668