Euro Telcos Face Storage Avalanche

A new European data retention directive designed to help law enforcement agencies tackle terrorist threats is set to cost Europe's telecom operators and ISPs a collective fortune, and it will likely provide rich pickings for storage vendors and integrators.

The European Union's Directive 2006/24/EC "on the retention of data generated or processed in connection with the provision of publicly available electronic communications services or of public communications networks" is already in legal force. It was hurried through the European Parliament following the terrorist attacks in Madrid (2004) and London (2005). (See EU Debates Data Retention.)

The debate on the directive is over. As a law, it's now a "must do," and a mandatory compliance date is set. The directive's requirements need to be implemented by the EU's 25 member states by September 15, 2007.

By that date, fixed and mobile telecom operators and ISPs will need to have data storage and retrieval systems in place that can deliver on the directive's exacting requirements. Based on a cost assessment by one country, the collective cost to EU service providers is set to be in the hundreds of millions of dollars.

Those service providers, by law, will need to retain and store call data records (CDRs) of voice calls, voice mails, and text and multimedia messages placed on their networks. Those records must include the telephone numbers, identities, and addresses of the calling and called parties, the time and duration of the call, and the type of service used.For Internet services the requirements are largely the same -- IP addresses, names and physical addresses, as well as email and VOIP service-usage records. The content of communications do not need to be saved and stored.

But the data that needs to be stored must be saved for a period from six months to two years.

According to pan-European organization European Digital Rights, the Ministry of the Interior in Finland -- a country with a population of little over 5 million people -- estimates the cost to Finland's telecom services companies of meeting those requirements will be up to 40 million ($50 million). The EU's total population is 457 million.

But it's not the physical data storage that will tax the service providers as much as the information retrieval requirements, noted Jerald Murphy, senior VP and service director at consultancy Robert Frances Group , during an online public presentation this week.

Carriers and ISPs "must be able to respond to data requests from law enforcement agencies without undue delay," he noted, which means that the data must be instantly accessible. "We're talking massive data volumes here -- tens or hundreds of millions of records each day" that will generate "hundreds of terabytes of raw data," which will need to be managed.Murphy noted that while carriers already capture and store call records, the Directive requires much more detailed retention and, crucially, speedy retrieval. "Storing the data is not the big issue -- extracting the data is a whole other story."

Carriers will need new IT and storage systems and bandwidth. They will need to train their staffs and implement new data retention systems and processes, he added.

Currently, telecom operators tend to use relational databases to store such records, but "these traditional data management solutions were not designed" for the data volumes the Directive stipulates. Extending existing carrier systems to meet the new requirements would be incredibly expensive, Murphy asserts.

He also noted that, while penalties for non-compliance have yet to be detailed, they're expected to match similar EU directives that carry jail sentences of up to 10 years and fines of up to $500,000.

The opportunities for storage vendors and IT systems firms appear lucrative, with old buddies EMC and SenSage forming a tag team to target the telcos. (See EMC Approves SenSage.)EMC's business development manager, Charlie McAlister, stresses that the systems carriers require will need to be able to handle "hundreds of billions of CDRs and then maybe the same again, or even double, when you add Internet usage records." He says one existing EMC telco customer in Europe generates 13.6 Tbytes of call-related data each quarter from just one department, which would add up to 108.8 Tbytes over the maximum Directive term of 24 months.

So how much does EMC believe carriers will need to spend to meet the Directive's mandate? McAlister tells Byte and Switch that the system costs are hard to estimate at the moment. He's come across some projections that put individual carrier hardware costs at up to €5 million ($6.25 million), but he believes those estimates are "probably on the high side."

But with the European Union housing hundreds of fixed and mobile operators, plus many more ISPs, even a low single-digit million cost per service provider would put the total cost into the hundreds of millions.

McAlister says EMC is currently putting together a "proof of concept" that will help the carriers determine the costs. He says service providers "are worrying about this now, and some are starting to budget. It's definitely on the carriers' agenda."

The carriers themselves, naturally, are not happy with the additional burden and costs, which are not due to be subsidized by the EU's member states. In a joint statement issued before the Directive became law, industry bodies, including the European Competitive Telecommunications Association (ECTA), the GSM Association, and the European Internet Service Provider Association (EuroISPA), criticized the detailed nature of the Directive and the lack of financial assistance. The statement also noted: "Beyond their economic consequences, far-reaching data retention obligations may also undermine Europeans’ confidence in new technologies and thus slow further down ICT [information and communication technology] take up, putting at risks Europe’s ICT sector competitiveness."— Ray Le Maistre, International News Editor, Light Reading. Special to Byte and Switch

Organizations mentioned in this article: