Making the Vall on VoIP

"We had been planning to test voice over IP as part of our IPsec IP security VPN deployment, says Charles Arduini, Lexent's CTO. "After 9/11, we saw how voice over IP would help with business continuity. That confirmed for us that VoIP was the way to go."

A few weeks later, Lexent began installing VoIP at its headquarters and then at its first remote office, a new site in Hicksville, N.Y., where it moved some of its operations group from Long Island City. It cost the company about $26,000 to install the VoIP system at the new office, versus the $36,000 it would have cost for a new PBX system.

It took only about a day to get VoIP running at the site since the underlying VPN was already operational. "Voice over IP can be more reliable and is quicker to install," Arduini says. A single-day installation isn't "something you can do when you have a new PBX to put in."

Lexent is now installing VoIP at the rest of its 12 remote sites on the company's new, nationwide IP VPN.

The IP-based voice and data infrastructure has already paid for itself, Arduini says. The company spent less than $200,000 on the voice-data VPN. Arduini says he doesn't have hard ROI (return on investment) numbers yet, but he knows the returns already have been positive. The VPN replaced Lexent's private frame relay WAN, which had cost the company $400,000 to $500,000 a year in network charges.Loud and Clear

Lexent learned the hard way about configuring VoIP with encryption. The first voice traffic between headquarters and the Hicksville office experienced a half-second of jitter. The jitter was not only audible on the calls, but it was visible in the diagnostics built into the Cisco 7940 IP phones. The problem began when Lexent was setting up the IPsec tunneling between the VPN in Hicksville and the PIX firewall at headquarters. Lexent's technicians configured the tunneling to terminate the VPN tunnels from the Cisco 4224 router at Hicksville onto the firewall.

"There was no hardware accelerator in our firewall, so anytime IPsec did anything on the firewall or it got busy, it started jitter on the voice packets," says Doug Haluza, director of engineering and new technology for Lexent.

Haluza and his team installed an additional Cisco 3640 router at headquarters and rerouted the IPsec tunneling from the Cisco 4224 router in Hicksville to the new box rather than to the firewall, and the jitter disappeared. Because the hardware accelerators off-load the encryption processing from the software, they eliminate jitter, Haluza says.

That the application was live in the organization made it much easier to detect the VoIP tunneling problem. "We were fortunate we had voice running when we deployed the VPN in Hicksville because we were able to backtrack and work around the problem," he says.Lexent had chosen the router-to-firewall approach initially because the PIX firewall provided fail-over redundancy at headquarters, and having one box do two jobs was easier to manage. "If we hadn't done any voice, we would have been able to go router to firewall," Arduini says.

Lexent's VPN runs on a 100-Mbps Ethernet backbone at the company's downtown headquarters and 256 Kbps or full T1 to its remote sites, including Garden City, where the Hicksville office recently was moved. An important component of Lexent's strategy has been the redundancy of the VPN, voice equipment included: "We have two of everything at headquarters and a mirrored configuration at our disaster-recovery site, plus a backup ISP," Arduini says.

The company is using traffic-shaping features in its Cisco routers to make sure voice gets priority and its carriers don't drop voice packets. "We make sure we drop data packets up front," Haluza says. The traffic shaping ensures that its heavy data applications, which include engineering files and aerial digital photos as large as 40 MB, don't interfere with voice traffic.

Single Message Box

Cisco's Unity Unified Messaging (UUM) is the main application Lexent is running with the VoIP system. The company's Microsoft Exchange mail, voicemail and faxes are integrated, so a user can pick up voicemail, e-mail and faxes from an office line, cell phone or e-mail box. A user also can forward voicemail and faxes to other offices. UUM runs on the Cisco Call Manager server.The call-handler feature of Unity's Automated Attendant tool lets Lexent operate virtual helpdesks and reception areas: "All our helpdesks have IP phones, so when a call comes in, the call handler routes it to the next available agent, wherever that person is," Haluza says. "This lets us run mean and lean helpdesks."

Lexent software developers, meanwhile, are working with the Cisco Avvid API set to write other phone-based applications. One demo application the team is building would let a supervisor in the field track his team's time on the job using an IP phone rather than having to lug around a laptop.

"It's too early to determine what benefit we see from that, but it's certainly the next logical step," Arduini says. Sometime next year, Lexent will begin testing video over the VPN.

Lexent's original plan was to upgrade its managed frame relay WAN service with more bandwidth. But a new managed frame service would have cost the company 50 percent more each month than it was already paying. Plus, the service provider (whose name Lexent won't divulge) required a three-year contract.

So Lexent's IT department proposed an alternative to company executives: a private VPN. The price tag would be half that of a new frame service ($20,000 vs. $45,000) and require only a one-year Internet service commitment. "This allowed us to recover our capital investment in the VPN the first year, including the additional voice equipment needed for the voice-over-IP pilot," Lexent CTO Charles Arduini says.Given the economic realities of the frame relay upgrade, Lexent's IT group didn't encounter much resistance to the VPN or, later, the VoIP project, except at first from the team within IT that had negotiated the new frame relay deal.

"First we had to sell ourselves on the idea by building a strong business case. Once we had that, we were able to diffuse any outside resistance," Arduini says.

But Arduini's team isn't quite finished selling VoIP. Before IT can add IP phones to a remote site, it has to produce a business case for migrating voice to IP. Lexent can convert a site to IP only when its legacy phone system lease runs out, or if the site needs a particular feature the existing phones don't have. For example, employees who work both in the office and in the field might need IP's mobility. "Until we reach this point with a site, we cannot make a strong enough business case for replacing the existing phone instruments," Arduini says.

Even though the VPN infrastructure is in place, the phones Lexent chose aren't inexpensive: The Cisco 7940 IP phones are about $545 apiece plus the $160-per-user licensing fee for Unity Unified Messaging, Cisco's VoIP application that integrates e-mail, voicemail and fax into a single box. The company chose the higher-end IP phones because of their online corporate directory and XML features.

Charles Arduini, 33, has spent 12 years in IT and has worked for Lexent for nearly three years. As vice president and chief technology officer he is responsible for all IT operations and the development of applications and new technology.

Education: M.S. in mathematics from Stevens Institute of Technology, B.A. in mathematics from Bucknell University

Biggest lesson of the VPN project: "Work closely with your vendors--Cisco, in our case. And you must have redundant head-end VPN routers and run networks in parallel to allow for a phased migration."

Biggest lesson of the VoIP project: "You need a team with both a voice and data background to have a successful and timely implementation."

Next time, I'll: "Start sooner. Offices that came online with traditional phone infrastructure could have been done with IP telephony instead, and now it's tougher to make the business case for replacing recently purchased equipment."Biggest technology flop ever: "The Y2K technology scare. Some mitigation was needed, but the measures ultimately taken were way overblown."

Best advice ever received: "If you don't know the answer, answer 'I don't know.' "

Worst advice I've ever received: "Technology is like the quote from the movie Field of Dreams--if you build it, they will come."

Biggest bet I've ever made: "Implementing IP telephony. So far, so good."

For fun: "Spend time with family and reading."Ideal wheels: "Good gas mileage and a CD player."