The Real Cost of VoIP

Something for nothing-that's the appeal of Voice over IP (VoIP). We've all heard the pitch before: Run your voice over the existing data network and you'll save on call charges, consolidate management, cut maintenance costs, eliminate duplicate wiring plants, and increase user productivity.

Yeah, right.

The fact is that while the price for VoIP equipment may finally be competitive with traditional PBXs, much of the real costs remain hidden. Expect to install Ethernet switches, upgrade those edge routers, and increase port speeds and Committed Information Rates (CIRs) if you want to run voice across a frame network-and that's not mentioning the time you'll spend troubleshooting, tweaking, and engineering the network to accommodate voice without starving other mission-critical applications.

"Every day I ask myself why I deployed VoIP," says Michel Labelle, supervisor of information and communications technology for a major city in British Columbia, Canada. And that's coming from a network manager who can show a demonstrable return on investment for his VoIP implementation.

We at Network Magazine understand Labelle's pain, having spent six months living and breathing the RFP bid process for a real, but anonymous facilities management company that manages some 2,000 customer-owned facilities across North America. The $1.5 billion company, which we'll dub ServiceNet, turned to Network Magazine for help in selecting its next-generation voice solution. With an old NEC PBX, a mix of key systems spread throughout its remote sites, and a flat Ethernet network in need of an upgrade, ServiceNet was due for an overhaul. VoIP seemed the logical choice. (For an overview of ServiceNet's network, see "The ServiceNet Network".)While VoIP would save ServiceNet on toll-bypass, the real incentive for the technology was very different. "We're trying to make our remote offices feel like they are more 'one with the company,'" says the company's director of IT. The company was also looking to "facilitate communications, reduce maintenance/support, add mobility features, and increase convergence options," he says. Moving to unified messaging while minimizing the involvement of (and dependence on) competitive local exchange carriers and the PSTN was also high on the agenda.

Network Magazine documented these requirements in a 70-page RFP requesting bids for a converged enterprise network. Respondents were expected to deliver pricing and detailed specifications for both the voice and data network. The respondents also needed to document features and functions relating to scalability, reliability, security, privacy, remote access, and telephone and software functionality. Over 30 vendors were invited to participate, but only ten of the biggest names in the VoIP business took up ServiceNet's challenge. The traditional PBX vendors were Mitel, Nortel, Avaya, Siemens, and NEC by way of Expanets, a network integrator. The traditional data providers were present through 3Com and Cisco via a solution pitched by Sprint. Independent VoIP providers were represented by Shoreline. WorldCom also replied with an IP Centrex solution (see the Table).

The RFP Bids

* Also in Bradley and in LA.

To help analyze the results, Network Magazine brought in Labelle, who aside from running an 800-seat Nortel VoIP installation, has been a voice and data engineer for the past 14 years and is intimately familiar with most of the VoIP architectures out there. After weeks of poring over the bids, it was decided that ServiceNet would be best served by a graceful migration to VoIP, which meant installing a gateway on the back of its PBX and gradually moving individual phones to VoIP. To this end, NEC stood out, partly because of its existing presence within ServiceNet, but also because of the ruggedness and resiliency provided in its proposal. Nortel and Siemens's low price point and value won them a spot on ServiceNet's short list as well.

At the same time, Sprint and Avaya's solution couldn't be ignored. Sprint's bid offered great resiliency, though its Cisco solution was weaker on some of the more exotic voice features. Avaya's pitch also provided great security, including a built-in firewall. Should NEC, Siemens, or Nortel fail to deliver, ServiceNet would need to investigate Sprint and Avaya more thoroughly.And that's not just paranoia. Siemens, for example, didn't stipulate installation or training costs, which could tack on another $150,000. Labelle's experience on this point was telling: "When we ran our RFP, an integrator pitched the Cisco solution at $1.1 million, while Nortel and the other solutions came in at $500,000 to $600,000," he says. "When our implementation is finished, we'll end up spending $1.4 million and still lack the resiliency and functionality provided for in the AVVID bid."

As for the remaining vendors, a combination of factors eliminated them from ServiceNet's short list. ServiceNet was very budget-conscious, which made the company very concerned about the bids from Avaya, Sprint, and WorldCom. Centralized management was also important, which made ServiceNet nervous about 3Com's NBX 5000. Though it offered much of the centralized functionality of a single-image solution, the NBX wasn't a true single-systems image, which is critical for both management's sake as well as end-user simplicity.

What's more, the handsets needed certain basic functions, such as hands-free intercom, call logging, and the ability to run Web-based applications on the high-end phones, which Shoreline didn't provide with its telephones. (ServiceNet only required 376 IP handsets, so the scalability of these solutions wasn't a big issue. Had ServiceNet required several thousand sets, however, the NBX 5000 would have been eliminated since it's limited to 1,500 seats.) The company also needed a new voice and data infrastructure, which Mitel's proposal didn't offer.PRICING

As our research for a VoIP solution began, we steeled ourselves for impossibly high bids that far outstripped traditional voice solutions. We were warned that VoIP could never match the price point and functionality of a traditional PBX, so we weren't entirely surprised to see bids that were twice as expensive-after all, these bids called for a major overhaul of the voice and data network.

Perhaps more surprising, however, was the number of bids that seemed to compete favorably with Avaya's Definity PBX, the market-leading voice solution. A Definity Server CSI in ServiceNet's main office in Chicago, with Avaya IP Offices in the 17 remote sites, would run $263,000 for the equipment alone.By contrast, the price tag for the common control equipment and handsets by Nortel ($268,405), Shoreline ($221,465), Sprint ($243,467), 3Com ($243,951), and NEC ($266,198) were under or nearly the same as the Definity. The remaining providers were substantially higher, with Siemens at $327,259, Avaya at $520,274, and Mitel ranging from $353,464 to $507,596, depending on whether the SX-200 IP KTS or 3340 Branch Offices were installed in the remote sites.

But those prices are misleading. For one thing, they don't include the cost of additional telephony services, such as voice mail, Automatic Call Distribution (ACD), and auto-attendant. Neither were maintenance, training, and management costs factored into many of the bids. Rough estimates show that these items would add another $240,000 to the cost of the Definity solution, not including training, or $144,000 when taking in Avaya's standard 40 percent discount. The VoIP bids varied significantly in these additional costs, largely depending on whether vendors included management, maintenance, and training costs. On average, however, these costs added another $127,000 (after discounts) to the costs of the common VoIP equipment.

Moreover, as the number of handsets grows, VoIP costs will likely increase more dramatically than digital costs. NEC, for example, pitched an i Series two-line phone for knowledge workers at $287, versus $187 for a comparable digital set. But add to that the cost of software licensing (eight seats at $27 per seat), a hub to deliver inline power ($21 per port on a 24-port Cisco 3550), and a UPS to deliver the emergency power ($700), and the total cost of an IP phone can easily double that of its digital equivalent.

THE DATA NETWORK

Of course, calculating the actual price for IP handsets, and VoIP in general, gets trickier as it starts to incorporate data equipment costs. All respondents insisted on upgrading ServiceNet's flat network to a switched network with the necessary QoS mechanisms in place to ensure toll-quality voice. The results ran anywhere from another $30,261 (Shoreline) to $387,684 (Sprint) for the data equipment. On average, data equipment constituted about 25 percent of the overall bid, or added anywhere from 40 percent (NEC) to 159 percent (Sprint) to VoIP equipment costs.With that investment, ServiceNet received the same basic LAN architecture from most vendors, albeit with different vendors' gear. Within the main site in Chicago, one Ethernet switch typically served as the core of the network, though Sprint used two. Hanging off this core complex were the workgroup switches connecting to the desktops, gateways, and access routers. Call servers typically connected directly to the core Ethernet switch, with the exception of Avaya and Cisco. Their call servers were connected to their own dedicated switch hanging off the core switching complex, possibly to account for increased traffic that flows between the primary and secondary call servers.

The LAN design of the remote sites was also straightforward, with an access router connected to an Ethernet switch. A survivability unit, also serving as a local gateway, connected to the Ethernet switch. PCs connected to the phones, which in turn connected to the Ethernet switch. Cisco and Avaya's remote site implementations were slightly different: Cisco placed the survivability function inside its upgraded access router. For Avaya, the remote routing function sat inside the Ethernet switch.

Part of the reason for adding data equipment has to do with voice's unique requirements. Another part of it might be that vendors are looking for a good excuse to move their data equipment. Avaya, for example, recommended replacing all of the existing Cisco switches with Avaya gear, even though the Cisco 3550s could support voice.

Avaya also insisted on replacing the access routers with its equipment, but this might just reflect Avaya's conservative design philosophy. The installed routers could handle QoS, but respondents didn't know how much memory existed in those routers. For an additional $42,500, Avaya advocated replacing them with its X330 WAN modules, which would slide into its Ethernet switches in the remote offices.

One of the benefits of going to a VoIP-specific company such as Shoreline is the ability to take advantage of low-priced third-party data equipment. Shoreline's data pitch added just 38 percent to the price tag of its VoIP gear, the lowest of all the vendors. This was largely accomplished by leveraging Asante switches in its design, whose perport costs can run 50 percent less than that of their Cisco counterparts.To help minimize data equipment upgrades, ServiceNet didn't insist on a fully fault-resilient design. The reliability of Ethernet switching equipment was such that the company felt it could sustain the risk of what might amount to about 10 minutes of downtime per year (99.99 percent uptime) without a problem. While most respondents chose to build the network around a single Ethernet switch, the Cisco AVVID design used by Sprint called for a fully resilient core network with two new Cisco Catalyst 4507R switches with redundant power supplies and supervisory engines serving as the core of the network.

Vendors also cut costs in terms of how they delivered power to the desktop. With each site able to support an emergency analog phone, ServiceNet only insisted on ten IP phones being powered via the network and capable of remaining active in the event of a power outage. Most vendors provided inline-power-capable switches throughout the network. Shoreline reduced the price of its bid by providing midspan power in the specific offices where those phones would be located.

Shoreline's midspan device is a separate standalone unit that connects to the end-node on one side and the network switch on the other. This is a viable solution for many organizations that might not want to upgrade their edge switches. But for new installations, the better technical approach is to use a Power over Ethernet (PoE)-capable switch. With a PoE-capable switch, users can move phones without IT's assistance, and every network drop becomes a potential voice outlet. The remaining vendors chose to implement PoE-capable switches for all the sites.

The other difference in the data infrastructure pricing involved the scalability of the network architecture. All of the bids recommend a 24-port Ethernet switch for the remote sites. This would be fine for most of the sites, but four already had 20 users. There's a high probability of maxing out those ports, particularly when accounting for resident servers, additional users, networked appliances, and printers. Siemens was the only vendor to recommend installing Enterasys Networks's 48-port 10/100 switch in the remote sites, giving them plenty of room for growth.

Finally, anyone who advises using excess bandwidth to run VoIP over the WAN has probably never played with the technology. Although a G.729 codec can drop voice down to a 24Kbits/sec IP payload (vs. G.711's 80Kbits/sec), the actual wire bandwidth can run closer to 38Kbits/sec. Clearly, ServiceNet was going to need to upgrade its WAN, particularly in the nine sites running frame relay ports at 64Kbits/sec or lower.Only three companies-Shoreline, Mitel, and Nortel-provided specific suggestions. Both Mitel and Nortel recommended moving to a minimum port speed of 128Kbits/sec, with Nortel and Shoreline advising a 32Kbits/sec CIR and Mitel advising at least a 96Kbit/sec CIR. Avaya simply said that an onsite analysis would be needed of the WAN, while Sprint didn't provide any specifics.

THE VOICE NETWORK

Perhaps the most subtle differences in cost between the traditional voice and VoIP solutions lie in the engineering of the network. It's no coincidence that every respondent insisted on doing an onsite analysis and wouldn't certify that their solutions would work otherwise. The interplay between QoS, VLANs, router queuing, traffic patterns, and calling patterns is too complex to derive from a simple document.

But what this also said was that ServiceNet needed to carefully consider the additional engineering costs not factored into the bids, such as those involved in dimensioning the WAN and the LAN. Getting a fix on those costs may be impossible from the bids alone, but some tell-tale clues exist for users to consider.

Jitter levels, for example, were extremely high in NEC's bid. Whereas most proposals wouldn't cite jitter tolerance, those that did claimed toll-quality voice at up to the typical 100ms. NEC and Shoreline, however, claimed toll-quality voice at up to 600ms and 200ms, respectively. A follow-up with NEC showed that it really could deliver voice at 600ms (though not at toll quality), indicating the ruggedness of NEC's proposed installation. Avaya and Nortel, by contrast, were limited to under 20ms and 40ms, respectively, indicating that the engineering required on the line would probably be substantially higher than with the NEC or Shoreline solutions.Differences also emerged in how effectively vendors could deliver survivability. Other than NEC, all of the vendors proposed resilient voice architectures within Chicago, providing redundant call control servers that would switch over to the backup server in the event of a failure. Avaya, however, was the only vendor that claimed to provide stateful failover. Any hardware or software failures would be completely transparent to the end users and devices, so existing sessions would be unaffected. Although most companies claimed that existing VoIP sessions would be unaffected since they were peer-to-peer, Avaya claimed that its solution could also maintain digital-to-IP calls and ACD callers in queue.

NEC's proposal called for deploying NEAX 2000s in three locations-Chicago, Los Angeles, and Bradley-and then connecting them together via NEC's signaling protocol, CCIS. Deploying three NEAX 2000s would increase resiliency and scalability, both in the number of IP phones (every NEAX is limited to just 448 IP phones for 384 simultaneous conversations) and in distributing the additional traffic loads from the incoming call-setup requests across multiple sites. There were challenges with the NEC proposal, of course. Should a hub fail, for example, a network manager would need to manually reassign phones to a different hub to continue phone service. NEC says that by the end of 2003, that reassignment will occur automatically.

Out at the remote sites, all of the vendors proposed survivable architectures that deliver basic call functionality in the event that the main switching platform was unavailable. The extent of that functionality varies. All of the vendors proposed centralizing certain applications within Chicago, such as auto-attendant, unified messaging, and ACD. However, loss of connectivity to Chicago would render these applications unavailable at their typical service level. Voice mail would still be available, but users would need to dial a number, and they would lose their voice mail message waiting light.

NEC and Nortel both claimed to reduce the price of their bids through new remote site equipment that they'll introduce in July 2003. NEC cut its bid by pitching the NEAX 2000 IPS DMR, a modified version of its NEAX IPS DM, the company's traditional remote site IP gear. The DMR offers comparable functionality to the DM, but costs 30 percent less.

Nortel dramatically slashed its price through its new Succession Small Branch Servers. The Succession Small Branch Server is a WAN-extended media gateway just like its Succession Small Branch Office, but uses a third-party gateway from Mediatrix. The Succession Small Branch Office has a suggested retail price of $15,775. The Succession Small Branch Server comes in at $1,680; $2,480 when the price of the Mediatrix gateway is included. At those prices, the Small Branch Server with the Mediatrix gateway represents a savings of $212,720 when calculated across the entire project.Not that everything is for free. The Succession Small Branch Servers don't include signaling servers, so IP phones end up registering back to the Chicago office. The problem is that "any disruption of communications with the primary site can have disastrous consequences as phones will reregister locally, then bounce back in a ping-pong effect if the link is unstable," says Labelle. "For longer link outages over 14 days, the phones at the remote sites would be disabled as the branch office systems cannot function longer than 14 days without synchronization."

Of course, architecture forms only one consideration of a VoIP bid. Remote access and terminal functionality, as well as the sort of ACD and call center functions that vendors can provide, must also be considered. Only after a careful assessment of those issues, weighed against the business drivers, will ServiceNet be able to make its final decision at the end of 2003.

David Greenfield, international technology editor, can be reached at [email protected].

The ServiceNet Network

The ServiceNet network consists of 18 sites connected through a frame relay network hubbed out of its main office in Chicago. The frame network will add Compressed Real-Time Protocol (CRTP) and Frame Relay Forum (FRF.12) when VoIP is deployed. The other sites are connected to the frame network at a mix of port sizes: Ten sites are connected at under 64Kbits/sec, five at 128 or 256Kbits/sec, and three at 512Kbits/sec or above.The central site in Chicago runs a NEC 2000 PBX, with key systems of various sorts in the remaining offices. A total of 376 telephony users use a mix of analog and digital phones. Under the new design, respondents must replace these phones with IP phones at the knowledge worker, agent, and executive levels. Knowledge worker handsets need two line appearances, six speed-dial buttons, and a two-line display. Agent handsets need four lines, six speed-dial buttons, and a two-line display. Executive handsets need up to four lines, 12 speed-dial buttons, and a Web display for custom applications. The NEC NEAX voice mail system is also housed in the Chicago office and is configured with 12 ports.

Except for Chicago, each office has a star-wired, flat 100Mbits/sec Ethernet network based on a 3Com SuperStack 24-port managed hub. Most of the computers outside of Los Angeles and Chicago act as Citrix desktops, getting their configuration and functionality from Citrix servers in Chicago. The networks run TCP/IP and Windows 2000 servers, while the edge router-either Cisco's 1720 running IOS 12.1, or 3Com's NETBuilder II-connects the offices to ServiceNet's frame relay network. Some offices contain their own file servers; others gain file and print services through Chicago.

Chicago has a star-wired network over category-five cabling that spans two floors. A network design, approved as this RFP was issued, built the Chicago LAN around two Cisco 3550s (each a 24-port 10/100 switch with Gigabit uplinks) connected to a Cisco 3550 12T, a 12-port Gigabit Ethernet switch. Gigabit Ethernet cards will connect servers delivering file and print services to the 12T. The remaining application services run off 15 servers connected to one of the 3550s. The second Cisco 3550 connects to six hub stacks, a NETBuilder router for connecting to the frame network, and a Cisco 3000 router for concentrating VPN connections coming from the Internet via a T1.

ResourcesNetwork managers interested in participating in future RFPs, either as candidates or experts, or seeking additional information about these bids should contact David Greenfield at [email protected].

A seminar discussing the results of this RFP will be conducted at the TECHXNY show in New York on September 16 and 17, 2003. See www.techxny.com for further details.

Much work has already been published on VoIP. Vendor-sponsored test results can be viewed at www.miercom.com. Allen Sulkin does an extensive analysis of the voice-centric features in VoIP offerings at www.teqconsult.com.

Be sure to check out Westbay's Web site, which offers a wide variety of calculators for calculating VoIP bandwidth costs. Westbay also kindly donated the software that made the voice engineering of ServiceNet's network possible. Go to www.erlang.com.