Network Monitoring Tool Eases Troubleshooting For Utility

Bristol Virginia Utilities (BVU) is the utility backbone for residents in a 125-square-mile area, providing electric, water, wastewater, cable, telecommunications and fiber-optic broadband services. The utility depends on a robust network to facilitate its operations. In the area of telecommunications, this requires a well-constructed network with the ability to real-time monitor and troubleshoot any situation that can potentially impact the 24/7 quality of service (QoS) BVU sets as its performance goal.

BVU's network is hooked into a 90% virtualized bank of HP servers. Each individual physical server runs six different virtual machines, all using VMware as a hypervisor. The utility divided the servers into two separate data centers that are eight miles apart from each other and tethered the primary and secondary data centers together with SCSI connections, says Stacy Evans, manager of network engineering at BVU.

“The relative proximity of the data centers enabled us to set up data mirroring and real-time failover capability between the two sites so that there was virtually no recovery time lag in the event of a need for failover," he says. "In addition, we set up a tertiary data center that was located 65 miles away from the primary and secondary data centers. This gave us another data center failover option in the event of a disaster that would affect the entire region that our main data centers were located in. We connected to this third data center over a wide-area network."

The WAN uses equipment from 40 different vendors. BVU uses Multiprotocol Label Switching (MPLS); the process simplifies routing table lookups and contributes to the quality of service the utility needs to transport data, voice and cable services to schools, banks and other high-power users within the 475-mile radius of its fiber-optic network.

Covering such a broad area meant network troubleshooting was becoming time-consuming and costly.

“When you have a 475-mile fiber loop that serves several different communities, our network staff has to cover a broad area,” says Evans. “There was a time four or five years ago when we had to send someone onsite to packet sniff in a location where we were experiencing difficulty. The technician would take an analyzer with him and then bring back a sample of network traffic that we would analyze here at the central office."

The business case was simple and straightforward: BVU needed a way for central staff to monitor and troubleshoot events occurring throughout the network in real-time, without having to dispatch network specialists from headquarters to diagnose problems.

[A recent survey of IT pros showed that newer technologies are making it hard to keep tabs on the network. Read the study's findings in "Cloud Services, BYOD Complicating Network Monitoring, Survey Shows."]

After considering several network monitoring products, the utility chose Network Instruments’ GigaStor. The monitoring tool scales out to work with network probes that Bristol network engineers place anywhere they want in their network, capture history of network performance for playback and analysis, issue alarms based on BVU's quality of service criteria, and recommends resolutions to problems.

“We had single-pane-of-glass visibility of traffic, malware and other elements important in maintaining our QoS standards,” says Evans of the deployment. “Just as importantly, we didn’t have to send staff out to remote sites to see what was going on with the network. This meant that our network engineers could spend more time on projects because our times to problem resolution were shorter."

The tool also saves money because BVU didn't have to send so many technicians on service calls. "The average service call in the field was costing us an average of $75 per incident," he says.

The investment in network monitoring has allowed BVU to keep its staff of eight network engineers and 35 support center personnel constant, without needing to hire extra people. “We keep a steady number of subcontractors on board for special projects, but we haven’t needed to add to staff,” says Evans.

Evans said that BVU's network staff has the ability to get as granular as it needs to with its network troubleshooting.

“We can see the impact of IPV6 traffic on our infrastructure, and narrow down our analysis to a single pipeline or even a single stream of data,” he says. "We also have the ability, based upon our technical needs and our budget, to scale out our network probes to more points in the network as we continue to expand.”

The ability to rapidly scale out analysis probes and dig into the root cause of any performance problems as the network grows is vital to BVU, which now plans to extend its reach to Johnsonville, Tenn., and could see more expansion ahead.

“Today, our primary customers are business and government, but we could be in more residential markets in the future,” says Evans. “We might even see ourselves enter into additional service offerings such as IPTV [Internet Protocol television] or automated utility meter reading [AMI], which would be delivered by the network."