Branch Office Management update from July 2004

Nothing magnifies IT efficiencies--and inefficiencies--more than remote sites. And despite some vendor claims, there is no silver bullet; remote support is complex and unique to your infrastructure and systems. But there are best practices for managing your far-flung branches, and here we'll share the lessons we've learned from our years on the job. Names are withheld to protect the (not always so) innocent, but the specifics aren't what's important. Rather, it's the concepts that are key.

Our philosophy: Managing remote sites is not about specialized products, but about maintaining standards and extending your organization's core infrastructure, policies and systems. Companies that let remote sites implement one-off technologies lose the cost benefits of standardization. Although migrations to more WAN-friendly technologies are expected, you must adhere to corporate standards.

The centralized IT group model is our focus here. There are other ways to support remote offices, but the centralized approach is by far the most prevalent, cited by 64 percent of our survey respondents.

The first step is to review your systems and infrastructure and develop a road map that defines where you want your remote support to be and likely obstacles. Sounds simple, but proper planning is often neglected. For example, unless your organization is a start-up, you've probably inherited an infrastructure and the systems built on top of it. Do you have older technologies that will impede effective remote-site management? Can you phase them out? Your road map should outline time frames for migration from legacy applications.

Remote support methods and tools should be part of your road map. For instance, locally run deployment tools may not have sufficient bandwidth when run across slower WAN links. And beware of implementing seemingly cost-effective tactical solutions for short-term problems. Anticipate when new projects will require contract renegotiations, communication line growth or technology changes. In this way, you'll avoid the costly redeployments and budget battles down the line.

[Lesson learned] Do it right the first time, or revisit the project again and again. A fast-growing company had applications running on a LAN, which worked adequately ... until remote sites started clamoring for access. The bandwidth required to run a key application remotely was cost-prohibitive. Luckily for this company, the application could run in a Unix environment with a true client-server database, thus limiting communications to screen updates and keyboard inputs. In this case, the solution was to migrate only the server platforms, but it could have been much worse; for example, the application developer might not have offered a client-server version of the program, or the underlying database might not have been cross-platform. This scrambling could have been prevented at system-selection time by ensuring the application was suitable for future branch expansion. Implementing the Unix version from the get-go would have saved data-migration, platform-conversion and reimplementation costs.

[Lesson learned] Require cross-functional teams for system selection. A manufacturing company bought a LAN-based sales-management program that included an update function for data synchronization. When the program was introduced at a location across the WAN, delays caused by slower links and occasional network congestion caused the application to abend. In this case, the software maker failed to include error-recovery routines sufficient to accommodate smooth WAN functionality--even though it advertised the product's ability to synchronize from anywhere. Moreover, tolerance for slower WAN links wasn't handled by error routines at the application layer, even though synchronization activities were controlled at that layer. While the software maker worked on the problem, the short-term solution was to boost WAN speeds or push the sync function closer to the remote sites.

This situation could have been avoided had the applications group included network team members in its software-evaluation process. Instead, this company application group made an assumption about suitability based on the software maker's claims, without proper acceptance testing by the network group.

Most IT people have been asked, "Can it be done?" Whether the subject is branch-office support or putting a sales rep on the moon, the answer is usually, "Sure--given unlimited time, money, gear and people." Since unlimited resources are a thing of the past, you should answer that one question with three:

In our experience, this threesome must always be in balance. How does this relate to managing remote sites? Here's an example: In calculating remote-branch support costs, you consider the obvious extended infrastructure expenses, but also conduct a risk-assessment analysis, including a review of failure points and how downtime will impact the business. You may find that after reaching a service-level agreement, driven by the business costs of downtime, the budget is inadequate to support the project. Time to send up a flare.

[Lesson learned] Things aren't always as cheap as they appear. A company decided to implement VPNs across the Internet for all its remote locations. The driver was WAN cost savings. Sounded great on paper, until the first ISP network outage. With no backup network, the company was dead in the water. Management fumed that IT had failed to assess the probability and cost of downtime, or to propose ways of mitigating the risk. Providing such information up front would have spared everyone a lot of trouble.

People are always the most expensive component of any IT endeavor. The decision of who to use and where is a difficult one--76 percent of our poll respondents said availability of local technical support is their biggest challenge. When determining personnel needs for remote sites, consider:

[Lesson learned] Share facilities to minimize costs. We worked with a company in which data was handled by the IT group and phones by the facilities group. Separate lines were provisioned and managed by each. Some remote sites actually procured their own lines locally. By combining the lines and renegotiating a single large contract, the company saved more than $1.5 million the first year in communications costs alone--all without changing the technologies. Simply sharing the channels more efficiently on the T1 lines and managing them from a single point provided huge cost savings.

[Lesson learned] Creative solutions can span technology hurdles and political camps. A company needed X.25 communications to deal with remote sites in technology-challenged areas. The network group had two factions: One favored more progressive IP-based systems, while the other insisted on tried-and-true X.25 systems. Arguments broke out. The creative and simple solution was to tunnel IP through the X.25 network where necessary to provide reliable transport and simplify transition to IP-based systems.

Long-haul communications services are one of the major components--and costs--of tying in remote sites. Without these services, there would be no communications, system access or real-time operations, so contingency planning in this area is crucial.

We've heard many newcomers to the networking profession claim that a VPN is all you need to connect remote sites. Wrong. Although a VPN may seem cost-efficient, without the proper business analysis and service-level agreement, you're operating on the lunatic fringe. Responsibility and accountability go hand in hand--your company's business (and your job) should not depend on a network for which no one is directly responsible. What levels of service can you commit to for Internet-based systems?

There's no way around it: IT managers supporting remote sites must do their homework on WAN connectivity. Address the potential impacts of design selections. Analyze traffic to identify current network-traffic flows and potential bottlenecks. Examine migration paths to newer technologies and VoIP convergence and factored them into the equation. Communications pipe sizing could lead you to a different technology. Mesh, hub-spoke and hub-hub-spoke models are all valid and driven by business requirements and the traffic flows they generate. Traffic analysis will also provide an opportunity for you to tighten the belt and ensure there's no unnecessary traffic traversing the WAN.

A hub-hub-spoke model is ideal for companies with overseas and/or regional traffic flows. Say manufacturer based in the United States has a multilingual, multinational network based in England into which all European Union countries are tied. Orders and consolidation information are relayed to U.S. systems for stock replenishment in the British central warehouse. A much smaller long-haul communications pipe is necessary between the British and U.S. sites--hence, the hub-hub-spoke model. Because the England-to-United States link is the most expensive, long-haul communications are minimized and special requirements (multilingual) are handled at the second hub.

[Lesson learned] Know before you go; look before you leap. We worked with a company that had outgrown its single building and decided to move the engineering/development group to a new site a couple of miles away. T1 lines were ordered, with bandwidth allocated based on an IT manager's gut feeling rather than on usage statistics. Day 1 was a nightmare: The bandwidth allocated was inadequate, and the engineering teams were rightly upset. Business came to a halt. The network team should have been allowed to monitor LAN traffic from the groups involved to determine the bandwidth necessary to maintain service levels across the WAN. Amazingly, the manager chalked it up to "working out the bugs" and wasn't disciplined. We would have considered that a career-altering decision.

[Lesson learned] Rush, and you'll go nowhere fast. One company was in such a hurry to complete its systems conversion, it chose dial-on-demand ISDN lines rather than wait for leased lines. Routers were set to establish connections on an as-needed basis. After implementation, the central network staff failed to monitor outgoing connections, and some keep-alive traffic caused unnecessary connections to be established every five minutes. When the bill arrived, it exceeded all estimates. In the rush to "get things done," costly router-programming oversights were made. Even worse, they weren't caught until flags went up in accounting.

Don't forget backup facilities when you're performing capacity planning for new lines or adding capacity to existing lines. And don't assume anything: If you're using a single carrier to provide both your primary and backup communications, get full disclosure on the circuits behind the scenes.

[Lesson learned] Redundant to what point? A nationwide company with centralized operations at corporate had multiple T1 lines for voice. One line was designated as backup for voice and data communications. Premises equipment was programmed to route traffic to the backup automatically when there were faults on the main circuit. Then, one day, it happened: A major T3 line went down. The premises equipment promptly changed over to the backup circuit ... which was also dead. The circuits terminated on the same T3, so the only real redundancy was in the local loop.

When managing remote sites, the less planning and streamlining done up front, the greater the need for IT staff intervention later. Some companies see remote-control software as the silver bullet for managing remote branches with limited manpower. But it's no replacement for proper planning.

Are end users well-trained? Are effective patch-management systems in place? Are users allowed to alter their own PC configurations? All these factors can affect how much those users will rely on your support staff.

So what's the best way to physically support remote sites? It boils down to time, money and resources again. Most organizations put their emphasis on a central IT staff, which costs less and uses fewer resources, but this group can become bogged down in unnecessary--and time-eating--firefights that could have been avoided with better training, standardized systems and controls. Some alternatives to a centralized support staff model:

One of the most surprising statistics from our survey is that more than 34 percent of respondents have no support, or only ad hoc support furnished by non-IT staff members, at remote branches. This neglect can lead to security problems and ineffective system use because non-IT staff will spend inordinate amounts of time on problems unrelated to their jobs, hurting productivity.

Linchpin of Support: Standards

Although not a direct cost consideration, the efficiencies you'll achieve--or not--in remote deployment and support hinge on company standardization. Nearly 45 percent of our poll respondents said standards compliance is the biggest challenge for remote branches. With standards in place, you can change to another vendor for any reason without worrying about knowledge-transfer, and support staff will be interchangeable.

And if PC hardware is standardized, hot spares can be maintained and swapped out by support staff to minimize turnaround times. Stock can be maintained at each remote site or centrally. Standardizing on a limited number of software configurations means disk-imaging software can be used to revert back to a working configuration quickly. A rollback can even be performed remotely overnight to avoid shipping costs and potential equipment damage.

In the same vein, if you have components that take a long time to order or install, an on-site spare should be kept ready. Also consider maintaining an "IT First Aid Kit" that contains common repair stock, such as power supplies.

[Lesson learned] Sometimes, the pieces absolutely, positively won't be there overnight. A nationwide company subscribed to the "ship it overnight" philosophy to route hot-spare PCs to replace troublesome remote machines. In one instance, three different shipments were made before the internal customer received a working replacement. Despite careful and appropriate packing by the IT team, PCs arrived damaged--in one case, manhandled to the point that most of the internal cards had worked loose from the motherboard! It took four working days before the user was back online. Consider keeping spares at each remote location, physical security and budget permitting.

In our survey, when we asked readers to rank remote-branch management tools in order of importance, documentation came in dead last out of 12 areas. That's unfortunate, because it's so easy--and inexpensive--to produce documentation, especially compared with the difficulty and cost of learning about a remote site. For example, central-office personnel may have never set foot in the branch they're supporting. Detailed information about the site, including reference materials, photos, diagrams, floor plans, communications outlets, wiring details and equipment lists, is invaluable. Products like Netviz can help organize and assemble this information, but it requires some effort.

Next to creating policy and procedure documentation, the most important component is information dissemination and company buy-in. In terms of remote branches, you must have ardent support, or it's a losing battle. To win this support, ensure that branches are given the same attention as local corporate departments.

Data Storage and Backup

Regardless of your applications and technology platforms, you must protect company knowledge and personnel investments with solid backup operations. You can accomplish this mission in several ways--for example, you can use centralized or decentralized servers. To determine what will work best for you company, answer the where, when and how questions:

Those with remote locations spread across the nation or the world face additional challenges. In our reader survey, 60 percent of the respondents said they have national branch locations, while 10 percent indicated they have international offices and 29 percent reported having a mix of both. Given the prevalence of remote offices, we've pulled together some best practices for managing far-flung locations.

National Challenges

International Challenges

A multinational corporation was working against the clock to build an international network. Unable to install fixed lines quickly enough, it opted for an ISDN network, which it believed would allow quick implementation (four to five days in most countries) and buy time for a longer-term solution with adequate bandwidth. This worked out fairly well in most countries. But when the company approached Telnor in Spain, the nationalized PTT said the ISDN switch was full and would not be available until someone relinquished service. Fortunately, the company was able to talk local IT folks from a sister unit into letting it share some WAN capacity as a stopgap measure.