Effective Change Management

1. Communication

When making changes to production hardware and software, communicate those changes and their associated risks to all involved, from end users to IT management. This may be the biggest key to success. Effective communication between users and IT helps to identify conflicts and prevent confusion. It gives the user community better control over the impact downtime. And open communication helps users understand the efforts required to maintain an efficient infrastructure and makes them feel they're part of the team.

2. Authorization

Form a team of user representatives, business line managers and IT admins to authorize each change. It's important to recognize that no single administrator knows every aspect of the business for which he or she works. It would be a stretch to expect a network administrator to know that the company's fiscal tax season was coming to a close or that the CEO was showing the SEC a set of corporate papers on the same night IT planned to shut down a series of production systems for routine upgrades. By mandating a series of checks and balances for change authorization, the business empowers itself with the ability to prevent change at inopportune times. Clearly, the business must take into consideration the need for system updates; however, IT must also take into consideration the fact that there would be no IT operations if there were no business.

Policies should be in place to identify which users can authorize changes and which users can actually make them. As part of the CM documentation process, IT management should evaluate requests, determine the resources needed, set priorities (often with the assistance of a change-control board) and provide cost estimates to user management. Once user management has approved the costs and any risks associated with the change, IT management should assign the appropriate staff and schedule the change.Authorization of changes should be documented on a standard request form that details the changes, provides appropriate justification and requires a predefined level of user management approval. These forms should also provide for approval by IT management and should be retained to serve as evidence that the change was authorized. There should be documentation that test results were reviewed and approved per policy.

As we learned in our reader poll (see charts), change is frequently authorized by technology owners, who are also the ones making the changes, and not an appointed change-management board. In fact, only 19 percent of survey respondents use a change-control board. Clearly, in many organizations, technology owners have been granted excessive authority for implementing change.

3. Approval

Ensure that business and IT management, separate from the CM team, approves all changes before they're placed in production. Approval should be based on satisfactory test results, evidence of thorough consideration for the proposed change, and an understanding of how the change could impact the productivity and security within the organization.

Change requests can be made on simple paper forms or complex electronic tracking systems. For small environments, with few tiers of management and even fewer systems administrators, paper request forms make complete sense. In large or complex organizations, an electronic CM system is probably worth implementing. A well-constructed system can route requests through various pathways based on predefined policy (for a unique take on a CM tool, see "Talk Quick and Dirty to Me,").A request to place a new file server into production might pass through three or four tiers of approval, including the division manager, the information security group, the architecture group and, finally, the backup-and-recovery group. Further, the system might even track the change throughout its implementation.

Whether paper or electronic change management is more appropriate for your enterprise, one thing is certain: Some form of approval should be implemented.

4. Documentation

Provide appropriate formal documentation of the authorization, testing and approval process, and ensure that all relevant system, operations and user documentation is updated as necessary. Accurate documentation is a key component of successful IT organizations and especially critical to any CM program.

Many variables must be taken into account when change is being considered. One (or more) of those variables is often represented by internal documentation. Misleading or erroneous documentation can, and often does, cause false assumptions to be made, putting the integrity of change projects at risk. We frequently work with entities that disregard their documentation, often omitting such items as wireless access points, modems and additional Internet points of presence. By failing to document these elements, change requesters may fail to identify a potential hazard and unnecessarily put the enterprise at risk.For CM, though, documentation is a two-way street. Once changes are implemented, and often even before, all necessary documentation must be updated to incorporate those changes. Otherwise, the people making the changes will be the very ones responsible for jeopardizing the success of the change projects.

5. Security

Set up physical and logical security to prevent unauthorized changes to your infrastructure. Many businesses live and die by the integrity of their information. Implementing a change without fully understanding the ramifications of doing so could place an enterprise--and its data--at risk. Changes should always be approved by the information security group prior to implementation. This is accomplished by placing the infosec group directly into the chain of approval. By doing so, potentially hazardous mistakes and oversights can be avoided.

6. Testing

Test changes adequately before deployment. The same rigorous testing that occurred during the initial design and implementation of an enterprise's infrastructure should take place prior to making changes to devices that support the production environment. Essentially, this means that IT and user personnel need to perform testing sufficient to ensure that the changes produce expected results. Discretion is required here, since some changes inherently represent greater risks than others. For example, changes to an operating system that supports a critical business application will require more testing than the replacement of a defective NIC. Where appropriate, a properly architected change-management program will also invoke procedures for application source-code assessments and the like.7. Scheduling

Schedule all changes and communicate the schedule to users and IT management. There should be procedures for scheduling changes to production systems or network devices in order to minimize disruption, confusion and the possibility of erroneous processing. There also needs to be a method for communicating this schedule to the users and IT personnel who will be affected by the changes. One common method of reducing the inconvenience changes can have on an enterprise is to group multiple changes into "releases."

This release concept, used widely by hardware and software vendors, reduces the steady volume of changes into a number of regularly scheduled events, either weekly, monthly or quarterly. Problems with this method include potential delays for critically needed changes and a larger potential for post-change problems with a corresponding increase in troubleshooting difficulty. In addition, a single cancellation because of business demands can throw critical changes on the back burner for an extra week, month or more. In such cases, there must be protocols for requesting emergency updates.

8. Emergency Procedures

Determine appropriate procedures for handling emergency changes that occur when prior authorization and approval cannot be obtained, as for a patch to block a new vulnerability. This is often referred to as a deviation standard. Also, in many organizations, information processing runs 24/7/365. As a result, there are very few times when change can be considered opportune. In these situations, changes must sometimes be made quickly, and completing a full change-approval process may not practical. Knowing that these events are inevitable, deviation procedures are essential.Deviation standards help define the when, who and how of reacting to an emergency event.

• "When" helps administrators and managers identify when a deviation is appropriate.

• "Who" defines who can make the call to permit a deviation, and who should provide verbal approval from both business and technical perspectives.

• "How" helps an entity determine which procedures are required before action is taken. Although the entity might be implementing changes in an emergency fashion, it doesn't mean that changes should be made ad hoc. It just means that the company is accepting more risk on the side of the change to prevent a potentially more costly result of not making the change fast enough.

Control elements that must go into a deviation standard include:• After-the-fact review of the change by management;

• After-the-fact testing, depending on the change; and

• After-the-fact entry into helpdesk and CM systems.

9. Segregation of Duties

Make sure that the people who request changes, authorize those changes and place them into production don't overlap with one another. The accounting industry has recognized a need for duty segregation for decades, but only recently has IT seen the light. For effective control over change in the enterprise, no one person should be permitted to request, approve and implement change. Permitting such autonomy opens the door for a number of potential problems.• The person making the change may not fully understand the needs of the enterprise, and may make changes that aren't in alignment with those needs.

• A dishonest employee could create back doors into the network, even into applications.

• If security concerns aren't fully understood or taken into consideration, an implemented change could open the enterprise to financial loss, a loss of reputation or legal liability.

Even the most prudent efforts to segregate responsibilities could be made null if employees from multiple groups collude to make change that could cause harm. The Sarbanes-Oxley Act of 2002 had this in mind when it forced sign-off from executives stating that internal controls had been implemented and audited, thereby holding those executives liable for misleading or fictitious information to prevent collusion.

10. Back-0ut PlansEnsure that all proposed changes can be reversed in the event of unforeseen negative consequences. As mentioned earlier, there are many variables to consider when dealing with large, complex environments. Outdated applications, unsupported OSs, custom-coded applications and complex security filters can turn even the most outwardly simple change into an enterprise denial of service.

To help reduce the effects of these negative consequences, include a well-documented back-out change procedure in the approval process. Back-out procedures might detail backup-and-restore procedures, or might be as simple as taking a Ghost image prior to change. Regardless, the procedure should detail step-by-step instructions. After all, by the time you've decided to implement your back-out procedures, you might be under a lot of pressure to get back online.

11. Closing Change

Finally, verify that all required change steps have been completed, and that all information concerning the completion of the change is recorded. Change management is a full life cycle, starting with a request and ending with a review of the entire process to validate that all necessary steps have been followed. Through post-change analysis, enterprises can continually improve their processes. They can also identify areas that may have gone unchecked, so that they might remedy the oversight before the business suffers.

Kevin Novak is the COO for Chicago-based security consultancy Neohapsis. Write to him at [email protected].