Special Issue--IT Automation: Best Practices

The automation of IT infrastructure operations is one of those subjects that comes up for any CIO or data center manager at least once every year--at budget time.

Maintaining the data center, network and other aspects of a company's IT infrastructure is a necessary expense, but it also attracts attention because of its high costs. Not surprisingly, IT managers want to automate infrastructure operations and reduce the cost of labor.Sarbanes-Oxley and other regulations have also stepped up the pressure on IT. If you run a data center, you are not only expected to protect it and keep it running smoothly, but also required to keep extensive records on everything that has happened there. This is a major increase in workload at a time of diminishing budgets.

The idea of automating certain IT processes is certainly not new. Scripting--even JCL in the old IBM mainframe days--is one example. But as IT operations grow more complex, scripting wears out its utility. Scripts are proving too time-consuming to maintain and are often dependent upon one individual who understands their logic.

Point products, such as network tools, provisioning tools and job schedulers, also have failed to live up to the promises of reducing the cost of IT operations. In fact, in some ways they've added to the burden of the IT infrastructure staff. They're viewed in many shops as having limited functionality coupled with complicated, expensive deployments. However, the latest automation tools are a marked improvement over earlier generations. Most can provide an integrated view of the IT infrastructure.

But despite the reticence on the part of some IT staffs, increasing pressures on the infrastructure staff coupled with newer, better designed and better integrated products, means that IT automation may finally be ready to achieve its promise.

Continue Reading This Story...

Beginning The Automation ProcessIn today's data center, there are three major challenges that force IT management to think about automating infrastructure operations.

» Complexity. Compared to the mainframe days, there has been an increase in the diversity and complexity in the typical IT infrastructure, though for some IT managers this trend has occurred slowly and almost imperceptibly.

The data center is filled with server farms, independent disk storage and diverse networking equipment. A newer layer of "software infrastructure" also has developed, which supports service-oriented architecture and Web-based applications with things like application servers, messaging services, identity management and authentication servers.

This may mean that the data center staff hasn't had the time to completely document all its processes for all of these varied systems and pieces of equipment. And the staff simply may not have the expertise to know everything it should be doing. Complexity also means that when something seems to break, it may be some time before the root cause is identified and the problem is fixed.

Takeaway: Make an accurate assessment of your infrastructure to gauge complexity. Make sure your staff has what it takes to deploy newer automation tools.» Money. As noted, the pressure is on. "Contain day-to-day IT expenses and give our users valuable new services and systems," is the edict from above. Those day-to-day expenses, which end users pay for, either directly through chargebacks or indirectly, often eat up significant portions of the organization's overall budget.

If IT is directly linked to your company's bottom line, shift as much of your budget as possible into cost-saving tools and away from everyday maintenance activities. That's the best way to win friends in your enterprise. Of course, you also need to take on the challenge of showing you are a competent financial manager--that you can get a good return on your investment. Depending on your site, IT automation may provide a quick and positive return on your investment.

If you have SLAs for the network or servers, IT automation tools can help to prevent situations where financial penalties will be assessed against you, also.

Takeaway: IT automation may be a good way to enable a shift of resources, away from maintenance and to providing new resources for users. How soon can you construct a way to prove ROI and keep IT allies that are higher up on the totem pole?» Expectations. It's also important to manage the expectations both within and beyond the IT department. The expectations for IT departments have never been higher. Users expect IT to control spam and viruses, provide 24/7 uptime, and to constantly oversee the data demands of regulations such as SOX (Sarbanes-Oxley), PCI (Payment Card Industry) and the like.

Particularly with SOX and some other regulations, there seems to be an expectation on the part of regulators and C-level execs of immediate and total compliance for all devices on the network. The ability to see a variety of reports about the state of compliance is often another expectation.

Aside from any formal SLAs at your site, the reality is that users expect uninterrupted service. You want to do whatever will help ensure that the performance and services that users expect will be delivered.

Takeaway: Expect automation tools to help you meet business and user expectations.

So, Why Not Automate?These three challenges are driving fresh demand for IT automation, but they also are the cause of resistance to the idea as well. Here are some reasons why data center staff shy away from IT automation.

» Complexity. The data center environment is way too complex for you to believe it can all be successfully automated and you are unaware of a set of best practices in IT automation that you can follow with confidence. This is new and scary territory. It will take a lot of time for data center staff to document and then program into the automation tools all of their processes. And they may not know all the rules that should be programmed into these tools.

» Money. In the past, implementing these tools was a long and expensive process. Some data center staff will worry about being out of job if these tools succeed and the budget shifts away from data center activities.

» Expectations. With all of the compliance requirements added in recent years, there is no time to take on yet another project. You worry that automated tools will increase your risks because the software could make some drastic and immediate change that will backfire with users.

Picking Your TargetsAs you go through the process of juggling pros and cons, there are fundamental questions that have to be answered:

» What IT processes and activities make sense to automate?

» What are the best first steps to undertake IT automation?

» What functions does it make sense to automate?

You can look at what can be automated from a variety of perspectives. Some vendors focus on particular domains, such as servers, network, storage, desktop and end-user technology, security, software infrastructure (such as database management systems) and applications. Others try to take a comprehensive approach.Some vendors focus on functions that apply to these individual areas:

1. Provisioning and base installation: These are not necessarily challenging tasks, but they are tedious. Depending upon the range of choices in an IT automation tool, there are several different ways of provisioning. Some are based on prebuilt models, some are hard-coded by your staff and others use an existing server as a template.

2. Patching and applying software updates: Performing this kind of maintenance to operating systems and network devices is another tedious IT task. In addition, in this era of zero-day exploits, patch management has turned into much more work--and more important work--than any of us anticipated.

3. Discovering and maintaining an up-to-date, accurate inventory of assets: This function, in turn, helps with license and maintenance fee management.

4. Job scheduling: While this was one of the earliest tasks to be automated, it continues to be an issue. These days this function extends to desktops and other devices beyond the traditional scope of scheduling of batch jobs on servers (or mainframes).5. Problem identification and troubleshooting: This is a critical function in the data center and means that the IT automation tool must understand the behavior and characteristics of all devices and systems on which it focuses. If that intelligence is not built in, then your staff or consultants will have to do so and that can be a lot of work. Problem identification also affects users, either by offering self-help or guidance for the help desk.

6. Problem reporting: Although it's an important function, it can end up being nothing more than a yet another alert on the screen of the data center staff. Or it could result in the automatic generation of a trouble ticket into your tracking system, such as Remedy. There is an obvious downside to IT automation tools that report problems--information overload. After a certain point, the alerts, events, and configuration changes just fly by on the screen.

7. Automated problem resolution: See above. Ideally, an IT automation tool would have enough intelligence to figure out a problem exists and then fix it automatically. A tool might execute a plan of action if certain conditions are met--whether or not the tool "knows" what the problem is. If it cannot do that, at least it could perform some basic fixing before turning the alert over to a human operator, while documenting all the relevant background details that caused the alert. This information is particularly helpful when it comes time to address the problem with the vendor of the offending software.

8. Detection of noncompliance and enforcement of compliance with standards and regulations: The best IT automation tools should start out with a good set of tests that encompass best practices.

9. Quarantining: This is one way to enforce noncompliance or eliminate a potential security threat. If the problem cannot be fixed, at least it can be isolated. Whatever is causing the problem--server, desktop, application--may have to be isolated in order to fix it. This is an area where computers can prevent harm to the system faster than any IT personnel.10. Change management: Another task that can be readily automated. Although it is typically thought of in relation to documenting changes to satisfy various regulatory requirements, effective change management goes beyond this. Within the IT infrastructure, change management is even more important in order to ensure that all the implications of a change are handled properly.

Areas Of Automation

To help determine which areas of your infrastructure to automate first, ask yourself these questions: Does the product fit the complexity in my data center? Does the product provide a decent ROI? Will it help us meet user expectations and create a less stressful work environment for the staff?

It is tempting to try to get the maximum return by automating as much as possible, as soon as possible. Yet, even the vendors with the most ambitious and comprehensive products, such as BladeLogic, IBM, Opsware, note that the most successful early adopters of IT automation start by tackling smaller projects. So where should you start?

Clearly, there is no standard answer for everyone. In addition to recognizing the limits imposed by the expertise of your staff, your answer will depend upon the mix of equipment you have and the quality of coverage for that mix ideally provided by each IT automation product. The answer will reflect your current areas of concern, along with expenses. It will also be based on which products have strengths in the areas where you can re-engineer your processes to garner the best possible return.Some vendors--such as Opalis Software and RealOps--emphasize "run book automation." In some ways, this is meant to appeal to more conventional mainframe operations managers. Whether run book automation is a good early target for your data center will depend upon whether you think your run books are adequate. If the run books are not adequate, then does the product provide a ready made cookbook that you can quickly adapt? If the products don't provide a good substitute, there will be lots of work to create the run books. Creating run books is certainly not a bad thing to do, but will add to the implementation time and significantly reduce the short-term ROI.

Because network monitoring tools from major vendors are fairly well entrenched at many sites, it's common not to focus on automation in this area. Companies like AlterPoint argue that this ignores a key benefit of their automation tools: the capability to save IT dollars by identifying over-provisioning of network equipment and incorrect assessment of maintenance fees by network equipment vendors.

Before you decide to focus on just one function or area as an initial step, consider the most significant benefit of the latest generation of IT automation tools. They provide an integrated view of your IT infrastructure and may ease deployment duties for staff who do not have the time and ambition to master several different tools.

The integration provided by some automation tools not only reduces the total number of tools, it can help identify problems that cut across different areas of the data center. For example, it can identify when network changes will affect applications or when server changes will cut off access to important network ports.

As a result, companies like Opsware and Optinuity have made an integrated product suite the basis of their market strategy. Opalis offers an "Integration Server." Other companies have teamed up to compete. For example, AlterPoint, which specializes in network automation, has alliances with BladeLogic, EMC and RealOps.The Result

Ideally, automating parts of the IT infrastructure will free your staff to develop higher level services deserving of their time and expertise. It makes more sense to have a highly skilled IT staffer work on innovations than on change management. That's the idealistic view. It's also true that you may need fewer IT staffers. Not everyone will be up to the challenge of moving away from low skill functions.

It's also true, however, that there will always be parts of the business that cannot be fully automated. You'll still need your experts to be around when problems fall out of the standard templates that your IT automation tools are handling.

Your quest to lower IT costs and the burden of the IT staff through automation will require you to address all these areas of concern to maximize the benefits and ROI. And it will be an ongoing process. Just as we are never really done automating the business operations of our companies, it's also true that we'll never really be done finding new targets in IT operations to automate.

Not In My Backyard Either

In fairness to the companies that have been developing these tools, there is another side to this resistance because, when it comes to IT automation, the usual scenario is reversed. Generally, it is the IT people who are encouraging non-IT employees in their workplaces to adapt to change and use technology to become more productive. When it comes to automating the data center or other IT processes, however, IT people are the users. Paradoxically, they will often react the same way that non-IT users react when the IT staff come calling with some new idea to automate business processes--and that's not necessarily with a warm welcome. Indeed, it has become something of a dirty little secret inside IT that many technologists themselves are often resistant to using technology to become more productive. Given the ever increasing pressure to justify IT spending, this is simply not a sustainable position.Determining The ROI

Financial Pressure on the IT budget is one of the four big challenges that motivate people to investigate IT automation tools, and demonstrating a ROI (return on investment) is a key to getting approval for new automation tools. Voyence, among others, provides a neat ROI calculator on its Web site; try running your numbers.

As one might expect, the vendors of IT automation tools have fantastic success stories for those seeking a quick ROI. Payback time is touted as months, not years. In some areas, like network management, the investment is supposed to pay for itself within a few weeks.Automated network tasks, according to one vendor, take five minutes, compared with conventional approaches taking hours. Another vendor shows savings in various server tasks ranging from 83 percent to 99 percent. It claims patching 1,800 Unix servers took only two hours, for example, versus 400 hours before its tool was used.

One indication that the ROI really is positive comes from those folks whose business absolutely depends upon running an efficient data center--the outsourcing vendors. They have been relatively early adopters of these tools. In a story from July 2006, one such company started out with an admin-to-server ratio of 1-to-20, but had by last year gotten to 1-to-150 and was planning on 1-to-400.

As always, you need to look at the total cost of ownership--including initial setup, your staff's time, and continuing maintenance--and expected time before the product can be used before you begin to estimate ROI.

Dr. Norman J. Jacknis is the first CIO of Westchester County, NY. He has had extensive experience in the private sector, most recently dealing with enterprise middleware software products. Write to him at [email protected]