Triage, IT Style

However, while Children's had executive-level ownership for its ERP project along with a market-leading software vendor and a respected integrator (Arthur Andersen LLP, which has since been bought by KPMG Consulting and renamed BearingPoint), the hospital relied too heavily on these partners while critical IT project responsibilities did not receive the attention and representation they required. That's because the primary project focus was the business flow and how the application would support it. In the understated words of Children's CTO Scott Ogawa, IT's input "was somewhat devalued."

Indeed. Many key members of the Children's IT department--called ISD (Information Services Division) internally--were shut out of ERP planning until May 2001, fully 18 months after the start of the project and just one month before equipment needed to be ordered. Only when the business information was gathered and development was ready to begin did IT get pulled fully into the loop. And we do mean fully--Andersen shifted the focus solely to IT. Ogawa recalls, "Andersen said, 'IT is up to the plate. IT is holding up the whole ball game now.' "

Because it did not have enough representation with the integration partner, the Children's IT staff was forced to scramble, quickly assessing hardware requirements and ordering equipment without knowing exactly how it would be used. Says Ogawa: "The vendor had a big say in the sizing recommendations." Then, the hardware sat idle for two months because of the false sense of urgency. Project confidence within IT suffered, especially as related to Andersen, which had experience rolling out PeopleSoft applications and on which the Children's IT group relied heavily for guidance.

"I knew some things were going on in the background, but I didn't get pulled in until they needed network access," says Children's Network Manager Mark Hourigan. If he had had time to plan and architect a solution properly, Hourigan says, he would have done things differently. Take the VLAN configuration. He did not know at the time how the development, testing and quality-assurance server environments were being built, so he was unable to segment the production servers to optimize them for the launch.

Hourigan wasn't the only IT person sent scrambling to meet PeopleSoft's and Andersen's requirements. Technical Services Manager Kevin Murray and his team are responsible for the hospital's Windows NT environment and helpdesk. Murray took the initiative and pressed for information about the Process Scheduler component of the PeopleSoft 8.0 architecture. To his chagrin, PeopleSoft was less than forthcoming about the size of the systems required to carry out the Process Scheduler job, he says, perhaps because Children's was one of the first large clients to run this Web-based version of the application.

After playing e-mail tag for awhile, Murray says he was eventually told by a PeopleSoft representative that a "beefy PC" would do the job; however, he was not convinced and asked PeopleSoft for specs in writing, which he eventually received. Still, he was not convinced one box would meet their needs. Turns out he was right--that "beefy PC" has become two Compaq DL580 servers, one running production human resources and financial modules, the other running development applications.

But the story doesn't end there.

In less than a year it became necessary to break the production box into dedicated HR and financial boxes, Murray says. If Children's had not questioned the recommended configuration and had gone with the single PC, it would have found itself rearchitecting just before the go-live date.

Some units did not fare as well. Stuart Cohen, an ISD project manager who got involved with the ERP initiative in May 2001, says the core Sun E450 servers were undersized, as was the storage environment.

Data and user volumes were miscalculated and the hardware was undersized, but tuning the software and database have kept the ERP system running smoothly, Cohen says. Despite the problems surrounding the hardware configuration, he says he was confident that IT would deliver a production-quality environment. "The concern was never ISD not delivering," he says. "It was the Andersen reliance on business flow."Concerns about Andersen and PeopleSoft notwithstanding, IT was tasked with making the April 2002 go-live date a reality for the entire institution.

One of the biggest hurdles Murray and his team faced was that, even though PeopleSoft 8 is a Web-based client architecture, there were still desktop considerations (aren't there always!). In this case, it was a rollout of Microsoft's Internet Explorer 5.5--which PeopleSoft 8 requires--to 3,000 desktops in three months, replacing Netscape Communicator.

The hospital's standard desktop platform was more than adequate to support the additional browser, but memory and rogue DLL conflicts were a concern. Murray's group performed extensive compatibility testing between core hospital applications and new software and started deploying IE to the desktops using Microsoft's Systems Management Server. The installation went very well but, no surprise, the helpdesk started receiving calls immediately.

"All of a sudden, an IE icon appeared, and people wanted to start using it," Murray says. "Users wanted to know if the default browser were changing and if so, why. We had a whole smorgasbord of 'What's this?' calls." But once Murray and his team explained to users that the changes were related to the ERP project, the calls subsided.

Helping users understand why the ERP project was happening in the first place, though, was a major concern for Children's. Posters, internal e-mail messages and director-level missives got the word out, but the focus was on the big picture. Also, Andersen was coordinating the initial user training, but Murray was concerned that little attention was being paid to user support post-launch. The hospital's existing helpdesk needed to be integrated into the long-term support of the ERP application. Training and policies needed to be established so calls could be answered properly from Day 1.

When the issue was brought up in meetings with Andersen, it was paid "lip service," says Murray. Finally, Children's assembled a multidisciplinary group--an "A-team" of ERP support comprised of eight to 10 human resources, finance and helpdesk personnel who provided end-user support during go-live. Since that date, the A-team has received nearly 700 phone calls asking for help and information.

This group has yet to be integrated with the hospital's helpdesk and does not use its BMC Remedy trouble-ticket system to track calls. Fortunately, users have gotten more comfortable with the ERP system, so the group has shrunk to one or two "Jacks-of-all-trades," Murray says.

Until this group is "triaged" into the helpdesk, he says, Children's has two support models: one for its PeopleSoft application and one for every other hospital application.

Down and Dirty

While the users and hardware were being prepped for the launch, Leo Judge, Children's manager of financial applications, was doing the real dirty work--the data conversion. Phase I of the PeopleSoft rollout involved replacing every legacy system for HR, payroll, accounts payable, general ledger, asset management, purchasing, inventory and materials management. Quite an undertaking, especially considering there were no fewer than four data formats in the mix: Oracle, RBMS, SQL and Pervasive Software's Btrieve.

While Andersen focused on building new business processes and logic for the PeopleSoft application, Judge's team performed data integrity checks on all the databases being ported into the new Oracle database. Judge had four people from the Children's application development team to assist his group in the conversion process.

"It was a very positive experience," he says, though "some issues arose" before the launch. For example, Judge says his team did not receive timely training on the new application architecture; this left them dependent on consultants longer than originally planned. Part of the problem: The team was stretched thin trying to support those pesky legacy applications. He also says that during the rush to meet the April deadline, Children's was not able to perform adequate end-to-end transactional testing, so performance bottlenecks could not be identified until the application was in production. This caused serious problems with report generation and database performance.

"Very little assistance from the vendor" was forthcoming when it came to database tuning, Judge says, so Children's ended up dedicating a database administrator to the PeopleSoft application for an extended period after the launch. Judge cited one payroll report that took 12 hours to complete before tuning. Since then the DBA and Judge's team have been able to cut report generation down to 90 minutes. Judge stresses that you can never have enough DBA talent for a project of this scope.

The app dev team had its own set of challenges. As far as Children's could determine, this project was the largest and most complex ERP rollout any medical facility had ever undertaken. During the planning phases, Children's could not find another hospital that had replaced all its core business applications simultaneously, making it difficult to predict the true scope of the project. And to add to the uncertainty, Children's was using PeopleSoft's first Web-based client release. Predictably, the implementation was buggy.

"Patches were coming out almost weekly," says Cohen. "We could not stop development to upgrade." In lieu of patches, they were forced to create workarounds. They performed patch management by hand, developer by developer. With so many application modules going live at the same time, tracking patches was difficult. (If you find yourself in patching hell, see "PatchLink Helps Keep Windows Closed.")

When Children's froze the code for the new application pre-launch, it fell so far behind in applying patches and bug fixes that PeopleSoft threatened to stop supporting the code. Ultimately, though, "the teams and personalities worked well together," Cohen says, and the code was stabilized in time for the April go-live date.

Fat Pipes, Will Travel

A problem that has plagued many an ERP implementation is insufficient network resources. Those "fat client" architectures of old never lent themselves to running across WAN links and to underpowered PCs and workstations. However, because Children's was using PeopleSoft's first Web-based client architecture, there was no need for a major network overhaul.

That was just fine with Network Manager Hourigan, who was already busy with a hospitalwide infrastructure upgrade that would support not just the PeopleSoft project but business initiatives yet to come.

At the time, the hospital's existing network backbone was a mesh of the now defunct Digital GIGAswitches running on 100-Mbps FDDI links. Hourigan and his team were building a new network core based on Cisco Catalyst 6000 Ethernet switches linking at 4 gigabits per second--40 times the capacity of the old network. When Hourigan was approached to monitor load testing of the ERP application, he understandably concluded that "bandwidth is not the issue." In fact, the SolarWinds.Net SolarWinds application used to monitor traffic between clients, Web servers and database servers barely registered the testing.

However, Hourigan needed to verify that the PeopleSoft application would work across its modem dialup, VPN and WAN infrastructures. Web-based or not, there were still concerns that the application might fail outside the hospital LAN. Testing showed that the application ran well within the usability requirements in all areas, albeit with a small performance degradation at remote offices. As the hospital expands the PeopleSoft application to 7,000-plus employees, Hourigan expects to see the network load grow but is confident the Cisco architecture will meet the demand.

The Big Day

Come go-live time, the Children's PeopleSoft implementation team focused on all the standard user items: application training and frontline support for trouble calls. What it missed, Murray says, was a golden opportunity to minimize user confusion about why people were being made to move to a new way of doing business. "Training is an opportunity to disseminate information about the direction of the hospital," Murray says, and to answer questions like "Why are we doing this project?" Training "is more than just mouse clicks and screen shots."

When we asked Murray if he'd ever been informed of the business initiatives that made the PeopleSoft project so critical, he said he had a pretty good idea, but added that most people were forming impressions on their own. He says executive management's understanding of the project goals never made its way down to the masses.

Clear from the start, however, was that the days of buying best-of-breed applications and customizing them beyond recognition were gone, CTO Ogawa says.

It was decided at the outset that the initial rollout of PeopleSoft 8 would maintain its out-of-the-box look and feel. That decision helped the hospital meet its go-live date but created some user backlash.

"We could make it pretty, or we could stick to our philosophy ... how it comes out of the box is how we're going to put it in," says Ogawa. "Maybe its not intuitive, maybe its cumbersome, but you're running the business."

Investing in numerous customizations would have directly impacted the implementation cycle of Phase I, so Children's is planning to add new functionality and improved navigation in Phase II--or ERP2, as it's called internally.

Many of these improvements can't come soon enough for users. "Not intuitive" and "difficult to navigate" are two phrases we heard repeatedly from Tanya Johnson, Ogawa's administrative associate. Johnson uses the new applications every day but says she has yet to see increased productivity. Moreover, Johnson says she saw many problems coming when she attended training before the launch, but Children's lacked a formal feedback mechanism. "I look forward to change," she says diplomatically.

Help is indeed on the way. Children's has formed a PeopleSoft Advisory Group to help prioritize user feedback and requests. The group comprises senior-level managers, one level below senior executives. (The reason for this cutoff, says CIO Dan Nigrin, is that senior executives lack insight into users' day-to-day needs.)

"You need to drill down to the folks who have more interaction with the system," Nigrin says. In fact, under Nigrin's direction the Children's technology group is soliciting the opinions and advice of both senior- and junior-level businesspeople in determining how to meet the needs of the hospital as a whole.

Back to the Future

The plans for ERP2 are well under way (see "Next Steps," left, for a preview, and keep an eye out for our followup stories in future issues). New application modules are due to be installed later this year, and navigation is being improved incrementally.

One new application is eBenefits. Due to go live later this month, it will allow for all 7,000 Children's employees to select their personal benefits coverage online; the PeopleSoft system is currently serving a mere 100 to 200 users in HR, finance and purchasing.

Opening the system up to all users will undoubtedly stress it, but the IT group is applying what it learned during Phase I to prevent the problems from reoccurring. We expect Children's IT group to succeed in implementing ERP2 because its ties back into the business have never been stronger--it's a case of "cultural ROI," as Nigrin puts it. We asked the CIO what he considers the critical component to his team's future success: "Whatever the industry, it's important to make sure your IT people understand the focus of the whole business relationship and make sure their work supports that."

James Hutchinson is Network Computing's director of editorial content. Write to [email protected].

When IT embarks on a project that affects an entire user base, it pays to have established a reservoir of goodwill.

With that in mind, Children's has instituted a program sure to help polish IT's image in all corners of the organization. Dubbed the "Buddy System" by Technical Services Manager Kevin Murray, the program encourages IT staffers to "adopt" a floor or area of the hospital. Each participating IT employee then gets to know his or her adopted users, listening to their needs and helping solve any technology problems that arise.

"Just having people walk through the hospital and identify themselve--'I'm from IT. Can I do anything for you?'--has led to incredibly positive feedback," says CIO Daniel Nigrin. Because Nigrin is also a staff physician, he speaks from experience when he says, "I'd get frustrated when I would come to a floor and walk up to a station and the keyboard was disconnected or the mouse was missing." Users in the area--doctors, nurses, support personnel--were either too busy to call the helpdesk for repair or didn't know where to go for support. Since starting the Buddy System, Children's IT has become more proactive, solving problems that once lingered and tarnished its image.

By helping to remove doctors and nurses from the tech-support process, the program allows for more focused patient care. All it takes is an occasional stroll through the halls.

Children's learned these lessons at the school of hard knocks:

• Before signing on for any large-scale business application integration, make sure the project has non-IT executive ownership--and know the execs' names.

• Crash the party: Force your way into meetings at the earliest possible stage; don't let businesspeople minimize IT's involvement.

• Pick a code-freeze date well before the go-live date and stick to it no matter what anyone says.

• Test, test, test. When it comes to ERP, fight for end-to-end transactional testing.

• Know your audience. Spend time plumbing the knowledge depths and skill sets of your integration partners. Don't let them make decisions outside their abilities.

• Size matters. Do proper due diligence before selecting and sizing your hardware environment; don't rely on the vendor to do it for you.

• Talk to your customers. Engage end users to ensure what's being built meets their needs. Don't let a vendor start any conversation with "It's been our experience ...."

• Love your DBA. Success or failure can hinge on how well your databases are tuned.

• Don't rely on the main project management team to cover everything. Build plans for every technology area--development, networks, systems, helpdesk.

• Don't underestimate the importance of report generation. You may end up tuning live systems just to get them to print a report.

• Train everybody, from end users to developers ... the earlier the better.

• When dedicating people to a large-scale project, be sure to back-fill to cover their responsibilities.

• Be prepared to fight those pesky but inevitable software bugs. Set a maintenance schedule for your development environment.

Hardware:

• Divide Windows NT ERP production environment into two parts: an HR production server and a finance production server.
Status: In progress

• Install Cisco network infrastructure.

Status: Early 2003

Software:

• Plan change in time sheet module (Time and Labor); Kronos is a possible replacement for the PeopleSoft module.

Status: Being evaluated

• Phase 1 of the PeopleSoft ERP rollout was "plain vanilla." The following modules will be part of Phase 2:

eBenefits online open-enrollment module lets employees select health care, retirement plans and other benefits

eProfile module lets employees maintain their own profiles

Budget module for hospitalwide integrated budgeting, a general line-of-business tool that will let admins interact with the financial system without gaining full access to all financial data (requires new hardware and software)

Grant administration module to manage research grant requests and expenditures

eProcurement online

purchase requisition module for internal and external procurement.

Status: Rollout begins in December.