On Location: MedicAlert's SOA Initiative Gets Healthy Prognosis

What's an IT professional's dream scenario for a new project? You'd work with innovative technology that could make a clear, positive impact on your company's success, perhaps even transform the way you do business. The project would be big enough to get noticed but not so big that you couldn't afford to make any mistakes. You'd have a nimble, capable team that could make decisions quickly and not get bogged down in bureaucracy. You'd be led by a visionary with a passion for innovation. The infrastructure on which you'd deploy your new system would be modern enough to get the job done, and where it wasn't, you could enhance, upgrade or rip and replace. You'll have a reasonable budget, spread out over several years. And you'd have complete support from all levels of management, top to bottom, strategic to tactical, and the CEO would understand, share in, and be genuinely interested and involved in the vision of how IT could transform your business.

This, in essence, is the scenario at MedicAlert, an international nonprofit that's been in the business of storing and maintaining personal medical information for more than 50 years. MedicAlert is in the midst of a hugely successful and forward-thinking implementation of an SOA (service-oriented architecture), and much of the credit goes to its development team for putting together a system that works while raising themselves up by their own bootstraps in skills and expertise.

But before you throw yourself whole hog into an SOA project, take a good look at the lessons learned by the development team along the way. Listening to the voice of experience can save many painful, and expensive, wrong turns.

Know Where You're GoingJason Krohn, manager of software engineering at MedicAlert, started us off with some observations about process and standards. "Definitely, we made some mistakes," Krohn says. "Everyone on the team was new to doing a service-oriented architecture. We started out right, by looking at contracts, trying to identify the essential Web services schemas in those contracts between our partners, before we started doing development. This is as opposed to looking at our business needs and starting to build our business logic first, then tacking on some sort of Web services to those."

In other words, MedicAlert learned that a cart-leading-the-horse approach is all too easy to get caught up in.

"As we continue to develop, it has really helped us to define what the structure is up front, and get all the different players, and a maximum of information, earlier in the process," Krohn says. "This approach has led to a shorter development cycle. Our engineering support team is able to start building unit tests, regression tests and test cases right away. Our business team is able to start working on clients right away. At the same time, our services guys are working on business logic for services. Everyone can come back and start plugging things in as they get their pieces done, making sure they work and that everyone has conformed to the contract we've already defined."

One of the biggest hurdles in the beginning, Krohn says, was trying to identify the right development tools. There weren't many vendors offering what MedicAlert needed, so the team started rolling out its own Web service-management tools."Trying to make those from scratch, we found out how hard it is, and how much time it was going to take. We realized very quickly that was not the business we wanted to be in," Krohn says. At that point, MedicAlert purchased a set of SOA management and monitoring tools from AmberPoint.

"AmberPoint and a number of other vendors have really given us a strong foundation to move forward with our architecture," Krohn says. "That was a big part of the success of our projects, getting that nice foundation from the best-of-breed toolsets we were able to bring in-house. We've relied on Altova for managing schemas, for developing XML messages or writing WSDL [Web Services Description Language]. And the standards communities have done a good job of publishing documentation that has let us stay on track, with whatever toolset we wanted to use. We were able to go back with that toolset and look at the standards, and ensure that we were following the proper practices."

That's been key--ensuring adherence to standards and not getting locked in to a particular technology or vendor. "We do most of our development back here in .Net," Krohn says. "But we've got Java clients hooked up to it."

The toolset used by MedicAlert's software development team has evolved over time, and now combines a primary IDE (Microsoft Visual Studio) with various third-party and open-source pieces, a mix of vendors and products indicative of the still immature SOA development arena. The team has had to learn as they go, exploring technology and tool options, some of which turned out to be dead ends, but ultimately putting together what we'd characterize as a combination of "best-of-breed," "in-house" and "what they could get."

When Tim Freeman was first hired as MedicAlert's senior developer and tasked with implementing a new application architecture, nobody had a clear idea what an SOA environment would look like, never mind what it could do. The basic concept, however, comprised the familiar developer's siren song of code reuse, combined with the possibility of developing and deploying a flexible, standardized way to access information across the enterprise. The first task the development team faced was trying to learn what SOA was, as opposed to what they thought it was. Accurate information was hard to find, and they quickly realized they were way out on the cutting edge for an SOA implementation."There were a lot of things we just discovered as we went," Freeman says. "I think we were hoping to find some type of documentation that would say, 'These are the steps, and these are where you need to do them, and this is exactly what you need to do.'" Unfortunately, the literature they found was less scientific than holistic, leaving the default trial-and-error method.

Because most of the team had a solid grounding in Microsoft's Visual Studio development environment, that was a natural place to start.

"Unfortunately, with SOA you can read a lot about it and you think you understand the main pieces that are delineated there, but you don't really understand them until you actually need them," Freeman says. "We didn't realize early on that we were going to need other tools, a mix of tools. We started with Visual Studio, and we tried to figure out how we were going to use it to accomplish everything we needed to do. Visual Studio made everything easy as far as creating Web services, WSDL and schemas, and we thought this might be all we needed."

And indeed, Visual Studio has been a core tool for the MedicAlert SOA implementation, much of which has been developed in C#. "The Microsoft tools are a good way to learn. They make it easy to generate bad code, but also to get a handle on figuring out how things work," Freeman says.

But as you may have surmised, it hasn't been all roses. Going with Microsoft technologies led the team down a few blind alleys. "I think the first thing most people do is say, 'I want a service-oriented architecture, I want services.' So they slap a Web service attribute over a method and say, voilà, I have services now,'" Freeman says. But of course, it's not that simple.".Net builds WSDL and all this other stuff for you, and you don't even care to look at it," Freeman says. "But eventually you realize that good WSDL actually is important, and the WSDL .Net generates really sucks."

One technical dead end frequently cited by the team was Microsoft's Enterprise Development Reference Architecture. EDRA at first seemed to be a boon but turned out to be configuration-heavy, complex and, Freeman says, "kind of flaky. We became EDRA experts because we were convinced that was the way to go. We knew we needed to be able to route messages and make some type of separation between the services and the actual implementation. We did a lot of research on EDRA ... more than what I wish we would have done."

Manage Up

SOA management is another area where there was a significant switch in design philosophy as the team learned by doing. Originally, the developers considered rolling their own management tools.

"We thought maybe we could build something that would monitor service requests, allow us to intercept traffic and do all the things we needed to do," Freeman says. "We quickly realized this was way bigger than what we were willing to take on." So, the development team started looking to service management vendors."We downloaded a lot of papers, read a bunch of stuff and talked to some vendors, and we eventually ended up with AmberPoint," Freeman says. "It did everything EDRA was trying to do, except it actually worked."

Originally, AmberPoint was tapped to monitor services, obtain metric-based performance information and handle some traffic management. But after working with the product, the team members discovered it allowed them to do much more: Grab, route and secure messages; encrypt and decrypt; and add a real service interface to what was happening under the covers.

Testing 1, 2

Testing and quality assurance continue to challenge the MedicAlert SOA development team. "We didn't have much Web services testing initially. We started out with XMLSpy, which is very useful, and very simple," says Natalie Doan, the team's QA analyst. "After awhile, we got our hands on more automated testing tools, such as Parasoft's SOAPTest, which helped us a lot [by] allowing us to manipulate just the message data, then automate it to simulate multiple users, providing both functional and load testing."

Doan adds that one of the biggest difficulties with testing Web services is that there are so many layers--it can be hard to pinpoint which function in a chain of processes takes the most time. For example, Microsoft BizTalk, which is used to handle much of the business logic in the system, calls many lower-level services, making it hard to pinpoint bottlenecks.On the other hand, AmberPoint's Extension Manager can monitor services in both production and test environments, reporting on uptime and service status, and logging the kinds of errors being generated on the production side, so they can be replicated, isolated and analyzed in the QA environment. Doan also routinely uses SOAPTest for regression testing. On the development end, several team members told us XMLSpy was a lifesaver for sending XML messages to Web services for testing purposes. Other key testing tools used during development include ProxyTrace and NUnit for .Net unit testing.

But there was an underlying problem: Most of the QA process--even the addition of an actual QA team for the SOA initiative--didn't happen until halfway through the development process. One reason for the delay was the pilot/prototype nature of the SOA implementation's early stages, but the difficulty inherent in exhaustively testing SOA projects was cited repeatedly by members of the development and QA teams.

One central lesson learned by the MedicAlert developers, sometimes through painful experience, was that there are lots of tried-and-true techniques and technologies out there--even for an area as new as SOA--and it's almost always better to make use of them. Take MedicAlert's initial decision to create its own cryptography module. The company ended up using standard SSL along with a third-party encryption library using WS-Sec standards.

"Unless you have a really specific fringe case, you can use standards," Freeman says. "We had enough to worry about without trying to roll our own security framework." Enough said.

Speaking of SSL ...IT architect Jorge Mercado had a clear understanding of the importance of SOA security--indeed, the nature of MedicAlert's business means security concerns trump all others. If the company couldn't lock down its SOA infrastructure, it may as well throw in the towel. Adding to the problem was the fact that Web services, historically, have not had extensive support for security.

"Web services basically reused this HTTP transport that was so successful and so ubiquitous, and started passing around XML on top of that. So that's what you have to start with. No matter what platform you're on or set of tools you're dealing with, Web services--and Web technology in general--are inherently not secure," Mercado says. "There are situations where you should, for instance, use SSL. But even so, you still need to be able to authenticate into a Web service, so using a name or password over SSL is a very popular combination--a proven, trusted way of securing both the Web service and the Web application.

"But that only takes care of part of the problem. Once that SSL tunnel has been terminated, what do you do then? For example, we have all of our communication using SSL up until the point where you hit the MedicAlert environment, at which point SSL is no longer used. There are some tunnels that use SSL because we have to. But where we don't have to, we don't like to because of the management overhead, both in initiating and terminating the SSL tunnels and managing the certificates that SSL requires.

"Certainly, you still need some semblance of a security infrastructure. But it's a lot simpler if you don't have to do SSL. Not everyone agrees with that statement, but what SSL gives you is the same transport out of the box, and you really don't have to worry about the data. But what you do have to worry about is, 'Is every place this message has to go SSL-enabled?'"

To avoid that issue, Mercado settled on message-level encryption. "If you are not relying on SSL, then this XML message can go anywhere it needs to go, and if it's encrypted, then you're OK," he says. "For anything that travels over the Internet, whether there's message-level encryption or not, we use SSL. Then, once messages come into our environment, we no longer use SSL. We use message-level encryption."Mercado says the team did extensive security analysis throughout the process of modeling services, taking into account different layers of services, each with its own requirements. "One of the things we did so that we could get going really quickly is start with services that don't require a heavy amount of security. We wanted to get our SOA infrastructure up and running; we didn't want to get bogged down too much by the security aspect of it. Then, as we moved along, we started dealing with medical record information and places where Social Security [numbers] started appearing," Mercado says.

"Of course, now we had to encrypt this data. We initially started with some schemes of our own to encrypt the data on a message level. We were writing some C code, wrapped by DCOM, to call out. And we knew we were operating on the Windows platform, so we were calling out to the operating system to do some low-level encryption for us, just basically on streams. In the end, it got as far as test, and we were not satisfied with our test results. So we scrapped it, and at the very last minute we decided to rely on SSL up to the XML firewall to encrypt messages that came into our environment. Any subsequent endpoint or downstream Web service that would consume those requests would then have to decrypt the message before it could process the data in that message."

Small Is Beautiful

There are times when less is more. In terms of team structure, Mercado and Freeman agree that a larger team would not have aided the process, and, in fact, could have slowed the SOA rollout. Both recommend small, manageable projects and a compact team for initial forays into SOA.

A similarly streamlined conclusion was reached with regard to process and modeling. The MedicAlert team experienced mixed results with formal enterprise architectural process and modeling tools."The team kicked around test-driven development but didn't really go there," Freeman said. "The XP process is not for everybody." It also trialed MDA and Agile MDA, then RUP (Rational Unified Process) and Scrum, but found it too difficult to process while at the same time trying to figure out what SOA was and how it worked.

Freeman says he liked RUP and thinks that if a lightweight version had been used throughout, the development pace could have been faster, but Mercado found RUP heavyweight and cumbersome, especially for such a small team. Freeman stressed that the introduction of any process tool requires someone to vigorously champion it and make sure it takes root. As development progressed, the team adopted some components of an Agile-inspired development process (see www.agilealliance.org).

Finding the right modeling tools was another challenge. The group tried Borland Together, but usage decreased over time, while Visio's toolset wasn't used consistently. Early prototypes relied more heavily on modeling, which dropped in importance as the team developed expertise in SOA development and eventually involved whiteboard sketching and taking pictures with camera phones--the bare minimum of modeling necessary to get the job done. Mercado and Freeman agreed that round-tripping--that is, using the model to generate code, altering it and then using the new code to regenerate an updated model--was a persistent problem when using any modeling tool. The team found that Borland Together had cleaner code generation and was better at round-tripping than Visio, which frequently required a lot of time to clean up the code. In the end, however, no modeling tool panned out.

Version Control

Another word to the wise: Service versioning is too often an afterthought, which can be a real problem once you have Web services being consumed by external entities. A workable versioning strategy from the start will save many headaches later."A company must be able to have a policy communicated to its WSDL schema developers, specifying that, in general, this is what's considered to be a breaking change versus a nonbreaking change," Freeman says. "Next, a company offering Web services to more than one customer needs a straightforward policy through which to communicate changes. The worst thing you can do is change services and the customer doesn't find out until suddenly the client breaks. Customers have to know where to go to get new versions, and how to know when new versions are available." Because there's no single, clear-cut way for doing service versioning yet, the versioning process must include naming conventions, using the XML namespace in WSDL and standards for custom attributes, as well as a clean process for updating services when required.

The Hard Way

A software system is only as good as the hardware it sits on, and MedicAlert's SOA implementation is no exception. IT architect Mercado lured network engineer Keith Machado to MedicAlert in September 2005 with visions of the company's technology innovations. At the time, MedicAlert was in the process of replacing antiquated 10/100-Mbps Ethernet hubs with Cisco switching and routing equipment, and adding a centralized SAN for managed storage. Machado says some decisions were made with an eye toward supporting the new SOA architecture.

"We want to restructure our DMZs, put new firewalls in and structure the environment to handle outside business partnerships, the external entities that will be taking advantage of our SOA," Machado says.

MedicAlert brought in two load-balanced Cisco 5520 Adaptive Security Appliances, which will be the primary front-end for firewalling. In the middle, the company plans to use Microsoft Internet Security and Accelerator Server 2004 to help with traffic distribution, with a multitier infrastructure and a minimum of two protected networks--one dedicated to SOA/e-commerce, the second for day-to-day business operations. To increase service availability, MedicAlert is in the process of configuring a secondary Internet connection through MCI, coming through a different PoP than the existing T1; all traffic will be load-balanced between the two connections.And MedicAlert isn't stopping there. "We want to deploy intrusion protection, as well as some internal access protection on our VPNs, utilizing 802.1X with certificates," Machado says. "One of the other things I'm setting up now is internal PKI (public key infrastructure) through Active Directory. We're utilizing Virtual PC quite a bit to prototype implementations before actually bringing them onto the infrastructure, and that ability to rapidly create and restore configurations is really nice."

Plans to expand existing voice-over-IP capability involve deploying a Cisco 3845 router for voice and data connections using QoS (quality of service). Today, however, the SAN and VoIP boxes share the same Gigabit Ethernet backbone with other internal data. MedicAlert runs 100-Mbps over Cat 5 wiring to all its desktops using a pair of Cisco 4510 switches with two 48-port GigE blades each. A pair of already-installed F5 Networks load balancers will handle all incoming Web traffic. Other near-term SOA-oriented infrastructure plans include rolling out SQL Server clusters and using Microsoft Operations Manager 2005 for system management, deploying agents and bringing in non-Microsoft devices through SNMP.

"When systems hiccup, we'll know about it," Machado says. And if MedicAlert succeeds, information about health hiccups will also be readily available to caregivers, keeping eHealthKey users in the pink.

Jorge Mercado
IT Architect

At Work: Responsible for leading MedicAlert's SOA implementationAt Home: 33 years old. Married, no children. Hobbies: Sports, reading

HOW HE GOT HERE: 2002 to present: IT architect, MedicAlert
2000-2002: Microsoft consultant

MOUTHING OFF: Best part of implementing Web services and SOA: "Getting to do what everybody's talking about."

Worst part of implementing Web services and SOA: "Doing what others are only talking about--very few [people] have actually done Web services, so there aren't any experts."

Most misunderstood part of my job: "In architecture and software development, most of your work is under the covers--people think that you don't do a whole lot."If I had the Web services/SOA project to do over again, I would: "Do more planning at the beginning."

My dream job: "Professional golfer."

When I retire, I will: "Play golf."

Best part about working at MedicAlert: "There's nearly zero politics for me. I can focus on my work."

What most people don't know about their personal medical information: "How important it is, and how useful it can be. Go over it with your doctor, and it could make you healthier in the long term."What most colleagues don't know about me: "Most people here think of me as a full-time technology guy, but I'm an avid golfer and I like to work in the garden."

Worst moment of downtime: "We had a Web service failure once, where we actually blew away one of our mail servers with some bad code."

In my car stereo right now: "Dave Matthews."

Favorite gadget: "My Audivox 5600 smartphone."

Tim Freeman
Senior DeveloperAt Work: Responsible for developing MedicAlert's Web services and implementing SOA

At Home: 27 years old. Married, one child. Hobbies: Music, ju-jitsu

Alma Mater: Sacramento State; B.S. in Business Administration with concentration in IS

HOW HE GOT HERE: 2004 to present: Senior developer, MedicAlert
2003-2004: Software developer, Healthcare Plan of San Joaquin

MOUTHING OFF:Best part of implementing Web services and SOA: "Seeing it work."Most misunderstood aspect of my job: "What an SOA is and how it's going to help the company."

If I had the Web services/SOA project to do over again, I would: "Develop ID access management."

My dream job: "Running my own deli."

When I retire, I will: "Buy a sports car."

What most people don't know about their personal medical information: "That they own it and they can manage it themselves."Worst moment of downtime: "We had one day when all the Web services went down at the same time."

Richard Hoffman, a former Network Computing technology editor, is the editor for DeveloperPipeline.com and an IT consultant and analyst living in New Hampshire. Write to him at [email protected].