Profile: Raymond James Financial

Raymond James has built a layered security fortress for its network and surrounding VPN--not only to prevent well-known security threats from disrupting its operations, but to stop criminals from stealing or damaging sensitive financial data. So far, the company hasn't suffered any outages due to security breaches.

"Any general Internet traffic coming into the network is going to pass through three IDS/IPS [intrusion-detection/prevention systems] before it gets onto the backbone," Loach says. This includes the firm's public Web site and e-mail, which is scanned multiple times for viruses, malware and spam. "We scan inbound and outbound e-mail at the desktop and at the mail servers," he says.

Raymond James' security architecture is built around tools from a variety of vendors. A Cisco Internet router uses access lists to split traffic into two streams--one for the firm's branch-office VPN traffic and one for incoming Internet traffic. Both streams go through a series of Check Point Software firewall/VPN clusters and Internet Security Systems' IDS/IPS, as well as a Trend Micro VirusWall. Another Cisco router checks the traffic against its access list at the other end.

This multilayered security architecture ensures that Raymond James' independent brokers, who access the company's backbone through the VPN, are thoroughly authenticated and given access only to the tools they're authorized to use. It also helps prevent the brokers from unwittingly passing a worm or virus into the backbone network.

But managing so many layers of security isn't easy. Loach and the data center security team use the Check Point SmartCenter Pro management platform, which gathers logs from the Check Point equipment as well as from their Juniper Networks SSL VPN appliances and WholeSecurity system log (syslog) data. SmartCenter Pro can access Cisco router access lists, but the other security data Raymond James has at its disposal doesn't go through SmartCenter Pro.It's up to Loach and the security team to sort through all the event logs to determine which alarms are real and how problems should be handled. The company is currently looking for an event-correlation engine that will show the relationship between security-related events automatically and help the IT team find the source of a security breach.

"An event-correlation engine will tie all of these layers together and help us demonstrate an ROI to management. It will show what we blocked today, and what the IDS and firewalls helped prevent," Loach says.

Raymond James is also adding more features to its SSL-based VPN. The firm is installing Check Point's Extender software onto its firewall/VPN boxes, which will let the SSL VPN go beyond standard Web, e-mail and terminal services to give branch offices direct, secure access to internal financial applications.

In addition to fighting the latest security threats, Loach and the IT security team are tasked with ensuring that Raymond James complies with regulations such as Sarbanes-Oxley, HIPAA (the Health Insurance Portability and Accountability Act) and the Graham-Leach-Bliley Act, which require financial services firms to prove they are protecting their customer and client data.

That's where the event-correlation engine will come into play. "We can tie events back to a specific part of a SOX or HIPAA regulation," Loach says. And the VPN appliances will help the company encrypt all customer information passed over the Internet.Too Much Information

Raymond James' multiple layers of security tools are a mixed blessing. They track and gather so much data on suspicious activity that it can take days to get to the root of the problem. Each day, the firm's routers and switches generate 50,000 lines of syslog data; the IDS/IPS tools generate another 15,000 to 20,000 lines. More than 600 servers spit out 15 million lines of logs that contain security-related information.

"We've deployed all these layers of security, and now we need something to handle that. I can't look at 5 million firewall logs per day," Loach says.

Even within Raymond James' IT organization, there is a need for greater coordination of information. There are more than 600 IT people in the company, including WAN, LAN, network engineering and security groups, and each department has its own monitoring tools for specific tasks. An automated event-correlation tool would boil down all of the data, Loach says.

Raymond James' fortified security architecture is formidable, but not impregnable. "If something does get in here, it usually walks in on a laptop," Loach says. The culprit is usually a "zero day" worm or virus that antivirus programs haven't detected.It's not easy to trace the source of an attack, either. Raymond James was hit hard last year by the Blaster worm, which propagated so fast it knocked an entire floor of the building offline in less than three minutes. Loach and his team spent days wading through millions of lines of code to pinpoint the source, only to find that the worm had originated from an outside consultant's infected laptop. The company has since instituted policies to check out laptops before they can get on the network.

Although security events can be traced manually, it's a process that can take anywhere from a day to a week, Loach says. "That's why we need to automate," he adds.

Raymond James financial has experienced two system outages in recent months, but they weren't security breaches. When those outages first occurred, however, no one was sure of their cause, and the security team was forced to plod through millions of activity log records by hand to find the source. In the aftermath, company executives began asking lots of questions about the security team's method of reconnaissance.

It was painfully obvious that the company needed a way to streamline and simplify the team's manual correlation of security incidents, says Scott Loach, senior information security engineer for Raymond James. Loach's bosses weren't willing to sacrifice any transaction time during stock-market hours, and they wanted to know why it was so hard to sleuth the outage.

"We sat them down and showed them an example of the work it entails to do this manually," Loach says. "Every once in a while, you have to make sure upper management is aware of how much time and money it costs to simply ask, 'Why?'"Once the company's top IT executives saw the labor involved in determining the origin of a problem, they were ready to hear about a solution. Loach and his team proposed an event-correlation system that would correlate log data and other information automatically to help determine the root cause of a problem. Loach and his colleagues eventually traced the problem to a policy glitch between its IDS/IDP tool and Check Point firewall.

The price tag for an event-correlation engine--anywhere from $200,000 to $300,000--gave the execs a little sticker shock, Loach says, but they still gave the project the green light. Now Loach and his team are evaluating tools.

Loach says management's support for the project came as no surprise--they've always been tuned into IT security. The company first formed its IT security department more than four years ago after a consultant discovered alarming vulnerabilities in the network. "Within a week, they had hired a vice president of IT security and formed an IT security department," Loach recalls.

The next challenge is to sell the execs on securing the firm's applications at Layer 7, Loach says.

Scott Loach -- senior Information security engineer, Raymond James FinancialScott Loach, 40, is a senior information security engineer for Raymond James Financial, based in St. Petersburg, Fla. Loach is in charge of the firm's global IT security infrastructure and runs its entire firewall and VPN architecture. He attended the University of Central Florida, where he studied computer science as well as accounting, and he's the founder and president of the only user-run Check Point Software user group. Loach has been at Raymond James for more than six years and in IT for 21 years.

On certifications: I hold a few, but a certification is not a replacement for on-the-job training and practical experience. Anyone can go to these classes, study the workbooks, memorize what's needed to pass the tests and get that piece of paper.

On balancing regulations with real-world security: We address these issues daily, on a case-by-case basis. Many of these mandates are still evolving or have recently been solidified. The key is strong, published policies, procedures, and checks and balances.

Biggest security threat for financial firms: Identity theft is at the top of the list. But the largest global threat is the first worm released on the Internet that propagates so fast and generates so much traffic that it effectively shuts down the Internet. If the Internet crashes, how do you reboot it? And what do you do if it crashes again, as quickly as you reboot it?

Biggest risk that paid off: Leaving Florida to work six years in Atlanta for a small start-up that went public while I was there. The stock options paid off in the end, and the experience I gained working on networks in over 20 large hospitals worldwide has been invaluable.Wheels: Three different 1969 Chevrolet Camaros and a 2004 Honda S2000--the new Honda for the sheer thrill of a car that redlines at 8000 RPMs and the Camaros for the sound and feel of a real V8.

For Fun: Working on my Camaros, fishing, diving, sailing--anything on the water.