Nancy's Specialty Foods Cooks Up Dual SAN Implementation

Today, StoneFly's synchronous mirroring lets the backup SAN take over if Nancy's primary SAN fails, because data is written to both SAN segments simultaneously. Working closely with StoneFly's engineers to beta test the dual SAN architecture, Nancy's first moved its Microsoft Exchange 2003 data to the mirrored SAN, then added data from its QAD MFG/Pro ERP (enterprise resource planning) application, which uses about 500 GB of the 3.2-TB SAN. Since then, Nancy's has moved other data to the SAN, including data from its Cisco Unity IP voicemail system and, most recently, its new RFID inventory system.

Cost was a major reason for Nancy's choice of the StoneFly iSCSI environment. The small, privately held company had a tight budget, so not only was a separate disaster-recovery site out of its price range, but so was a Fibre Channel SAN. Nancy's went with a dual IP SAN architecture on-site, which was half the cost of a Fibre Channel SAN. "We know the IP SAN is not as powerful as a Fibre Channel SAN [in terms of speed], but in our environment, it's good enough," Choy says. Nancy's iSCSI SAN runs on a Gigabit Ethernet backbone, about half the speed of Fibre Channel, which can run anywhere from 2 Gbps to 4 Gbps.

The need for a SAN became clear when Nancy's installed Microsoft Exchange 2003, along with Active Directory. The company's DASD (direct-access storage device) architecture couldn't support the fault-tolerance requirements of the Exchange implementation. Nancy's had already bought additional servers for Exchange 2003, and the IT group was migrating Nancy's ERP application from HP-UX to an Intel-based Linux platform. The company had a choice: Add more DASD or build a SAN."From a disaster-recovery perspective, we wanted to cluster the Exchange servers, which you can't do with attached storage," Choy says. "So a SAN became the obvious choice."

And Choy wanted to centrally manage storage, too. "You have to provision direct-attached storage individually, and its utilization is hard to control," he says. The result: Nancy's server storage was either underutilized or nearly tapped out. After evaluating both Fibre Channel and iSCSI products, Nancy's settled on StoneFly. It was a little cheaper than competitive products, and its Storage ConcentratorT i3000 switches are modular. Choy says the company also wanted to be able to mix and match different brands of disk, which the StoneFly system supports. Like other iSCSI SANs, the StoneFly system uses inexpensive serial ATA drives, which come with 200 GB to 250 GB of storage per disk. Nancy's currently uses about 1.5 TB of the 3.2-TB SAN, so it has plenty of storage headroom.

The iSCSI SAN is browser-based, which made it easy to configure, Choy says. Nancy's created an Exchange cluster on the SAN and basically copied and pasted file data to it. Sliding the mirroring software into the SAN was straightforward after the month of beta testing, he says.

There are some trade-offs with Nancy's iSCSI SAN. For example, if server hardware fails, you can't boot it directly from the SAN switch--you must boot from the application server. Booting from the SAN, which has long been a feature in Fibre Channel, would shrink the downtime window to a bare minimum, Choy says. To make up for this shortcoming, Nancy's is evaluating third-party iSCSI adapters that come with an embedded TCP/IP stack. The adapter also would eliminate the need to rebuild a "broken" server, because the boot code is stored on the SAN disk. This feature wasn't available for iSCSI when Nancy's first bought its IP SAN. "We still boot the hosts from their local hard drives," Choy says.

Nancy's, which is best known for its quiche, bakes more than 25 tons of gourmet food each day and distributes it through large grocery chains like Kroger and Safeway, and to big-box club warehouses like Costco and Wal-Mart. Since Wal-Mart is its biggest distributor, Nancy's has been preparing to comply with Wal-Mart's RFID initiative, which requires its suppliers to ship their products to the discount chain with RFID tags. Nancy's is already deploying RFID for internal inventory control as an eventual replacement for its barcode system. In the first phase, the company is placing a passive tag on its frozen-food boxes that identifies the product. This lets the warehouse forklift operator know where to put the products and in which bins. "That way, we're able to track the movement of the product in-house," Choy says. Specific product information, like quiche Florentine, is stored in the RFID application. Current RFID technology makes it difficult to read too much information from the tag, Choy says, so that data stays within the application server.Nancy's is also building out a wireless LAN to support its in-house RFID system.The Cisco Aeronet 802.11b/g WLAN supports the vehicle-mounted computers on Nancy's forklifts, which move products from inventory to packaging for shipment. "We've expanded our WLAN coverage because of the RFID project," Choy says. And he hasn't given up hope for someday building an off-site SAN for true disaster recovery, but so far, the company has no official plans to develop one.

It's no Surprise that disaster recovery is a priority for Nancy's Specialty Foods--it would be for any company in the earthquake-prone San Francisco Bay area. But for most small and midsize enterprises, the main obstacle to building an off-site DR facility is money. So when Terence Choy, Nancy's network manager, pitched an on-site, mirrored StoneFly i3000 iSCSI SAN selling at half the price of a conventional Fibre Channel system, top execs quickly embraced it.

"We pointed out how the IP SAN would help us--it wasn't that hard of a sell," Choy recalls. Nancy's had already secured an additional floor of its headquarters building for office space, so putting the backup SAN there was a no-brainer. Choy translated the storage project into business terms, explaining that if one SAN went down, the other would automatically take over, preventing Nancy's operations from being disrupted. The dual system would also alleviate inconsistent disk utilization.

The timing of the project couldn't have been better. Choy and his team proposed it during the company's new 2003-2004 fiscal year budget-planning phase.

In total, the system cost Nancy's upward of $60,000--including the Asante IntraCore 35160-T Gigabit Ethernet switches--and the company got an ROI within 18 months."We talked about the technology and the cost, and we were able to plan ahead for the next fiscal year," Choy says. "Because the IP SAN was cost-efficient, it fit into our budget constraints."

The trade-off, of course, is that the data center is still at one location, albeit with a mirrored SAN on different floors. Nancy's hasn't priced a separate DR site, but the cost could be in the millions. Still, the company's data is safe with its existing setup: Nancy's sends backup data to EVault's backup service nightly over the Internet. But were the entire building to lose power or catch fire, Nancy's would have no network. Choy admits IT would prefer a separate DR site, but such a facility is beyond Nancy's budget priorities for now.

Meanwhile, although the Wal-Mart RFID initiative (see Get Ready For The RFID Wave) is in the early phase, it has already spawned other IT projects at Nancy's. The food manufacturer is now deploying RFID internally for inventory control, which will let Nancy's better leverage the RFID technology it will start using with Wal-Mart next year. The IP SAN will be crucial for storing RFID data when Nancy's deploys the RFID tags on its frozen-food boxes. "Without a SAN, it would make RFID deployment complicated," Choy says. There are no new mandates from its other discount- or grocery-chain customers thus far, he says, "but you never know."

Terence Choy

Terence Choy, 44, is network manager at Nancy's Specialty Foods in Newark, Calif., a privately held maker of frozen gourmet entrees and hors d'oeuvres. Choy oversees Nancy's enterprise network, including a mirrored IP SAN at its headquarters. He's been with Nancy's for 11 years and in IT for 19 years.Why iSCSI SANs get a bad rap: "Fibre Channel SAN vendors see the threat [to their business] and need to protect their turf."

How many IT guys does it take to put in a Fibre Channel SAN? "I'm not sure, but the Fibre Channel SAN vendors send their own engineering team to do the setup anyway and then charge a hefty price."

Regrets about not going with a Fibre Channel SAN: "Only if money had been no object, but otherwise, no."

The difference with the food industry's IT needs: "Our IT objectives quite often are driven by the time line of complying with our customers' mandates. For instance, Wal-Mart's RFID compliance."

How IT imitates life: "Sometimes I have my whole day planned. But after being busy all day, none of the things I did are what I planned for. So my advice: Always be prepared for the unexpected."What Choy's co-workers don't know about him: "I am a very good poker player. You won't see my face on TV soon ... I don't play poker tournaments. But I do play regularly in poker rooms, casinos and online."

His favorite Nancy's dish: "I like them all. Besides the quiches, I especially enjoy our mushroom turnover. I can cook, too, but I have a great girlfriend who never lets me in the kitchen. Nancy's won't have any competition from me."

Where's Nancy? "I see her all the time, though she sold the company in 1999. She's still an icon. We [sometimes] 'play' on her yacht."

Wheels: "A Honda Accord. It has a great air-conditioning system for my long summer commute in Northern California."

Favorite team: "The San Francisco Giants."After hours: "Poker, of course, and golf."