On Location: American Airlines Center's Wi-Fi Network

Ironically, the high-tech Aruba network is actually a step back from the center's original vision, which called for fans to have Internet access from their seats. Cuban and Dallas Stars owner Tom Hicks scrapped that idea, citing the distraction factor. But don't take that as a sign that creative ideas for implementation aren't welcome. "Going forward, anything that doesn't impact fans' attention while the game is going on is workable to me," Cuban says.

Indeed, AA Center operators are committed to using wireless technology in a wide variety of applications. The building opened in 2001 with limited wireless capabilities, including wireless ticketing and a wireless point-of-sale system to support some of the building's memorabilia and merchandise outlets. But the functions weren't connected, and there was no wireless infrastructure in place.

"We had several wireless systems in the building, but for the most part, they were what I'd call proprietary--they were single use," Mayne says. Single use or not, wireless was working for in-seat services, and he set out to expand the system's reach, with the help of Heinlein and Glasser-Seinera. Heinlein and his IT staff wanted to deploy a wireless infrastructure to support applications across the building, and Glasser-Seinera wanted to create a wireless showcase that would attract the sponsorship of a major vendor. In February 2004, Heinlein outlined the technical requirements for the network and helped narrow the potential field of providers. Then Glasser-Seinera approached the shortlisted vendors about putting their gear on display in the arena.

"This is really an excellent place for a technology vendor to show what it can do," says Glasser-Seinera. About three million people pass through the AA Center doors each year, many of them high-power decision makers in their respective companies. Major brands such as UPS and Dr Pepper already have showcases in the building, and Dell Computer provides many of the PCs used throughout the facility in exchange for marketing exposure.Together, Glasser-Seinera, Heinlein and officials at Aruba worked out a similar deal in which Aruba provides free professional services to the center in exchange for signs, video advertisements and other promotions throughout the arena. The AA Center did pay for the Aruba gear, to the tune of $27,800, according to an Aruba spokesman.

Heinlein and his team began deploying the Aruba network last summer. Although some APs were simple to install--there's one plugged into a wall jack in every conference room--others were trickier, such as the APs suspended from the arena ceiling, several hundred feet above the basketball court (for a detailed description of the network, see "Managed Wireless Is Just the Ticket," ).

"I've been up on that catwalk, but as few times as possible," says Heinlein. "That's why I have a staff."

Today, the wireless network supports numerous functions, including some unrelated to the Mavericks or Stars. On nonevent days, for instance, the center rents out its conference rooms and meeting spaces to businesses or groups that want something different from conventional hotel meeting rooms. The Wi-Fi network lets each visiting group connect laptops or access the Internet without wires, turning each conference room into a wireless hotspot.Wireless technology also helps the AA Center support concerts, rodeos, monster-truck rallies and other performances that need to communicate within the arena by, for example, letting performers easily tune sound systems or run specialized applications. In the past, the center provided wired LAN connections, but a growing number of performers expressed a preference for wireless. Now, the Aruba network can support nearly any type of wireless device brought in by entertainers or convention groups, providing a wireless infrastructure as well as connections to the wired network, external networks or the Internet.

But the WLAN gets its greatest workout during basketball and hockey games, when the full gamut of wireless applications are in play. This stressing of the system is what we came to see. Putting down our bags, we begin walking the center to see Wi-Fi in action.

Down on the hardwood, the Mavericks are warming up for tonight's game. They are loose and relaxed, decked out in practice gear and razzing each other before a drill. The coaches, looking strange to us in shorts and T-shirts rather than their game-time coats and ties, don't look so relaxed. The practice is quickly and sternly called back to order.

Up on the Terrace Level, we're talking with Joe Skendarian, general manager of SportService, which operates virtually all of the center's food concessions. Throughout the building, Skendarian's staff is unloading trucks and unpacking boxes in anticipation of a hungry crowd. The wireless network plays a surprisingly integral role in that process.

SportService and the AA Center use a food-services ASP (application service provider), InfoGenesis, to track, process and control inventory of the many noshes available at the arena. On the back end, the system tracks the ordering and stocking of food items, keeping an up-to-the-minute tab of what's on hand and where everything is located. On the front end, InfoGenesis acts as a point-of-sale system, tracking each food item, recording payments and even processing credit cards for fans paying that way."For years, the front-end systems and the back-end systems have been separate, and you had to sit down weekly or monthly and try to balance them so you would have an accurate count of what had been sold and what was in stock," Skendarian says. "When you do it that way, it's almost easier to use a manual system, because at least then you aren't working with conflicting information."

With InfoGenesis and the Aruba network, however, concession staff can log in new items as they arrive and indicate their exact whereabouts. The system tracks items bound for the center's 38 concession stands and 15 premium restaurants as if each had its own warehouse, but the food can still share a common freezer or refrigerator, saving space and money.

Once the concessions begin serving, the system records each item purchased, through wired cash registers or wireless devices used by wait staff in the premium seating areas. The InfoGenesis system processes purchases and updates the central inventory as each item is used. When a particular item runs low, the system automatically places a restocking order; the concessions are replenished from the central warehouse or an order is placed with the appropriate vendor.

Combined, the InfoGenesis system and the wireless network have enabled SportService to cut its in-stock inventory, eliminating waste, Skendarian says. "We used to have to keep enough for two or three turns [events] on hand, but now we stock about a turn and a half," he says. "It's made us a lot more cost-efficient, and we're improving quality at the same time."

The InfoGenesis system data also enables trending, so that the AA Center can better predict where to place its inventory during an event."I can literally tell you how many hamburgers we need at a particular stand at 7:45 on a given game night," Skendarian says. "Before, we would have a certain amount of dollars set aside for waste, spoilage or theft, but now our inventory is so tight that we're planning those dollars toward improving quality."

Of course, the system isn't flawless--occasionally, a stand will still run out of hot dogs in the middle of an event. When that happens, Skendarian uses the data from the InfoGenesis system to do a postmortem. "We'll look at the report and say, we did 120 at that location the last six events, why did we do 180 tonight?" he says. He might never explain the run on dogs, but without the wireless system he wouldn't even know to ask the question.

It's after lunch now, less than seven hours till tip-off, and some of the game-night staff has begun to trickle in. Before the night is over, the Center Operating Co. will balloon from its 125 full-time core employees to more than 1,000, including ticket takers, concession staffers, security guards, ushers, in-seat waiters and waitresses and cleaning staff. In the next several hours, the center's IT infrastructure will go from 125 users to more than 750, many of them on wireless devices.

Next stop: the IT room, which is really more of a control room than a data center because servers, networking and telecom gear are spread out across the building. Here the center's five-person IT staff is gearing up for the game. Louis Yuan, enterprise engineer for the facility, shows us a diagram of the wireless network while Heinlein pulls up a management screen that will serve as the primary network monitor during the event. On another screen, Derric Paige, POS specialist, is checking the status of wireless devices while answering user queries from the center's permanent staff.

There's a lot of give-and-take in the low-walled IT room--with such a small staff, Heinlein has cross-trained most of his people so that not all must be present for every event. Still, several of his young staffers are single, and they don't seem to mind working lots of evening events during the year. There is an obvious camaraderie among the crew.

Heinlein shows us how the AA Center administers wireless network guest users, principally reporters and photographers who must transmit stories and photos from the arena back to their offices. Guests are charged a minimal fee for use of the network, then are given password access to the wireless infrastructure. If a user attempts to grab a free ride, Heinlein can spot them on the monitor and disconnect them if necessary. The Aruba mesh is not yet complete--the center hopes to double the number of APs and add a redundant Aruba 5000 switch--so Heinlein can't pinpoint the exact location of an unauthorized user in the building, but he can identify the general area of access. Once additional APs are added, Heinlein and his team will be able to triangulate the exact location of an unauthorized user in the building.

So far, there are no problems to report. But it's still early, and the majority of the network's users have yet to arrive.

2:56 p.m.

Just over four hours till game time. There are no Mavericks fans in the building yet, but the AA Center is beginning to buzz with activity. Down on the court, the baskets have been lowered to about five feet so maintenance workers can check the rims and nets. We resist the urge to run over and make a slam dunk in a real NBA hoop.Three tiers above us, Jeff McGinnis, the center's audio engineer, is sitting next to one of the arena's huge Crown Audio IQ amplifiers, laptop in hand. McGinnis is using the computer to measure and adjust the speaker's output, a job that took a lot more legwork before the WLAN.

When we go upstairs to take a look at his sound board, McGinnis explains. "I have software from Crown that shows me the status of each amp, and I can do tuning from here," he says. "But the only way to really hear what the amp is doing is to go over to it and use my ear." In the past, he had to schlep from the main sound board to each amp, listen, then come back and tweak. It was a lot of walking. Now, thanks to the WLAN, he can bring a laptop over to the amp and listen and tune at the same time.

Before a concert, the wireless network also helps McGinnis work with performers' sound people to synchronize the band's amps with those already in the arena. "I can just sit down with the sound guy anywhere, and we can put our laptops side by side to do the tuning," he says. "Sometimes they like the sound of our system so much that they use more of our equipment than their own."

It's below 50 degrees outside AA Center, chilly for a Dallas afternoon, even in winter. Wearing gloves, hats and thick jackets, the center's parking-lot attendants prepare for the arrival of the more than 5,000 cars that will bring most of the 18,000 fans coming to watch tonight's sold-out game.

In each lane of the parking area stands an attendant with a wireless barcode reader. These readers tell the attendant where to direct the driver--to a press/media lot, a premium lot or a general parking lot. The old color-coded parking passes did this as well, but the barcoding also gives parking managers an accurate count of how many cars are in each lot."We can make better use of the available spaces," says Larry Buersmeyer, director of parking services at AA Center. "If one of the reserved lots is only half full, and we know that there are only so many more passes that haven't come in yet, we can make those spaces available to people in the general lots."

Making more efficient use of space has enabled the center to increase parking revenues for some events by as much as 30 percent--in fact, the wireless system has already paid for itself, Buersmeyer says. For example, the center has saved on the number of attendants because the wireless system lets Buersmeyer's team analyze traffic patterns and hire just enough people to staff the active lots, rather than paying attendants to watch low-traffic or empty lots.

Parking passes are often still mailed out with tickets, but AA Center and TicketMaster have teamed up to let fans purchase tickets online and print their parking passes on their home computers.

"That can be a problem sometimes, like when a customer folds his pass right on the barcode, and the machine can't pick it up," Buersmeyer says. In those cases, a number printed with the barcode can be entered manually.

To make things run even smoother, Buersmeyer says he hopes in a few months to put wireless devices in the hands of all attendants, not just those working entry lanes.5:30 p.m.

It's exactly two hours before game time, and all the doors at AA Center have been locked. From now until the final buzzer, anybody who comes through the doors will need a ticket or a pass. The Ccnter's security staff--about a dozen officers assisted by the Dallas police--begins its event watch.

To help with surveillance, the center has deployed throughout the facility 176 cameras that provide views of everything from the front doors to cash registers to remote storage rooms. Not long ago, those cameras could be monitored only from a central control room, but the wireless network is changing that. In a conference room, Rich Aguilar, public safety manager for AA Center, shows us a handheld device that he wears on his belt. If an officer reports a problem over the radio, the Aruba network lets Aguilar bring up any of the 176 cameras with a few keystrokes. He can then assess the situation and, if needed, radio the closest officers to respond.

Only Aguilar and J.D. Hancock, vice president of safety for AA Center, are using the wireless devices because the system is still in the testing phase. So far, Aguilar and Hancock like being able to measure the effectiveness of their security team, and if all continues to go well, they will acquire units for officers to carry next season.

"Aside from making J.D. and myself more mobile, it's already cut down on some of the radio chatter," says Aguilar. "If we get a report, we can just review the incident right there and make a decision on how to handle it. We don't have to go back to the control room or get an officer to describe it."In the press area, reporters, photographers and broadcast media are grabbing snacks while they unpack their gear and prepare to cover the game. Unlike last year, though, nobody is looking for a phone jack or Internet connection; most will file their content wirelessly, using the Aruba network to gain access to the Internet and VPN connections back to their respective offices.

"It's not unusual now to see reporters or photographers filing from outside the locker room or in a hallway," says Rick Ericson, a spokesman for The LeMaster Group, which handles press for AA Center. "It's made a difference in the way they work."

For the center's IT group, however, the media represent one of the toughest administrative burdens on the wireless network. Although most in-house wireless devices are single function and cannot be used to access other applications, reporters bring laptops that could pass along a virus or sneak a peek at the center's back-end systems. For that reason--and to separate the press from unauthorized users trying to ride the network--Heinlein requires the media to register for wireless use.

"It's a bit of a pain for us and them, but it's worth it," Heinlein says.

7:05 p.m.It may still be 25 minutes to tip-off, but across the AA Center, it's showtime. On the court, forward Dirk Nowitzki, the seven-foot-tall NBA scoring leader, is practicing his dunk. In the hallway, the Mavericks dancers are going over their moves. At every concession stand, lines have begun to form. And billionaire Mark Cuban assumes his customary seat--not in a skybox, but in the stands, out in the crowd.

At the doors, ticket takers are admitting Maverick fans, but they aren't using the age-old rip-and-return process. Rather, they use wireless devices that can read the barcode on each ticket and give the location of the ticket holder's seat. And, as with parking passes, many of the fans coming through the door carry tickets they've printed on their own computers using TicketMaster's online service.

Barcoded tickets don't just speed up the process of getting fans in their seats, they also track attendance, help people who have lost their tickets, provide re-entry for fans who need to leave the building, and aid in determining staffing levels for various entry doors.

"The teams can also use the data to find out who's not using their tickets. They will often contact them to find out why they didn't use the tickets, and what the team can do to help," says Eric Bruce, the AA Center's director of ticketing.

The barcode system does carry some risk--for example, a scalper might photocopy a PDF ticket. "If that happens, the first one to register that barcode will get in, and all the others will be stopped," Bruce says. "But oftentimes, we can track the ticket back to its original buyer by looking at credit-card information, so it's risky for the scalper, too."8:20 p.m.

The Mavericks are not in top form, having allowed the Golden State Warriors to jump to a 14-point lead in the first period. However, they've regained their composure and closed the gap. Looks like it's going to be a game. The music at the breaks is loud, and there's an energy in the arena, but the crowd is relatively subdued. They want something to cheer for.

There's a large television in the IT room, but nobody is watching the game. Yuan is pecking at his computer screen and watching The Biggest Loser between keystrokes. Paige, who keeps the POS systems running, is monitoring the status of wireless devices across the building. He's just returned from one of the premium seating levels, where he restored life to a handheld device used by one of the in-seat wait staff. "One of the most common things we see is that, when they have a problem, the users will try to reboot the device," Paige explains. "But these things are secure, and they get locked out. So I go up and reset it for them."

Most of the problems we saw were in this same vein: Wireless devices with temporary hiccups; a computer in one of the luxury suites not working properly; a newspaper photographer having trouble accessing the network.

Heinlein says the team has seen critical problems too, but so far, they've been rare. Some months ago, the center lost its wide-area connection to InfoGenesis, leaving both wired and wireless devices unable to transmit inventory data or process credit cards. The transaction data wasn't lost--it was stored locally in the devices--but there was no way to verify credit-card numbers. Smaller sales were allowed through, but larger transactions had to be called in by phone. The outage lasted about two hours and was eventually traced back to problems on the InfoGenesis end of the connection.The significance of a true outage isn't lost on any of the IT people in the room, or out in the stands. "Obviously, with our customers involved, it's imperative that the experience exceed expectations," Cuban says. "Performance is critical."

As the game winds down, it doesn't look like there's any danger of an outage tonight.

10:22 p.m.

The IT staff has switched the TV channel back to the game, and the Mavericks still have a chance with a few minutes left. Some good outside shooting cuts the Warriors' lead to four, but the defense fails. Final score: Golden State 111, Dallas 107. Most fans have stayed in their seats, and they now push toward the parking lots in hopes of avoiding the inevitable traffic jam.

Paige and Yuan have left the IT room to speak to friends or check out the crowd. Putting on his jacket, Heinlein fills us in on his future hopes for the Aruba network: a redundant server to ensure uptime, and some additional APs to complete the wireless mesh, providing broader access and the ability to triangulate the location of a specific user.For tonight, however, the network has done its job.

"It's not perfect. We can always get better," says Heinlein, sounding a bit like an NBA coach. "But we'll take a night like this anytime."

Tim Wilson is Network Computing's editor, business technology. His background includes four years as an IT industry analyst and more than 14 years as a journalist specializing in networking technology. Write to him at [email protected].

Our seventh "On Location" documentary-style case study takes us to Dallas' American Airlines Center, home of the NBA Dallas Mavericks, the NHL Dallas Stars and a state-of-the-art Aruba Wi-Fi network. Although Aruba was chosen based partly on a services-for-advertising deal forged by Lori Glasser-Seinera, the AA Center's vice president of digital media, we found the center's IT team, led by Joe Heinlein, more than happy with the wireless LAN gear's security, coverage and manageability.

We spent a day shadowing the center's IT staff, from that first cup of coffee to the final buzzer signaling the end of a Mavs game. In "Nothing but Air," we give you an inside look at what it's like to be an IT pro in a major-league sports venue and how the WLAN is helping to increase the AA Center's revenues through tighter supply lines and better customer service. In "Managed Wireless Is Just the Ticket,", we delve into the details of the Aruba system and the challenges Heinlein and his team face keeping up with the myriad demands of fans, media professionals, and the center's employees and suppliers.

Title: President and CEO, Center Operating Co.

At Work: Responsible for development, design, construction and day-to-day operations of American Airlines Center

At Home: 49 years old. Married, two daughters. Hobbies: fishing, golf

Alma Mater: University of Utah; B.S. in leisure studiesHOW HE GOT HERE: 1998 to present: President and CEO, Center Operating Co.

1991 to 1998: General manager, Arrowhead Pond of Anaheim (home of the NHL's Mighty Ducks)

MOUTHING OFF: The best part of having a wireless network in the AA Center: "Making the systems available to our employees, so that our service will be second to none."

The worst part of having a wireless network in the center: "The cost of getting cutting-edge technologies in place--and before somebody else does."

If I had a bigger IT budget, I would: "Integrate all of our systems."Greatest business challenge: "Creating and fashioning a company that meets the needs of three separate organizations: Mavericks, Stars and the city of Dallas."

Most misunderstood aspect of my job: "People think my job is fun because I get to watch all the events and meet all the superstars and performers, but that's really not the case."

I love technology when: "It becomes a valuable resource allowing our workforce to become more efficient and makes our guests' experience more rewarding."

I hate technology when: "It doesn't work--but that doesn't happen here."

My next career: "I'm happy here. Our owners are committed to being the best."When I retire, I will: "Spend time with my family."

Favorite Dallas Maverick: "That one's easy--Mark Cuban [team owner]."

Title: Vice President, Digital Media Services

At Work: Responsible for bringing technology sponsors and cutting-edge technologies to American Airlines Center

At Home: 40 years old. Married, no children. Hobbies: charity work, outdoor activities, shoppingAlma Mater: University of Texas at Austin; B.S. in speech communication

HOW SHE GOT HERE: 2000 to present: Vice President, Digital Media Services, AA Center

1986 to 2000: Account executive, Xerox Corp.

MOUTHING OFF: What a wireless network means for AA Center: "It enhances the building, makes it more secure and helps us manage it; all of these things ultimately affect our guests. And it sets us up to move into the future."

Most common question from fans: "Whatever happened to getting the Internet in the seats?"Greatest business challenge: "Finding new, out-of-the-box technology opportunities for our sponsors."

Most misunderstood aspect of my job: "What I do. Nobody knows what a VP of digital media services does."

I love technology when: "It works."

I hate technology when: "It doesn't work--or I don't understand it."

My next career: "Growing this one."When I retire, I will: "I don't think I'll ever retire. I love the energy in working and making things happen."

Favorite Dallas Maverick: "Dirk Nowitzki. He's awesome."