SDN: Is Big Data a Killer App?

Software-defined networking is a hot topic, and for good reason. Just as the hypervisor transformed the data center, SDN has the potential to transform the data center network, and in the process enable a new wave of powerful applications. If done right, SDN will speed the provisioning of network services and, perhaps more importantly, create a new software layer that can be used by applications to extract more value out of the network and the computing systems that they connect.

I've been involved with IT Infrastructure for pretty much all of my career. As much as I love it, I have to remind myself that infrastructure is important only to the extent that it supports the delivery of valuable applications. Applications produce results that have meaning and value. Infrastructure--that is, computing, storage and networking--are just the means to that end.

The term "application" shows up a lot in SDN conversations, and that's good, because it more closely links SDNs to where the real value is created. But I also wonder if the term as used today is potentially misleading. For example, consider that in the canonical architecture diagram that's emerged around OpenFlow-based SDN, firewalls and load balancers are "applications."

I'm not sure I buy that. I've always known those things to be "infrastructure," like routers and switches. Excel and Oracle Financials and Hadoop are applications. With SDN, the network infrastructure is now programmable (that is, software-driven), so I suppose firewalls and load balancers can be seen as applications in the sense that they are software programs that instantiate functions in the network via the SDN controller. That said, I think firewalls and load balancers are better labeled as "software-defined infrastructure."

What I'm wondering is if SDN will cause an IT revolution in application development. Think of the PC: While PC makers had to get the hardware right, it was the emergence of killer applications such as spreadsheets and desktop publishing that really drove adoption.

A true killer app for SDN would be one with significant economic benefits to lots of companies, and one that requires SDN in order to be viable on a broad scale. I think big data may be one such application area. You're probably thinking, "Oh, great--let's take over-hyped SDN and multiply it by over-hyped big data and see what we get!" That's fair, and all we know that the answer will lie somewhere between huge and zero. But a few data points suggest there may be something here.

First, big data has unlocked big value for some companies. For example, it's been reported that Google generates 95% of its revenue from the ability to target ad placements based on search terms. That's an extreme and as-yet unduplicated example, but big data apps will be used pervasively if they can drive even a few percentage points of revenue (or save equivalent costs) for many organizations.

Second, a theoretical analysis presented this summer by researchers from IBM and Rice University suggested that network-aware applications could reduce execution times for key Big Data operations by 70%.

In a more recent experiment, which was conducted by Infoblox, an SDN-aware version of Hadoop achieved a 40% reduction on a key Hadoop benchmark when executed over an SDN network. The Hadoop application set up prioritized queues (flows) in the underlying OpenFlow switches at initialization, giving the highest priority to traffic from particular HTTP ports--specifically, those ports corresponding to the most latency-sensitive Hadoop traffic. As a result, critical Hadoop traffic was forwarded ahead of other, less critical traffic, and so the job completed faster, even when executing on a congested network.

The 40% performance improvement is impressive--even more so when you consider there was no feedback or runtime optimization. Applications that speak SDN can dynamically monitor network state along with app execution and make network optimizations on-the-fly.

The implications of this simple experiment could be significant for SDNs and big data. Applications like Hadoop rely on distributed computing in order to achieve acceptable performance. Distributed computing requires a network. If SDNs can enable applications to make better use of distributed processing resources, then big data applications can deliver acceptable performance using much less computing hardware, making them affordable for more organizations. That could make big data a killer app for SDNs.

This is just one example. Delivery of rich media like video and audio also has potential. Just as the PC (and now mobile devices) unleashed waves of programmer creativity, it will be exciting to see what programmers--perhaps flowgrammers?--will do when their applications can define the network.

Richard Kagan has spent the last 30 years working in telecommunications, networking and security.