Technology: Not as Easy as It Looks

• We used to build heavily routed networks that followed the physical characteristics of our buildings and environments. Now, at least in theory, VLANs can simplify network management by separating networks logically rather than physically.

• We used to spend countless hours training and supporting users as they grappled with interfaces that were anything but intuitive. Now we save precious time thanks to the advent of HTTP/HTML user interfaces.

So, how can we say IT is becoming more complex and testing is getting harder? Software engineer Matt Quail's "law of software complexity" theory (madbean.com/blog/43) explains the dichotomy this way: "The underlying complexity of a given problem is constant. It can be hidden, but it does not go away. ... Complexity is conserved by abstractions. In fact, apparent complexity can be increased by abstractions, but the underlying complexity can never be reduced."

Quail goes on to say that complexity can be shifted by creating abstractions that hide it at one level but let it linger at other levels. He also says these abstractions leak, causing the hidden complexity to seep out over time. And, he says, since the nature of a problem is constant, you have to change the bounds (definition) of the problem if you're going to simplify matters.

I think he's on to something.Take network protocols. The seven-layer OSI model is a prime example of how abstractions don't reduce complexity but can break it into manageable chunks. We've created many protocols to manage the various layers, but none of the layers has become less complex; we've just narrowed down the number of protocols so we can focus on the few--Ethernet and TCP/IP, for instance--that work well. In Quail's parlance, we've changed the bounds of the problem to exclude the other protocols. Ethernet and TCP/IP do leak, however, and we've had to do some tweaking as network speeds have increased. Sooner or later we'll run out of IP address space, too, so we'll have to shift to IPv6.

Authentication is another example. There are 5,642 different ways to authenticate over a network, according to my estimates. Eric Rescorla, author of numerous security Internet drafts and RFCs, recently wrote an Internet draft taxonomy of authentication mechanisms (www.ietf.org/internet-drafts/draft-iab-auth-mech-01.txt). His taxonomy is based on the fact that we have multiple protocols to solve a single problem. Rescorla says the problem is severe enough that clients and servers are forced to go through multiple levels of negotiation just to determine the protocol they'll use for the authentication itself. Yick!

And now back to user interfaces. It's easy to slap a front-end GUI on just about any application, but that doesn't necessarily simplify the app, no matter how much the marketers would like us to believe it does. When something goes wrong with an application, we still need to understand the inner mechanisms before we can diagnose and solve the problem.

So here's a restatement of Quail's law that may seem obvious but demands our attention: If you build a network or an application system for which you can find a more elegant, simpler solution, your original "solution" introduced unnecessary abstractions. And while they can hide complexity, they sure can't solve it. So much for technology testing--and life--getting easier.

Post a comment or question on this story.