Code Spaces: A Lesson In Cloud Backup

Over the past few years, I’ve been getting that foreboding feeling when I listen to the clouderati natter on about how the cloud is the solution to all your IT problems. If, they argue, you’d just put all your applications in the cloud you’d never have to back up a server again. Unfortunately, the folks that ran Code Spaces drank their own Flavor Aid and are now joining Monty Python’s Norwegian Blue parrot in pining for the fjords. 

The story of Code Spaces' death is a relatively simple one. It set up shop as a SaaS provider to software developers offering source code management tools like Git and Apache Subversion, all on Amazon Web Services (AWS). As long as things were good at AWS, things were OK at Code Spaces as well.

Then the hacker appeared. The hacker somehow gained access to Code Spaces' AWS control panel account, then got the folks at Code Spaces’ attention by mounting a denial-of-service attack and delivering a ransom note through the AWS control panel.

The Code Spaces folks responded as geeks everywhere would: They started nailing down their control panel, changing passwords and the like. However, the hacker was watching them via a backdoor account he created, and when he saw them locking doors, he set off the metaphorical bomb he’d hidden in the basement and started deleting AWS objects like EC2 machine definitions, S3 buckets, and, of course, all the backups. 

As you can see in the message at www.codespaces.com, that was enough to kill Codes Spaces as a going concern. At the moment, the Code Spaces crew is doing what I’m sure is yeomen’s work recovering what can be recovered for their customers. Then it’s resumé writing time.

What can we learn from this sad, sad story? Several things. For one thing, there was a failure of security. A company as dependent on AWS as Code Spaces should have instituted some basic security options that would have made deleting its primary data and backups a lot harder.

The first security option is multi-factor authentication. If you can afford the cloud service, you can afford the extra few dollars a month to require two-factor authentication with a key fob. Code Sources doesn’t know who the hacker was; if access to its accounts required a key fob, the company would know whose fob was being used.

Speaking of which, isn’t there, or shouldn’t there be, the equivalent of an Emergency Power Off (EPO) for AWS management? I mean a button that would prevent any object state changes, somewhat like ESX’s maintenance mode, until the cloud provider can really verify the identity of an administrator?

What Code Spaces apparently didn’t do was implement simple role-based management: Rather than give one user account full administration to the AWS infrastructure, create several different accounts. After all, deleting a backup is, or should be, a rare occurrence. If backup deletion is limited to a specific account, and the key fob for that account is kept in the office safe, the backups would have been safer.

However, I think it’s narrow minded to view this event as strictly a security failure. The Code Spaces folks were also guilty of violating the backup administrator’s rule of 3-2-1, which requires three copies of any piece of data, on two different media with one offsite.

By storing their primary and backup copies of data on AWS, they failed to be on two different types of media and be in two locations. Some could argue that S3 in multiple regions satisfies the requirement of two locations, but as long as it’s one console, I wouldn’t count it as offsite. That's because it remains vulnerable to this type of administrator identity attack, which is, frankly, more likely to be a disgruntled employee/ex-employee than a hacker.

The backup of your cloud environment should be at very least to cloud storage managed by another account that shares nothing, except possibly billing information, with the primary account. Sending those backups to a completely separate service would make me sleep even better at night. Corporate IT has to still be responsible for keeping data safe, even if it’s been sent to the cloud.