Yesterday, Amazon’s S3 storage service went down in its Northern Virginia datacenter for the better part of 4 hours, and took parts of a bunch of prominent websites and services with it….including ours. If you missed this story, you can read about it here.
We pride ourselves on the quality of our real estate coaching & training services, and want to apologize to all our members & listeners for the service interruption they experience on our site.
This was a surprising & frankly somewhat scary event, because we selected Amazon S3 and Cloudfront as our media-hosting platform precisely because of it’s speed, stability & track-record of performance. Obviously we were only 1 of many companies affected by this, and in truth the issue wasn’t so much Amazon itself as it is a symptom of potential weakness in the modern internet.
When DARPA originally sponsored the internet, one of their primary goals was to create a non-stop, decentralized communications network capable of surviving a nuclear war. Thus, when outages occur, they typically don’t affect the entire internet – only isolated nodes. That creates an inconvenience, but ultimate doesn’t threaten the entire ecosystem.
Outages have happened, and they’ve almost always been isolated by region, telecommunications carrier, or internet service provider. There have been a few bad ones, especially during the 1990’s when demand for bandwidth outstripped availability – but generally speaking, the decentralized nature of the internet has limited any kind of catastrophic failure.
Yesterday’s outage was fairly catastrophic, though. Major parts of the internet literally stopped, leaving users simply unable to access websites, databases, and other services. So how, in a decentralized communications network, does something like this happen?
Over the last 10 years, many of the internet services that we all use have migrated to cloud based storage & processing. For us, this means media hosting – for others, it’s meant databases, hosted software applications, and even entire websites. While the hosting for most of the websites that carry these services is decentralized, the services themselves are becoming increasingly centralized with large providers like Google, Cloudflare….and Amazon Web-Services.
Cloudflare recently reported a major security leak that could have potentially exposed private data to hackers, coming only 2 months after Yahoo! was breached by hackers with a loss of data for over a billion accounts. Now Amazon – except this time not hackers (that we know of), only a data-center outage.
The question in the long-term is how centralized communications works in a decentralized environment. Imagine for a moment if instead of Amazon the outage had been with Apple – what would it mean for billions of iPhones to suddenly lose some or all of their connectivity? That’s only one example among many, and it sheds light on a new danger to the internet that we all need to take more seriously.
