Security experts warn mega internet crash could be the tip of the iceberg

View
comments

Security experts have warned that yesterday’s internet crash that took major sites offline could be just the tip of the iceberg. 

An outage at a little-known firm that speeds up access to websites knocked a lot of top internet destinations offline on Tuesday, disrupting business and leisure for untold millions globally. 

Millions of users across the globe reported problems trying to access web pages, with Netflix, Twitch and news websites including the BBC, Guardian, CNN and the New York Times hit by the issue. 

The problem was caused by the US firm Fastly, a content delivery network (CDN) company which helps users view website content more quickly. 

The problem was quickly resolved, and the company blamed a configuration error in its technology.

But the incident raises questions about how vulnerable the global internet is to more serious disruption.

San Francisco-based Fastly’s ‘edge server’ computing technology is used by many of the world’s most popular websites, such as The New York Times, Shopify, Ticketmaster, Pinterest, Etsy, Wayfair and Stripe. The British government is among its clients.

The company provides a content delivery network – an arrangement that allows customer websites to store data such as images and videos on various mirror servers across 26 countries so that the data is closer to users, and thus shows up faster. 

David Warburton, of the cybersecurity company F5 Labs, said centralisation is fairly new for the internet – and will probably continue to raise issues. He told The Guardian: ‘The web as a whole was intended to be decentralised. 

‘By not relying on any one central system, it meant that many different components could fail and internet traffic could still find a way to get where it needed to go. 

‘What we’ve seen over the past decade, however, is the unintentional centralisation of many core services through large cloud solution providers like infrastructure vendors and CDNs.’

Former deputy national security adviser for intelligence security between 2014 and 2018, Paddy McGuinness, said yesterday’s incident should serve as ‘a wake-up call’, and the Government should expand the current security approach.

He said: ‘We need resilience as an explicit policy goal, especially on the new networks we are building to deliver services to the citizen. A ”secure by design and default” mantra is welcome but it isn’t enough in itself.’

Many of Fastly’s customers are news sites that use its technology to update their websites with breaking news. 

Buzzfeed, for example, used Fastly to cut the time its users took to reach the site by half. Fastly had $290.9 million in revenues last year. 

Customers rely on Fastly and its rivals to host and protect their website data from denial-of-service attacks and disruption from spikes in traffic. Had this outage been more serious, customers could have moved to competitors such as Cloudflare or Akamai. 

But that’s not simple; many businesses would have had to scramble and might have suffered losses.

‘You can’t switch quickly to another service unless you had it set up ahead of time,’ said Doug Madory, an internet infrastructure expert with the traffic-measuring company Kentik. ‘If Fastly were down for a day, that would be pretty bad.’

Even if they do have an alternative provider, engineering a smooth switchover from one to another is not for the faint of heart, said Ben April, chief technical officer of Farsight Security.

Madory and other experts said Fastly and its competitors spend heavily and devote major engineering resources to reducing the possibilities of such outages and ensuring they can recover as quickly as Fastly did on Tuesday.

Such outages are not new – but not at all common. ‘There may be years between when a company has an outage like this,’ Madory added. ‘I think we are going to have these very rare but probably impactful short outages for the foreseeable future.’

Like the content distribution network world, cloud computing – when computing services are entrusted to a remote provider – is dominated by just a few major players led by Amazon Web Services, Google and Microsoft. Amazon, the biggest cloud provider, periodically has brief outages, which are a big deal for customers.

‘And if it became a major outage of, say, more than six, eight hours – but days – it could put companies out of business,’ said Josh Chessman, an analyst with the tech market researcher Gartner Inc.

The question is: What could cause such a serious outage that might destroy customer data? A major cyberattack is one possibility. Another is fire or catastrophic natural disaster. These businesses, after all, are based in datacenters. In March, a fire at a datacenter in Strasbourg, France, owned by a major cloud computing firm knocked out service to millions of websites.

Businesses and consumers should be thinking seriously about how much they should rely on the cloud for their most valuable data. ‘If there’s an outage, what’s the impact on our business?’ Chessman asked. Perhaps it makes sense not to rely on a cloud-based service for your company’s email if you’d go bankrupt without it during a two-week outage.

But running your own email and backup services is complicated and costly – one reason companies turned to the cloud in the first place.

David Vaskevitch, a former Microsoft chief technical officer and CEO of the photo management app Mylio, said people have grown so accustomed to the always-on internet – everywhere we go, we carry a pocket computer with us – that we wrongly assume it will be available.

‘It’s not very realistic and it’s not a good way to live,’ said Vaskevitch, 67. ‘The internet is always there – until it isn’t.’

Despite the vast interconnectedness of the world, it can still be wise to store some data locally, said Vaskevitch. Instead of streaming all our music, we should think about saving some locally. Same goes for email – for instance, in an arrangement where you store it on the computing device you use most.

‘Your device is both the best friend of the internet and the best insurance policy,’ he said. ‘When the internet goes down, if you arrange things carefully, you can still do most of the things you need to do.’   

Hundreds of websites worldwide crashed yesterday morning following a massive internet outage – with the UK government, Amazon and Spotify among those experiencing issues. 

The aim of CDNs is to reduce latency – the delay from the moment a user makes a request to the exact instant they receive a response. The higher the latency, the worse the user experience. 

But if the service suffers a failure, as Fastly’s did yesterday, it prevents the companies that use it from operating on the net at all. 

Many of the world’s biggest websites run on the ‘edge cloud’ platform’s network, hence the mass outage. 

Fastly first posted an error message at 10.58 BST (05.58 ET), saying it was ‘investigating potential impact to performance with our CDN services’.

It later tweeted shortly after midday UK time: ‘We identified a service configuration that triggered disruptions across our POPs globally and have disabled that configuration. 

‘Our global network is coming back online.’ 

Users took to social media to vent their frustrations about the outage.

One called it an ‘internet apocalypse’, while another said ‘everything just shut out of nowhere’.

Another tweeted that the internet was ‘broken’.

The outage saw visitors to a vast array of sites receive error messages including ‘Error 503 Service Unavailable’ and ‘connection failure.’

Streaming sites Netflix, Twitch and Hulu were also hit by the problem. 

Some sites including the UK government website were offline entirely, while others such as Twitter had more specific errors, such as not showing emojis.  

Travelling Britons revealed their frustration at not being able to complete their passenger locator form because the Gov.UK website was down. 

Among them was Priya Bhargava from London, who tweeted: ‘@GOVUK hello your website is down I need to submit a passenger locator form by this eve. Pls can this get looked at ASAP. Thanks!!!’ 

Another, Jo Thornhill, tweeted: ‘@GOVUK your website is down and I need to complete a passenger locator form ASAP.’

And a third, Richard Pearson, from Nottingham, said: ‘Need to fill out passenger locator forms to return to the uk but http://gov.uk is down so I can’t. Great.’

Passenger locator forms are required by British border officials for those returning from all countries abroad.

These must be completed online at Gov.UK, although those aged under 18 may be included on adults’ forms if they are staying at the same UK address.

The form details your home address, passport number and test package booking reference.

The official Gov.UK Twitter account said: ‘We are aware of the issues with http://GOV.UK which means that users may not be able to access the site. 

‘This is a wider issue affecting a number of other non-government sites. We are investigating this as a matter of urgency.’ 

Outage tracker website DownDetector also reported problems for Squarespace, Shopify, Vimeo, Imgur, Tidal, Weightwatchers and Kickstarter. UK chemist Boots was also affected.

The Guardian earlier tweeted: ‘The Guardian’s website and app are currently being affected by a wider internet outage and will be back as soon as possible.’

Other websites hit by the issue included the online discussion platform Reddit and French newspaper Le Monde. 

A CDN is a platform of servers that helps minimize delays in loading web page content.

Jake Moore, a cybersecurity specialist at ESET, said: ‘Web pages are located all over the world so content delivery networks are placed to distribute the data evenly by reducing the physical distance between where it’s actually held and the end user.

‘This helps users around the world view the same high quality information and content without any lag or slow loading times.

‘With Fastly down, millions of web pages will be affected.’