3.6 million websites taken offline after fire at OVH datacenters
10th March, 2021
Around 3.6 million websites across 464,000 distinct domains were taken offline after the major fire at an OVHcloud datacenter site in Strasbourg overnight.
More than 18% of the IP addresses attributed to OVH in Netcraft’s most recent Web Server Survey — which took place two weeks ago — were no longer responding at 06:00-07:15 UTC this morning.
Thankfully, everybody is safe; but OVH said the fire in its SBG2 datacenter was not controllable and no data is likely to be recoverable. Part of its SBG1 datacenter has also been destroyed. Firefighters were protecting SBG3 throughout the night, and although there was no direct fire impact on SBG4, it was also unavailable due to the whole site being isolated. Consequently, all services in SGB1-4 have been offline.
Websites that went offline during the fire included online banks, webmail services, news sites, online shops selling PPE to protect against coronavirus, and several countries' government websites.
Examples of the latter included websites used by the Polish Financial Ombudsman; the Ivorian DGE; the French Plate-forme des achats de l’Etat; the Welsh Government’s Export Hub; and the UK Government’s Vehicle Certification Agency website, which got a new SSL certificate by 10am and is now back online with a UK hosting company.
Unsurprisingly for a French hosting company, the most affected country code top-level domain (ccTLD) is
.fr, which had 184,000 knocked-out websites spread across 59,600 distinct domain names – these account for 1.9% of all
.fr domains in the world. In comparison, there were only 24,100
.uk websites hosted in the affected datacenters, across just 8,700 unique domains. Most of the affected websites use the generic
.com top-level domain, amounting to 880,000 websites across 180,000 domains.
Posted by Paul Mutton in Around the Net, Hosting, Performance, Security
BTC-e: Better hosting than the Feds
4th August, 2017
The btc-e.com domain – previously operated by the BTC-e Bitcoin exchange – has barely been online since being seized by the US authorities on 28 July.
Since being seized, the btc-e.com domain has pointed to a different web server, hosted by 1&1 Internet in the United States. It now displays nothing more than a customary seizure notice, announcing that it has been seized as part of a joint law enforcement operation involving the FBI, IRS, DoJ, FDIC, Homeland Security and the Secret Service.
But evidently, hosting a 383 KB PNG image on a static HTML page is harder than it might seem. Most requests to the new site either fail to connect, or are very slow – much slower than when the site was still operating as an exchange for Bitcoins and other cryptocurrencies. Back then, btc-e.com was served via the Cloudflare content delivery network, which explains the relatively stellar performance in the run-up to its seizure.
The seizure of btc-e.com relates to a large-scale money laundering operation, which included Bitcoins stolen from the now-defunct Mt.Gox exchange. It is not clear whether the poor performance of the new site is simply being caused by an unsuitable hosting platform, or by deliberate protest attacks from aggrieved parties. Users who had Bitcoins tied up in BTC-e may never get them back.
Posted by Paul Mutton in Around the Net, Dogfood, Domains, Hosting, Performance
BBC websites back to normal, DDoS mitigation reverted
5th January, 2016
The BBC's websites are now back to normal, four days after being taken down by an effective DDoS attack on New Year's Eve.
The BBC mitigated the attack within a few hours by moving its main website onto the Akamai content delivery network, which restored access to its millions of users. However, during this mitigation period, some of the BBC's other websites – which were still hosted at the BBC – remained mostly unreachable.
The BBC's DDoS mitigation was only temporary, and last night it moved its main website off Akamai, back onto a netblock owned by the BBC. This move resulted in another short outage on 4th January, followed by several hours of slightly slower response times within the UK. By the 5th January, the response times had settled down to be almost comparable with when it was using Akamai.
However, as expected, response times from other countries are no longer as fast as they were when the BBC's main website was hosted on the Akamai CDN. Response times from the US are notably slower, but currently no worse than they were before the DDoS attacks on New Year's Eve.
During the period in which the BBC's main website was hosted on the Akamai CDN, its legacy News website at news.bbc.co.uk remained hosted at the BBC. This was mostly unavailable during this period, with most client connection attempts being reset.
This site's availability was restored to normal at the same time that the main BBC website moved off Akamai. This suggests that the connection resets were a deliberate attempt to mitigate basic DDoS attacks, rather than as a direct side effect of a sustained DDoS attack. However, this approach was not ideal – while some browsers (such as Chrome) would automatically retry the connection attempt (often successfully), other browsers would give up at the first failure.
Posted by Paul Mutton in Performance, Security
BBC websites still suffering after DDoS attack
4th January, 2016
Since suffering a crippling DDoS attack on New Year's Eve, some BBC websites are still experiencing significant performance issues.
Around 07:00 UTC on 31 December 2015, the main BBC website at www.bbc.co.uk was knocked offline after being subjected to a distributed denial of service attack. For the following few hours, requests to the BBC website either eventually timed out, or were responded to with its 500 Internal Error test card page. A group called New World Hacking later claimed responsibility for the attack, which it carried out as a test of its capabilities.
The British Broadcasting Corporation is the public service broadcaster of the United Kingdom, and the outage had a significant impact on its user base: The BBC's news, sport, weather and iPlayer TV and radio catchup services are all delivered via www.bbc.co.uk.
At the time of the attack, www.bbc.co.uk was served from a netblock owned by the BBC. It seems that service was restored by migrating the site onto the Akamai content delivery network, after which there were no apparent outages.
|OS||Server||Last seen||IP address||Netblock Owner|
Moving www.bbc.co.uk onto the Akamai CDN also resulted in some significant performance benefits, particularly from locations outside of the UK. For example, prior to the attack, most requests from Netcraft's New York performance collector took around 0.4-0.6 seconds to receive a response, whereas after the site had migrated to Akamai, all requests were served in well under 0.1 seconds. These performance benefits are typical when using a globally distributed CDN, as cached content can be delivered from an edge server within the client's own country, rather than from a remote server that can only be reached via transatlantic cables.
However, not all of the BBC's websites have migrated to Akamai, and some of these are still exhibiting connectivity issues in the aftermath of the attack. For example, search.bbc.co.uk and news.bbc.co.uk are still hosted directly at the BBC, and these are still experiencing problems today.
The BBC's News service is currently found at www.bbc.co.uk/news, but up until a few years ago it used to be served from its own dedicated hostname, news.bbc.co.uk. This legacy hostname is still used by some webpages today, but mostly redirects visitors to the new site at www.bbc.co.uk/news. This conveniently collates all of the BBC's main online services under the same hostname, but at the expense of introducing a single point of failure. If each service were still to be found under a different hostname and on different servers, it might have offered further resilience to the initial attack.
As shown above, news.bbc.co.uk was also affected by the DDoS attack which took down the main BBC website, but eventually came back online later that day without having to relocate the website. However, the following morning (New Year's Day), it started to experience significant connectivity problems.
It is unclear whether this indicates a separate ongoing attack, or an attempt at mitigating such attacks, but nonetheless, it is likely to affect lots of users: Many old news articles are still served directly from news.bbc.co.uk, and some users habitually reach the news website by typing news.bbc.co.uk into their browsers. Some regularly updated pages also continue to be served from news.bbc.co.uk, such as horse racing results.
Posted by Paul Mutton in Performance, Security
PayPal.com unexpectedly offline
12th June, 2015
PayPal.com experienced some unexpected outages today, leaving many of its 165 million active customers unable to access the site via its paypal.com domain.
All of Netcraft's geographically distributed performance monitors have showed outages of at least an hour so far today.
But requests to www.paypal.com continued to work throughout the outage, and customers could successfully log in by browsing directly to that hostname; however, it is likely that many users visit the site by typing only "paypal.com" into their browser's address bar.
The shorter paypal.com address does not serve a homepage; its sole purpose is to redirect visitors directly to the main www.paypal.com website. This shortcut saves the user having to type "www." into his browser, but during the outage, anyone attempting to reach the site via this commonly used method would have faced error messages similar to this (unless the redirect has been cached from a previous visit):
Even a short outage can have a huge impact on a financial giant like PayPal. In 2014, PayPal handled 4 billion payments with a total payment volume of $235 billion.
The cause of the outage is as yet unknown: The PayPal Notifications website does not show any scheduled maintenance for the live site until tomorrow, and does not currently list any unscheduled events corresponding to today's outage.
Posted by Paul Mutton in Performance
Amazon goes down in Europe
16th February, 2015
Some of Amazon's European retail sites and video streaming services went down last night, causing a flurry of complaints across social media. The affected sites included amazon.co.uk, amazon.de and amazon.fr.
These outages are particularly notable, as Amazon has a considerable amount of experience hosting websites. It has one of the largest hosting infrastructures in the world, which is used not only by itself, but also by thousands of its Amazon Web Services customers.
Amazon is the world's largest hosting provider in terms of web-facing computers, accounting for more than 6% of the 5.1 million computers in Netcraft's February 2015 Web Server Survey. 52,000 of Amazon's web-facing computers are located in Ireland, which is where its European retail sites are hosted.
Amazon's presence in Ireland has grown astonishingly since Amazon Data Services Ireland opened the first of its three Irish EC2 Availability Zones in 2007. Remarkably, more than three-quarters of all web-facing computers in Ireland are now operated by Amazon, and these account for 2.7% of all web-facing computers in the Europe, Middle East and Africa region which it is designed to serve.
Amazon's US site at www.amazon.com, which is hosted in the US, was not affected by last night's outages.