BBC websites back to normal, DDoS mitigation reverted

The BBC's websites are now back to normal, four days after being taken down by an effective DDoS attack on New Year's Eve.

The BBC mitigated the attack within a few hours by moving its main website onto the Akamai content delivery network, which restored access to its millions of users. However, during this mitigation period, some of the BBC's other websites – which were still hosted at the BBC – remained mostly unreachable.

The BBC's DDoS mitigation was only temporary, and last night it moved its main website off Akamai, back onto a netblock owned by the BBC. This move resulted in another short outage on 4th January, followed by several hours of slightly slower response times within the UK. By the 5th January, the response times had settled down to be almost comparable with when it was using Akamai.

The main BBC website experienced another short outage last night as it moved off the Akamai CDN.

The main BBC website experienced another short outage last night as it moved off the Akamai CDN.

However, as expected, response times from other countries are no longer as fast as they were when the BBC's main website was hosted on the Akamai CDN. Response times from the US are notably slower, but currently no worse than they were before the DDoS attacks on New Year's Eve.

Response times from the US are now much slower again (although international visitors would typically visit bbc.com rather than bbc.co.uk).

Response times from the US are now much slower again (although international visitors would typically visit bbc.com rather than bbc.co.uk).

During the period in which the BBC's main website was hosted on the Akamai CDN, its legacy News website at news.bbc.co.uk remained hosted at the BBC. This was mostly unavailable during this period, with most client connection attempts being reset.

news.bbc.co.uk is now functioning normally, too.

news.bbc.co.uk is now functioning normally, too.

This site's availability was restored to normal at the same time that the main BBC website moved off Akamai. This suggests that the connection resets were a deliberate attempt to mitigate basic DDoS attacks, rather than as a direct side effect of a sustained DDoS attack. However, this approach was not ideal – while some browsers (such as Chrome) would automatically retry the connection attempt (often successfully), other browsers would give up at the first failure.

BBC websites still suffering after DDoS attack

Since suffering a crippling DDoS attack on New Year's Eve, some BBC websites are still experiencing significant performance issues.

Around 07:00 UTC on 31 December 2015, the main BBC website at www.bbc.co.uk was knocked offline after being subjected to a distributed denial of service attack. For the following few hours, requests to the BBC website either eventually timed out, or were responded to with its 500 Internal Error test card page. A group called New World Hacking later claimed responsibility for the attack, which it carried out as a test of its capabilities.

Requests that did not time out were eventually met with the BBC test card error page.

Requests that did not time out were eventually met with the BBC test card error page.

The British Broadcasting Corporation is the public service broadcaster of the United Kingdom, and the outage had a significant impact on its user base: The BBC's news, sport, weather and iPlayer TV and radio catchup services are all delivered via www.bbc.co.uk.

Performance chart for www.bbc.co.uk, showing the primary outage period.

Performance chart for www.bbc.co.uk, showing the primary outage period.

At the time of the attack, www.bbc.co.uk was served from a netblock owned by the BBC. It seems that service was restored by migrating the site onto the Akamai content delivery network, after which there were no apparent outages.

OS Server Last seen IP address Netblock Owner
Linux nginx 3-Jan-2016 88.221.48.170 Akamai
Linux nginx 2-Jan-2016 95.101.129.88 Akamai Technologies
Linux nginx 31-Dec-2015 95.101.129.106 Akamai Technologies
Linux nginx 30-Dec-2015 212.58.244.70 BBC
Linux nginx 29-Dec-2015 212.58.246.54 BBC
Linux nginx 28-Dec-2015 212.58.244.71 BBC

Moving www.bbc.co.uk onto the Akamai CDN also resulted in some significant performance benefits, particularly from locations outside of the UK. For example, prior to the attack, most requests from Netcraft's New York performance collector took around 0.4-0.6 seconds to receive a response, whereas after the site had migrated to Akamai, all requests were served in well under 0.1 seconds. These performance benefits are typical when using a globally distributed CDN, as cached content can be delivered from an edge server within the client's own country, rather than from a remote server that can only be reached via transatlantic cables.

Performance chart for www.bbc.co.uk from  New York, highlighting the improved response times and successful attack  mitigation after switching to Akamai.

Performance chart for www.bbc.co.uk from New York, highlighting the improved response times and successful attack mitigation after switching to Akamai.

However, not all of the BBC's websites have migrated to Akamai, and some of these are still exhibiting connectivity issues in the aftermath of the attack. For example, search.bbc.co.uk and news.bbc.co.uk are still hosted directly at the BBC, and these are still experiencing problems today.

The BBC's News service is currently found at www.bbc.co.uk/news, but up until a few years ago it used to be served from its own dedicated hostname, news.bbc.co.uk. This legacy hostname is still used by some webpages today, but mostly redirects visitors to the new site at www.bbc.co.uk/news. This conveniently collates all of the BBC's main online services under the same hostname, but at the expense of introducing a single point of failure. If each service were still to be found under a different hostname and on different servers, it might have offered further resilience to the initial attack.

The performance chart for news.bbc.co.uk shows massive outages long after the DDoS attack on New Year's Eve.

The performance chart for news.bbc.co.uk shows massive outages long after the DDoS attack on New Year's Eve.

As shown above, news.bbc.co.uk was also affected by the DDoS attack which took down the main BBC website, but eventually came back online later that day without having to relocate the website. However, the following morning (New Year's Day), it started to experience significant connectivity problems.

Most requests to news.bbc.co.uk are still failing.

Most requests to news.bbc.co.uk are still failing. Some browsers, such as Chrome, may automatically retry the request.

It is unclear whether this indicates a separate ongoing attack, or an attempt at mitigating such attacks, but nonetheless, it is likely to affect lots of users: Many old news articles are still served directly from news.bbc.co.uk, and some users habitually reach the news website by typing news.bbc.co.uk into their browsers. Some regularly updated pages also continue to be served from news.bbc.co.uk, such as horse racing results.

Most Reliable Hosting Company Sites in December 2015

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 EveryCity SmartOS 0:00:00 0.000 0.093 0.064 0.128 0.128
2 Lightcrest unknown 0:00:00 0.004 0.276 0.006 0.023 0.027
3 One.com Linux 0:00:00 0.004 0.203 0.037 0.105 0.105
4 Memset Linux 0:00:00 0.004 0.153 0.064 0.157 0.245
5 ServerStack Linux 0:00:00 0.004 0.125 0.067 0.135 0.135
6 Netcetera Linux 0:00:00 0.004 0.075 0.084 0.171 0.171
7 Codero Citrix Netscaler 0:00:00 0.004 0.177 0.092 0.189 0.381
8 GoDaddy.com Inc Linux 0:00:00 0.008 0.264 0.007 0.018 0.018
9 Datapipe Linux 0:00:00 0.008 0.145 0.012 0.024 0.031
10 Swishmail FreeBSD 0:00:00 0.008 0.144 0.062 0.125 0.168

See full table

EveryCity had the most reliable hosting company site in December 2015. Despite moving into new offices, its website was the only one to respond to all of Netcraft's requests. EveryCity has maintained its 100% uptime record throughout 2015, and has made it into the top ten 11 times during the year. It also had the most reliable hosting company site in May.

In second place in December was Lightcrest, which also appeared in the top ten in November. It experienced only one failed request, with an impressively fast average connection time of 6 milliseconds. Lightcrest operates its cloud services using its own Kahu Compute Fabric infrastructure, without outsourcing any components to third-party cloud providers.

In third place – also with a single failed request, but with a slower average connection time – was One.com. Established in 2002, One.com now has over 270 employees with companies registered in Denmark, India and Dubai.

Six of December's top ten hosting company sites ran on Linux operating systems, while Swishmail used FreeBSD, Codero used a Citrix Netscaler device, and EveryCity used SmartOS. The latter is a community fork of OpenSolaris, featuring the ZFS file system, DTrace dynamic tracing, kernel-based virtual machines and Solaris Zones operating system-level virtualisation.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

December 2015 Web Server Survey

In the December 2015 survey we received responses from 901,002,770 sites and 5,579,077 web-facing computers, reflecting a loss of 2.0 million sites, but a gain of 39,900 computers.

Apache suffered the largest loss of 13.4 million sites, followed by Microsoft, which lost 5.0 million. A good part of this month's overall losses were caused by expired .xyz domains, which resulted in nearly 9 million .xyz websites disappearing from the internet. Despite the widespread losses caused by the demise of these websites, nginx managed to gain 7.1 million sites overall, which was the largest growth seen by any web server vendor.

The .xyz top-level domain was made available to the general public on 2 June 2014 and immediately received strong support from Network Solutions, which registered nearly 100,000 .xyz domains during the first ten days of operation. Controversially, Network Solutions gave away many .xyz domains for free to customers who already had the corresponding domain under the .com TLD. This was done on an opt-out basis, and the domains were only free for the first year, leaving some customers surprised when each domain became due for renewal at a cost of $38 this year.

Google's parent company, Alphabet Inc, is one of the most notable users of the .xyz TLD with the domain abc.xyz, while some of the other popular .xyz sites include adult sites and torrent search engines. The .xyz TLD has also proven reasonably popular with fraudsters: Netcraft found phishing sites on 150 .xyz domains throughout November 2015.

This month's changes have caused Apache's leading market share to fall by 1.41 points to 35.6%, while nginx's site share has increased to 17.4%. A little over a year ago, Microsoft was in the lead, but has recently been floating around in second place, currently 9.2 percentage points ahead of nginx, and 9.0 behind Apache.

As well as gaining the largest number of sites this month, nginx also showed the largest growth in terms of web-facing computers, growing by 17,000 to reach a total of 765,000. Despite their site losses, Apache and Microsoft also gained a reasonable number of web-facing computers (10,400 and 6,100), while Lighttpd and Google suffered small losses.

A relatively unknown web server, Safedog, was found serving nearly ten times as many websites as last month, making it now the 7th most commonly used web server software with 6.3 million hostnames. However, the number of web-facing computers with Safedog installed is very low – less than 300 – and nearly all of these are running the deprecated Windows Server 2003 operating system. All websites using this Chinese server software claim to be running Safedog 4.0.0, which appears to be a cloud security system.

2015 has been a turbulent year in terms of hostnames, with the total number of sites rising from 877 million in January, to 901 million in December, but dipping as low as 849 million in April. Apache has continued to lead the market throughout the year, with Microsoft following in second place, getting to within 4.1 percentage points of Apache's share in October. In web-facing computers, nginx has shown remarkably consistent growth in its market share, while both Apache and Microsoft have declined. nginx is now installed on 13.71% of all web-facing computers, compared with 11.03% at the start of the year, and its market share within the top million sites has also grown noticeably from 21.09% to 24.29%.

Total number of websites

Web server market share

DeveloperNovember 2015PercentDecember 2015PercentChange
Apache334,095,10237.00%320,676,75935.59%-1.41
Microsoft244,906,58627.12%239,927,01326.63%-0.49
nginx149,967,73316.61%157,001,01817.43%0.82
Google19,622,6242.17%20,362,6782.26%0.09
Continue reading

Most Reliable Hosting Company Sites in November 2015

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 One.com Linux 0:00:00 0.000 0.211g 0.035 0.100 0.100
2 GoDaddy.com Inc Linux 0:00:00 0.009 0.258 0.008 0.019 0.020
3 Datapipe Linux 0:00:00 0.009 0.145 0.012 0.025 0.031
4 XILO Communications Ltd. Linux 0:00:00 0.009 0.213 0.063 0.126 0.126
5 EveryCity SmartOS 0:00:00 0.009 0.094 0.065 0.131 0.131
6 Anexia Linux 0:00:00 0.009 0.191 0.083 0.169 0.169
7 Hivelocity Linux 0:00:00 0.009 0.174 0.087 0.174 0.174
8 Lightcrest unknown 0:00:00 0.013 0.278 0.007 0.019 0.023
9 INetU Linux 0:00:00 0.013 0.146 0.066 0.131 0.131
10 ServerStack Linux 0:00:00 0.013 0.130 0.066 0.132 0.132

See full table

One.com topped the table after successfully responding to all of Netcraft's requests in November. It last appeared in the top 10 in June 2015 when it placed ninth. The Denmark-based company offers shared hosting packages — all of which include unlimited traffic — from data centres operated by Interxion.

After having the most reliable hosting company site for the past three months, GoDaddy narrowly missed the top spot in November. Although five other sites also had two failed requests, GoDaddy's site is placed second based on its faster average connection time of 8ms.

Also with just two failed requests, Datapipe placed third in November. Datapipe has appeared in all but one of the top 10 lists in 2015 and has maintained a 100% uptime record for more than nine years. For the seventh time, Datapipe was recognised as one of New Jersey's 50 fastest growing companies, after a year that included the acquistion of GoGrid and DualSpark.

The top 10 is once again dominated by websites using Linux, which is used by eight of the top 10 sites. EveryCity, the only site that has been in the top 10 for every month of this year so far, uses SmartOS. Microsoft Windows is absent from the list for the fifth consecutive month.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

World Bank hacked by PayPal phishers

Hackers have broken into a website operated by the World Bank Group, which was subsequently exploited to host a convincing PayPal phishing site. The fraudulent content deployed on the site was able to benefit from the presence of a valid Extended Validation SSL certificate.

Extended Validation certificates can only be issued to organisations that have gone through a stringent set of verification steps, as required by the CA/Browser Forum. To recognise the high level of assurance offered by an EV certificate, most browser software will display the organisation's name in a prominent green box next to the address bar.

A PayPal phishing site, using an Extended Validation SSL certificate issued to the World Bank Group.

A PayPal phishing site, using an Extended Validation SSL certificate issued to the World Bank Group.

The EV vetting process effectively guarantees that the domain used in this attack is operated by the organisation specified in the certificate, which in this case is the World Bank Group. Implicatively, any visitor to this site is likely to trust the content it displays.

But of course, this guarantee goes out the window if the site has been compromised by an attacker. That's exactly what happened on Tuesday, when fraudsters deployed a PayPal phishing site into a directory on climatesmartplanning.org, allowing the fraudulent content to be served with an EV certificate issued to The World Bank Group.

The Climate-Smart Planning Platform is an initiative led by The World Bank, which makes it easier for developing-country practitioners to locate and access the tools, data and knowledge they need for climate-smart planning. Given its noble goals, it seems a shame that its website has been affected by this fraudulent activity.

The day after the attack, the website became temporarily unavailable (displaying only a Red Hat Enterprise Linux test page), before later coming back online with the fraudulent content removed. But today, it became evident that the site is still vulnerable to attack, as its homepage has now been defaced by a group called "Virus iraq".

A World Bank Group website hacked by "Virus iraq".

A World Bank Group website hacked by "Virus iraq" (19 November, 2015).

This is not the only time The World Bank's reputation has been tainted by the work of fraudsters – its name is also often used in 419 scams.

Tuesday's phishing attack started off by asking the victim to enter his or her PayPal email address and password. These credentials were submitted to a logcheck.php script on the server, which carried out some validation to prevent bogus data clogging up the phisher's haul.

The phishing site rejects invalid email addresses.

The phishing site rejects invalid email addresses.

After logging these stolen credentials, the phishing site claims it is temporarily unable to load the user's account. The victim is prompted to confirm their "informations" in order to access their account.

The next page asks for several details that would help the fraudster carry out identity theft. These details include the victim's name, date of birth, address and phone number. After these have been submitted, the victim is prompted to confirm payment card details by entering his full card number, expiry date and CSC (CVV) number.

The previous page also has a checkbox to specify whether or not the victim's card uses Verified by Visa or MasterCard SecureCode. If this box is checked, the next page will prompt the user to enter his 3-D Secure password, thus allowing the attacker to make fraudulent purchases on sites that are are protected by these additional layers of security.

Stealing the victim's 3-D Secure password.

Stealing the victim's 3-D Secure password.

After this final password has been stolen, the victim is redirected to the genuine PayPal website, leaving the attacker with the ability to make fraudulent purchases using either the victim's PayPal account or credit card.

At the time of writing, the Climate-Smart Planning Platform website remains defaced, but the phishing content has been removed.