BBC websites still suffering after DDoS attack

Since suffering a crippling DDoS attack on New Year's Eve, some BBC websites are still experiencing significant performance issues.

Around 07:00 UTC on 31 December 2015, the main BBC website at www.bbc.co.uk was knocked offline after being subjected to a distributed denial of service attack. For the following few hours, requests to the BBC website either eventually timed out, or were responded to with its 500 Internal Error test card page. A group called New World Hacking later claimed responsibility for the attack, which it carried out as a test of its capabilities.

Requests that did not time out were eventually met with the BBC test card error page.

Requests that did not time out were eventually met with the BBC test card error page.

The British Broadcasting Corporation is the public service broadcaster of the United Kingdom, and the outage had a significant impact on its user base: The BBC's news, sport, weather and iPlayer TV and radio catchup services are all delivered via www.bbc.co.uk.

Performance chart for www.bbc.co.uk, showing the primary outage period.

Performance chart for www.bbc.co.uk, showing the primary outage period.

At the time of the attack, www.bbc.co.uk was served from a netblock owned by the BBC. It seems that service was restored by migrating the site onto the Akamai content delivery network, after which there were no apparent outages.

OS Server Last seen IP address Netblock Owner
Linux nginx 3-Jan-2016 88.221.48.170 Akamai
Linux nginx 2-Jan-2016 95.101.129.88 Akamai Technologies
Linux nginx 31-Dec-2015 95.101.129.106 Akamai Technologies
Linux nginx 30-Dec-2015 212.58.244.70 BBC
Linux nginx 29-Dec-2015 212.58.246.54 BBC
Linux nginx 28-Dec-2015 212.58.244.71 BBC

Moving www.bbc.co.uk onto the Akamai CDN also resulted in some significant performance benefits, particularly from locations outside of the UK. For example, prior to the attack, most requests from Netcraft's New York performance collector took around 0.4-0.6 seconds to receive a response, whereas after the site had migrated to Akamai, all requests were served in well under 0.1 seconds. These performance benefits are typical when using a globally distributed CDN, as cached content can be delivered from an edge server within the client's own country, rather than from a remote server that can only be reached via transatlantic cables.

Performance chart for www.bbc.co.uk from  New York, highlighting the improved response times and successful attack  mitigation after switching to Akamai.

Performance chart for www.bbc.co.uk from New York, highlighting the improved response times and successful attack mitigation after switching to Akamai.

However, not all of the BBC's websites have migrated to Akamai, and some of these are still exhibiting connectivity issues in the aftermath of the attack. For example, search.bbc.co.uk and news.bbc.co.uk are still hosted directly at the BBC, and these are still experiencing problems today.

The BBC's News service is currently found at www.bbc.co.uk/news, but up until a few years ago it used to be served from its own dedicated hostname, news.bbc.co.uk. This legacy hostname is still used by some webpages today, but mostly redirects visitors to the new site at www.bbc.co.uk/news. This conveniently collates all of the BBC's main online services under the same hostname, but at the expense of introducing a single point of failure. If each service were still to be found under a different hostname and on different servers, it might have offered further resilience to the initial attack.

The performance chart for news.bbc.co.uk shows massive outages long after the DDoS attack on New Year's Eve.

The performance chart for news.bbc.co.uk shows massive outages long after the DDoS attack on New Year's Eve.

As shown above, news.bbc.co.uk was also affected by the DDoS attack which took down the main BBC website, but eventually came back online later that day without having to relocate the website. However, the following morning (New Year's Day), it started to experience significant connectivity problems.

Most requests to news.bbc.co.uk are still failing.

Most requests to news.bbc.co.uk are still failing. Some browsers, such as Chrome, may automatically retry the request.

It is unclear whether this indicates a separate ongoing attack, or an attempt at mitigating such attacks, but nonetheless, it is likely to affect lots of users: Many old news articles are still served directly from news.bbc.co.uk, and some users habitually reach the news website by typing news.bbc.co.uk into their browsers. Some regularly updated pages also continue to be served from news.bbc.co.uk, such as horse racing results.

Most Reliable Hosting Company Sites in December 2015

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 EveryCity SmartOS 0:00:00 0.000 0.093 0.064 0.128 0.128
2 Lightcrest unknown 0:00:00 0.004 0.276 0.006 0.023 0.027
3 One.com Linux 0:00:00 0.004 0.203 0.037 0.105 0.105
4 Memset Linux 0:00:00 0.004 0.153 0.064 0.157 0.245
5 ServerStack Linux 0:00:00 0.004 0.125 0.067 0.135 0.135
6 Netcetera Linux 0:00:00 0.004 0.075 0.084 0.171 0.171
7 Codero Citrix Netscaler 0:00:00 0.004 0.177 0.092 0.189 0.381
8 GoDaddy.com Inc Linux 0:00:00 0.008 0.264 0.007 0.018 0.018
9 Datapipe Linux 0:00:00 0.008 0.145 0.012 0.024 0.031
10 Swishmail FreeBSD 0:00:00 0.008 0.144 0.062 0.125 0.168

See full table

EveryCity had the most reliable hosting company site in December 2015. Despite moving into new offices, its website was the only one to respond to all of Netcraft's requests. EveryCity has maintained its 100% uptime record throughout 2015, and has made it into the top ten 11 times during the year. It also had the most reliable hosting company site in May.

In second place in December was Lightcrest, which also appeared in the top ten in November. It experienced only one failed request, with an impressively fast average connection time of 6 milliseconds. Lightcrest operates its cloud services using its own Kahu Compute Fabric infrastructure, without outsourcing any components to third-party cloud providers.

In third place – also with a single failed request, but with a slower average connection time – was One.com. Established in 2002, One.com now has over 270 employees with companies registered in Denmark, India and Dubai.

Six of December's top ten hosting company sites ran on Linux operating systems, while Swishmail used FreeBSD, Codero used a Citrix Netscaler device, and EveryCity used SmartOS. The latter is a community fork of OpenSolaris, featuring the ZFS file system, DTrace dynamic tracing, kernel-based virtual machines and Solaris Zones operating system-level virtualisation.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

December 2015 Web Server Survey

In the December 2015 survey we received responses from 901,002,770 sites and 5,579,077 web-facing computers, reflecting a loss of 2.0 million sites, but a gain of 39,900 computers.

Apache suffered the largest loss of 13.4 million sites, followed by Microsoft, which lost 5.0 million. A good part of this month's overall losses were caused by expired .xyz domains, which resulted in nearly 9 million .xyz websites disappearing from the internet. Despite the widespread losses caused by the demise of these websites, nginx managed to gain 7.1 million sites overall, which was the largest growth seen by any web server vendor.

The .xyz top-level domain was made available to the general public on 2 June 2014 and immediately received strong support from Network Solutions, which registered nearly 100,000 .xyz domains during the first ten days of operation. Controversially, Network Solutions gave away many .xyz domains for free to customers who already had the corresponding domain under the .com TLD. This was done on an opt-out basis, and the domains were only free for the first year, leaving some customers surprised when each domain became due for renewal at a cost of $38 this year.

Google's parent company, Alphabet Inc, is one of the most notable users of the .xyz TLD with the domain abc.xyz, while some of the other popular .xyz sites include adult sites and torrent search engines. The .xyz TLD has also proven reasonably popular with fraudsters: Netcraft found phishing sites on 150 .xyz domains throughout November 2015.

This month's changes have caused Apache's leading market share to fall by 1.41 points to 35.6%, while nginx's site share has increased to 17.4%. A little over a year ago, Microsoft was in the lead, but has recently been floating around in second place, currently 9.2 percentage points ahead of nginx, and 9.0 behind Apache.

As well as gaining the largest number of sites this month, nginx also showed the largest growth in terms of web-facing computers, growing by 17,000 to reach a total of 765,000. Despite their site losses, Apache and Microsoft also gained a reasonable number of web-facing computers (10,400 and 6,100), while Lighttpd and Google suffered small losses.

A relatively unknown web server, Safedog, was found serving nearly ten times as many websites as last month, making it now the 7th most commonly used web server software with 6.3 million hostnames. However, the number of web-facing computers with Safedog installed is very low – less than 300 – and nearly all of these are running the deprecated Windows Server 2003 operating system. All websites using this Chinese server software claim to be running Safedog 4.0.0, which appears to be a cloud security system.

2015 has been a turbulent year in terms of hostnames, with the total number of sites rising from 877 million in January, to 901 million in December, but dipping as low as 849 million in April. Apache has continued to lead the market throughout the year, with Microsoft following in second place, getting to within 4.1 percentage points of Apache's share in October. In web-facing computers, nginx has shown remarkably consistent growth in its market share, while both Apache and Microsoft have declined. nginx is now installed on 13.71% of all web-facing computers, compared with 11.03% at the start of the year, and its market share within the top million sites has also grown noticeably from 21.09% to 24.29%.

Total number of websites

Web server market share

DeveloperNovember 2015PercentDecember 2015PercentChange
Apache334,095,10237.00%320,676,75935.59%-1.41
Microsoft244,906,58627.12%239,927,01326.63%-0.49
nginx149,967,73316.61%157,001,01817.43%0.82
Google19,622,6242.17%20,362,6782.26%0.09
Continue reading

Most Reliable Hosting Company Sites in November 2015

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 One.com Linux 0:00:00 0.000 0.211g 0.035 0.100 0.100
2 GoDaddy.com Inc Linux 0:00:00 0.009 0.258 0.008 0.019 0.020
3 Datapipe Linux 0:00:00 0.009 0.145 0.012 0.025 0.031
4 XILO Communications Ltd. Linux 0:00:00 0.009 0.213 0.063 0.126 0.126
5 EveryCity SmartOS 0:00:00 0.009 0.094 0.065 0.131 0.131
6 Anexia Linux 0:00:00 0.009 0.191 0.083 0.169 0.169
7 Hivelocity Linux 0:00:00 0.009 0.174 0.087 0.174 0.174
8 Lightcrest unknown 0:00:00 0.013 0.278 0.007 0.019 0.023
9 INetU Linux 0:00:00 0.013 0.146 0.066 0.131 0.131
10 ServerStack Linux 0:00:00 0.013 0.130 0.066 0.132 0.132

See full table

One.com topped the table after successfully responding to all of Netcraft's requests in November. It last appeared in the top 10 in June 2015 when it placed ninth. The Denmark-based company offers shared hosting packages — all of which include unlimited traffic — from data centres operated by Interxion.

After having the most reliable hosting company site for the past three months, GoDaddy narrowly missed the top spot in November. Although five other sites also had two failed requests, GoDaddy's site is placed second based on its faster average connection time of 8ms.

Also with just two failed requests, Datapipe placed third in November. Datapipe has appeared in all but one of the top 10 lists in 2015 and has maintained a 100% uptime record for more than nine years. For the seventh time, Datapipe was recognised as one of New Jersey's 50 fastest growing companies, after a year that included the acquistion of GoGrid and DualSpark.

The top 10 is once again dominated by websites using Linux, which is used by eight of the top 10 sites. EveryCity, the only site that has been in the top 10 for every month of this year so far, uses SmartOS. Microsoft Windows is absent from the list for the fifth consecutive month.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

World Bank hacked by PayPal phishers

Hackers have broken into a website operated by the World Bank Group, which was subsequently exploited to host a convincing PayPal phishing site. The fraudulent content deployed on the site was able to benefit from the presence of a valid Extended Validation SSL certificate.

Extended Validation certificates can only be issued to organisations that have gone through a stringent set of verification steps, as required by the CA/Browser Forum. To recognise the high level of assurance offered by an EV certificate, most browser software will display the organisation's name in a prominent green box next to the address bar.

A PayPal phishing site, using an Extended Validation SSL certificate issued to the World Bank Group.

A PayPal phishing site, using an Extended Validation SSL certificate issued to the World Bank Group.

The EV vetting process effectively guarantees that the domain used in this attack is operated by the organisation specified in the certificate, which in this case is the World Bank Group. Implicatively, any visitor to this site is likely to trust the content it displays.

But of course, this guarantee goes out the window if the site has been compromised by an attacker. That's exactly what happened on Tuesday, when fraudsters deployed a PayPal phishing site into a directory on climatesmartplanning.org, allowing the fraudulent content to be served with an EV certificate issued to The World Bank Group.

The Climate-Smart Planning Platform is an initiative led by The World Bank, which makes it easier for developing-country practitioners to locate and access the tools, data and knowledge they need for climate-smart planning. Given its noble goals, it seems a shame that its website has been affected by this fraudulent activity.

The day after the attack, the website became temporarily unavailable (displaying only a Red Hat Enterprise Linux test page), before later coming back online with the fraudulent content removed. But today, it became evident that the site is still vulnerable to attack, as its homepage has now been defaced by a group called "Virus iraq".

A World Bank Group website hacked by "Virus iraq".

A World Bank Group website hacked by "Virus iraq" (19 November, 2015).

This is not the only time The World Bank's reputation has been tainted by the work of fraudsters – its name is also often used in 419 scams.

Tuesday's phishing attack started off by asking the victim to enter his or her PayPal email address and password. These credentials were submitted to a logcheck.php script on the server, which carried out some validation to prevent bogus data clogging up the phisher's haul.

The phishing site rejects invalid email addresses.

The phishing site rejects invalid email addresses.

After logging these stolen credentials, the phishing site claims it is temporarily unable to load the user's account. The victim is prompted to confirm their "informations" in order to access their account.

The next page asks for several details that would help the fraudster carry out identity theft. These details include the victim's name, date of birth, address and phone number. After these have been submitted, the victim is prompted to confirm payment card details by entering his full card number, expiry date and CSC (CVV) number.

The previous page also has a checkbox to specify whether or not the victim's card uses Verified by Visa or MasterCard SecureCode. If this box is checked, the next page will prompt the user to enter his 3-D Secure password, thus allowing the attacker to make fraudulent purchases on sites that are are protected by these additional layers of security.

Stealing the victim's 3-D Secure password.

Stealing the victim's 3-D Secure password.

After this final password has been stolen, the victim is redirected to the genuine PayPal website, leaving the attacker with the ability to make fraudulent purchases using either the victim's PayPal account or credit card.

At the time of writing, the Climate-Smart Planning Platform website remains defaced, but the phishing content has been removed.

November 2015 Web Server Survey

In the November 2015 survey we received responses from 902,997,800 sites and 5,539,129 web-facing computers. This reflects a monthly gain of 24.7 million sites, and 47,200 computers.

This month's website growth was dominated by Apache, which gained nearly 31 million sites – more than eight times as many as nginx, which had the second largest growth amongst the top three. Helped by a loss of 22 million Microsoft-powered websites, Apache's market share has increased to 37%, with its lead over Microsoft more than doubling to 9.9 percentage points.

This sizeable shift in market shares can be mostly attributed to 17 million websites whose domain names became due for renewal. This caused them to be moved from IIS servers to a set of domain holding pages hosted on Apache servers.

Despite Apache also having the greatest growth in web-facing computers this month, with an increase of 23,405 computers, its market share grew by just 0.03 percentage points. In contrast, nginx's similar growth of 21,004 computers increased its market share by 0.27 percentage points.

The number of web-facing computers using each vendor's software serves as a more stable metric, due in part to the cost of provisioning machines. Conversely, website counts are more prone to large fluctuations, as a single computer can serve countless websites at little incremental cost.

Demonstrating this disconnect, Tengine – an nginx fork developed by Alibaba – made a significant contribution to the overall growth in hostnames despite being used on only 5,100 web-facing computers. While the number of sites using this server grew by nearly 30%, rising to 42 million, the number of active sites using Tengine actually fell by 5%.

nginx continues to increase its presence amongst the top million sites. It now powers an additional 2,708 of the top sites, with Apache, Microsoft and Google each losing out to make room. nginx also showed the largest active sites growth in November, growing by 1.6 million (+6.2%) to reach a total of 27.9 million.

Since the launch of Yunjiasu ("fast cloud") in December 2014, more than 2.5 million sites (and 108,000 active sites) are now being served by a modified version of nginx called yunjiasu-nginx, making it the 10th most commonly used web server software by hostnames. Most of this growth has taken place in the last few months, with the total number of sites using this server growing by more than 5x since August.

Yunjiasu is operated by Chinese search engine giant Baidu, in collaboration with CloudFlare, who are responsible for the similar cloudflare-nginx server that is currently used by more than 5 million sites. Baidu's Yunjiasu offers the same features and functionality as CloudFlare (CDN, DNS, DDoS protection, etc.), but it is optimised for performance and regulatory controls within China.

By combining Baidu's network of 17 mainland China data centers with CloudFlare's 47 data centers outside of China, it is possible to start addressing some of the performance issues that have been dampening the appeal of Chinese hosting companies. For example, the largest hosting company in China, Aliyun, only allows its customers to host websites within China, and although it provides its own CDN service, all of the nodes are also within China. Websites that are hosted in China, and available across the combined CloudFlare/Baidu network, will benefit from much greater availability and faster load times from outside of China. Symmetrically, websites that are hosted outside of China will load faster and become much more available within China.

One of the first customers to be served across Baidu's network was TechCrunch, whose local Chinese edition (techcrunch.cn) was previously only available about 50% of the time within mainland China. CloudFlare claims that it now achieves nearly 100% availability, with an average page load time of 2.5 seconds rather than 17. CloudFlare customers must explicitly opt in to enjoy the performance benefits of the China network: To overcome technical, economic and regulatory issues, Baidu operates all services within China, while CloudFlare operates all of those outside, and by default, no CloudFlare customer traffic will pass through the China network.

Total number of websites

Web server market share

DeveloperOctober 2015PercentNovember 2015PercentChange
Apache303,234,89734.53%334,095,10237.00%2.47
Microsoft267,012,32230.40%244,906,58627.12%-3.28
nginx146,229,30716.65%149,967,73316.61%-0.04
Google19,931,8622.27%19,622,6242.17%-0.10
Continue reading