February 2011 Web Server Survey

In the February 2011 survey we received responses from 284,842,077 sites.

Apache saw the largest increase in terms of both market share and absolute growth this month, with 9.6M new hostnames equating to a 1 percentage point increase in market share. This continues the general upward trend seen for Apache since January last year. The most significant increase occurred in the United States, where 7M new Apache hostnames were recorded. Once again, significant contributions to Apache's increase were seen at AmeriNOC (4.6M) and Softlayer Inc (1.3M). Apache also made a net gain of 817k hostnames in the Netherlands as the result of a 1.3M increase at Axoft Group.

nginx and lighttpd also made gains this month, although lighttpd's market share remained static as a result of the increases detected for the other major web server vendors.

Microsoft and Google both lost hostnames and market share this month. Microsoft's most significant loss came in Germany, with a 239k drop. This came as a result of 237k fewer hostnames being recorded at Kabel Deutschland.

Total Sites Across All Domains
August 1995 - February 2011

Total Sites Across All Domains, August 1995 - February 2011

Market Share for Top Servers Across All Domains
August 1995 - February 2011

Graph of market share for top servers across all domains, August 1995 - February 2011

DeveloperJanuary 2011PercentFebruary 2011PercentChange
Continue reading

Egypt back online, but some sites kept offline

Renesys earlier confirmed that Egyptian internet providers had returned to the internet just before 09:30 UTC; however, a few important sites mysteriously went back offline a short while later. www.mcit.gov.eg came online for a brief period, but then went offline again less than an hour later:

Before Egypt shut down internet access, the online collective Anonymous had been carrying out a distributed denial of service attack against this site; however, that attack did not appear to succeed at the time. Meanwhile, www.egypt.gov.eg has been online solidly since Egypt returned to the internet, whereas www.moiegypt.gov.eg has been coming and going:

This site was also attacked as part of an online protest by Anonymous, which resulted in some short outages on 26th January. A tweet from AnonymousIRC suggests that this site may be being kept offline by a second DDoS attack:

We are continuing to monitor the performance of several Egyptian sites at http://uptime.netcraft.com/perf/reports/performance/wikileaks

is.gd URL shortener suffers downtime

The popular is.gd URL shortening service was reportedly unavailable for a few hours this morning, effectively breaking thousands of shortlinks posted to Twitter and other social networking sites. During the outage, the site's public-facing load balancer responded to PING requests, but was refusing HTTP connections to port 80.

is.gd is one of the most popular URL shortening services in current use – it has shortened 334 million URLs to date, which have been accessed more than 11 billion times. The service is wholly owned by UK hosting company Memset, which hosts the site on their own servers. Since December, Memset has also provided a shorter v.gd service, but this has only attracted 61 thousand URLs so far.

Memset told Netcraft that today's fault was caused by the failure of some virtual machines in the frontend cloud, which is responsible for accepting HTTP requests from the load balancer. These have been restored and the site is now back up and running with improved monitoring processes.

is.gd is primarily maintained by its creator, Richard West, a freelance developer and technologist. Memset proudly describe it as an "ethical" URL shortener; in particular, they have pledged to support is.gd as a free service indefinitely, will never place third-party adverts on the site and claim to be one of the most proactive URL shorteners in preventing spam and misuse.

Other sites hosted by Memset, including its own main presence at memset.com, were unaffected during the is.gd outage.

Most Reliable Hosting Company Sites in January 2011

Rank Company site OS Outage
DNS Connect First
1 Datapipe FreeBSD 0.000 0.061 0.015 0.031 0.042
2 iWeb Technologies Linux 0.000 0.077 0.038 0.077 0.077
3 www.codero.com Linux 0.000 0.129 0.040 0.291 0.484
4 Multacom FreeBSD 0.000 0.156 0.068 0.138 0.402
5 INetU unknown 0:00:00 0.004 0.049 0.021 0.063 0.135
6 Kattare Internet Services Linux 0:00:00 0.008 0.152 0.083 0.170 0.342
7 www.cwcs.co.uk Linux 0:00:00 0.008 0.248 0.115 0.464 0.871
8 New York Internet FreeBSD 0:00:00 0.013 0.061 0.020 0.052 0.130
9 Swishmail FreeBSD 0:00:00 0.013 0.051 0.021 0.043 0.117
10 Rackspace F5 Big-IP 0:00:00 0.013 0.067 0.024 0.049 0.049

See full table

Four of the hosting companies sampled this month experienced no failed requests at all to their websites, and these four are therefore ranked by average connection time. Top of the table is Datapipe, which has been in the top ten in ten of the last twelve months, and in the top three seven times in the same period. The company offers a variety of services including colocation, cloud computing and hosted servers from datacentres in the U.S., the U.K. and China.

Second this month is iWeb Technologies, which has also appeared in the top ten regularly over the last year. iWeb have been offering hosting services for over ten years with a range of products which includes shared web hosting, managed hosting and dedicated servers.

www.codero.com ranked third this month. The company is a relative newcomer to the hosting industry, having begun life as a division of aplus.net in July 2009. It now offers dedicated and managed hosting services aimed specifically at small businesses.

In terms of operating system, the top ten hosting companies are evenly split between Linux and FreeBSD this month with four of each and one company running F5 Big-IP.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

Police.uk fails to cope with public demand

A new local crime and policing website for England and Wales was launched late last night at police.uk. The revamped site provides instant access to street-level crime maps and data – or at least, it did until curious members of the public woke up this morning.

In what could arguably be described as a media-driven DDoS, the new site has received a lot of publicity on the internet, radio and television today. As a result, a huge number of visitors appears to have swamped the police.uk site with traffic, causing it to break. Search results are currently returning error messages, or a blank page with a 503 Service Unavailable response header.

One response worryingly suggests there are no police in London:

The new police.uk site has been developed by advertising agency Rock Kitchen Harris, who also developed the original CrimeMapper site for all 43 English and Welsh police forces in 2009. The launch was announced today on their website, where they said:

"We not only designed, built and manage the site we also arranged the hosting using a mix of servers, with the public website using scaleable cloud hosting."

Despite the use of scaleable cloud hosting (in this case, Amazon EC2), the site does not appear to be holding out too well. Amazon's EC2 hosting service does provide a facility called Auto Scaling, which deals with traffic spikes by automatically increasing capacity, but it is not clear whether RKH have enabled this feature. WikiLeaks notably used Amazon EC2 when the Iraq War Logs and Cablegate sites went live, both of which coped well with the initial large volume of traffic.

Netcraft was unable to speak to anyone in the web team at RKH, as they are, understandably, "a bit tied up at the moment", but it was confirmed that the current problem is a result of too much traffic.

Anonymous attacks websites in Egypt

Following the recent uprising in Tunisia, thousands of demonstrators took to the streets in Egypt yesterday to demand an end to President Hosni Mubarak's rule. The online collective known as Anonymous has joined in the protests by orchestrating distributed denial of service attacks against key Egyptian websites.

Operation: Egypt began its recruitment campaign 3 days ago, inviting participants to join the #OpEgypt channel on its IRC network. As with the previous attacks against PayPal, MasterCard and Visa, volunteers are being sought to install and run the Low Orbit Ion Cannon (LOIC) software, which can automatically bombard a website with a large volume of traffic.

Last night, Twitter confirmed that it had been blocked in Egypt:

Some of the earlier DDoS attacks carried out by Anonymous had used Twitter feeds to announce targets to the automated attack software. IRC appears to be the primary control point now, with the current target set to www.mcit.gov.eg – The Egyptian Ministry of Communications and Information Technology.

It is not clear how many people are involved in this attack, but our performance data for www.mcit.gov.eg shows the website is currently up and responding to HTTP requests from around the world.

This could suggest that the number of volunteers taking part in the attacks has continued to decrease over the past few months. In an interview earlier this month, Anonymous member Sven Slootweg said that he did not expect many more DDoS attacks as the impact is limited.

To muster up more attackers, the IRC channel also invites users to use a web-based version of the LOIC attack tool, which can even be used from mobile phones. Further discussions, including which targets to attack, are being carried out on the interactive multi-user PiratePad site:

This also lists www.moiegypt.gov.eg as a main target, which has experienced some short outages (performance graphs).