August 2015 Web Server Survey

In the August 2015 survey we received responses from 874,408,576 sites and 5,391,301 web-facing computers, representing a net gain of 25 million sites and 40,978 web-facing computers since last month.

Microsoft was responsible for much of the growth in web-facing computers this month, reversing the losses seen last month. This month there was an increase of 15,668 web-facing computers powered by Microsoft web server software, accompanied by a gain of 6.1 million sites. Microsoft has recovered some web-facing computer market share as a result of the increase; however, it remains on a gradual declining trend – it now stands at almost 2 percentage points below its share this time last year.

nginx performed well across all metrics again this month, gaining 3,421 sites in the top million sites, 6,491 web-facing computers, and 983,000 sites overall. nginx is the only vendor experiencing consistent increases in market share, and is now used by 22.61% of the top million sites, and 12.68% of web-facing computers.

Apache also made gains this month, with 1,243 additional web-facing computers and 2.3 million additional sites. However, it lost 4,775 sites in the top million sites, where its market share is now 47.78%. Despite the net gain in web-facing computers, Apache has again seen a small loss in its market share, which now stands at 46.26%.

LiteSpeed gained 486,000 sites this month, bringing the total number of sites using LiteSpeed's web server to just over 5 million. LiteSpeed uses the same configuration format as Apache and is designed to be a drop-in replacement.

LiteSpeed was the first major web server vendor to add support for the final version of HTTP/2 after it was standardised in May. HTTP/2, which is based on Google's SPDY protocol, aims to improve the performance of HTTP by changing how it is encoded on the wire. It does not change HTTP's semantics to ease compatibility with existing applications. While the standard defines a cleartext version of the protocol, all major browsers only support HTTP/2 over TLS. Out of the 45,819 SSL sites that negotiated the final version of HTTP/2 over TLS this month, 21,695 (47.35%) were served by LiteSpeed.

An initial patch was released by nginx this month for adding HTTP/2 support. The patch is still in development – full HTTP/2 support in nginx is expected by the end of 2015.

Microsoft IIS 10 is the first release of IIS that provides HTTP/2 support. IIS 10 is included in Windows 10, which was released in July, and Windows Server 2016, which is currently in public beta testing and expected to be released in early 2016.

mod_h2, an Apache module which provides HTTP/2 support, was donated to the Apache Foundation in June and merged into the development version of Apache. mod_h2 will be backported to Apache 2.4, the current stable release branch.

Total number of websites

Web server market share

DeveloperJuly 2015PercentAugust 2015PercentChange
Apache325,696,51438.34%327,985,96837.51%-0.83
Microsoft225,282,71326.52%231,429,14626.47%-0.05
nginx131,460,06315.47%132,443,39115.15%-0.33
Google20,255,4242.38%19,933,0952.28%-0.10
Continue reading

Millions still running the risk with Windows Server 2003

More than 600,000 web-facing computers — which host millions of websites — are still running Windows Server 2003, despite it no longer being supported.

The number of web-facing computers running Windows Server 2003 has been on a gradual decline since its peak usage in 2011, but many servers are still using it. Mainstream support for Windows Server 2003 ended in July 2010.

The number of web-facing computers running Windows Server 2003 has been on a gradual decline since its peak usage in 2011, but many servers are still using it. Mainstream support for Windows Server 2003 ended in July 2010.

Extended support for Windows Server 2003 ended on July 14, 2015. Crucially, this means that Microsoft will no longer be issuing security updates for any version of Windows Server 2003. US-CERT warns that these unsupported installations of Windows Server 2003 are exposed to an elevated risk of cybersecurity dangers, such as malicious attacks or electronic data loss.

Windows Server 2003 was originally launched over 12 years ago, with the latest major update being released 8 years ago in the form of Service Pack 2. This update was particularly beneficial for web servers, as it added the Scalable Networking Pack (SNP), which allowed for hardware acceleration of network packet processing.

Fifth of the internet still running Windows Server 2003

Netcraft's July 2015 Web Server Survey found 175 million websites that are served directly from Windows Server 2003 computers. These account for more than a fifth of all websites in the survey, making the potential attack surface huge.

Most of these sites (73%) are served by Microsoft Internet Information Services 6.0, which is the version of IIS that shipped with Windows Server 2003 and the 64-bit edition of Windows XP Professional; however, it is rare to see the latter being used as a web server platform.

The remaining Windows Server 2003-powered sites use a variety of web server software, with GSHD 3.0, Safedog 4.0.0, Apache 2.2.8 (Win32), kangle 3.4.8, NetBox Version 2.8 Build 4128 and nginx/1.0.13-win32 being amongst the most commonly seen Server headers. While vulnerabilities in these software products can be addressed by applying patches or updates, future vulnerabilities in the underlying Windows Server 2003 operating system may never be fixed.

14 million of the sites did not send a Server header at all, so it was not apparent whether the web server software used by these sites could be updated, but the underlying computers could still be identified as running Windows Server 2003. Netcraft determines the operating system of a remote web server by analysing the low-level TCP/IP characteristics of response packets, and so it is independent of whichever server software the site claims to be running.

Backend servers might also be exploitable

In addition to the 175 million websites that are served directly from Windows Server 2003 computers, a further 1.7 million sites served from other operating systems sent the Microsoft-IIS/6.0 Server header. This indicates the presence of backend Windows Server 2003 machines behind load balances and similar devices that are not running Windows.

For example, if the TCP/IP characteristics of a web server's response indicate that it is running Linux, but the HTTP Server header reports it is using Microsoft-IIS/6.0, then the Linux machine is likely to be acting as a reverse proxy to a Windows Server 2003 machine running IIS 6.0. Although the Windows Server 2003 machine is not directly exposed to the internet, it may still be possible for a remote attacker to exploit certain Windows and IIS vulnerabilities.

How many Windows Server 2003 installations are exposed to the web?

Netcraft has developed a technique for identifying the number of unique computers that act as web servers on the internet. The 175 million sites that use Windows Server 2003 make use of 1.6 million distinct IP addresses. However, an individual computer running Windows Server 2003 may have multiple IP addresses, which makes this an unsuitable metric for determining how many installations there are.

Further analysis of the low-level TCP/IP characteristics reveals a total of 609,000 web-facing computers running Windows Server 2003. This is over 10% of all web-facing computers, and shows the true potential cost of migration, as software licensing is typically charged on a per-machine rather than per-IP address basis.

Who's still using Windows Server 2003?

China and the United States account for 55% of the world's Windows Server 2003 computers (169,000 in China and 166,000 in the US), yet only 43% of all other web facing computers.

Within China, more than 24,000 of these computers are hosted by Alibaba Group. Nearly half of these are hosted by HiChina, which was acquired by Alibaba in 2009, while 7,500 are hosted at its rapidly growing cloud hosting unit, Aliyun.

Aliyun still allows its customers to create Windows Server 2003 virtual machines.

Aliyun still allows its customers to create Windows Server 2003 virtual machines.

One of the most prominent companies still using Windows Server 2003 on the internet is LivePerson, which is best known for the live chat software that allows its customers to talk to their visitors in realtime. Its main site at www.liveperson.com uses Microsoft IIS 6.0 on Windows Server 2003, and several other sites related to its live chat functionality — such as sales.liveperson.net — also appear to use IIS 6.0 on Server 2003, but are served via F5 BIG IP web-facing devices.

Even some banks are still using Windows Server 2003 and IIS 6.0 on their main sites, with the most popular ones including Natwest, ANZ, and Grupo Bancolombia. These sites rank amongst the top 10,000 in the world, and hundreds of other banking sites also appear to be using Windows Server 2003.

ING Direct and Caisse d'Epargne are also using IIS 6.0, but these sites appear to be served through F5 BIG-IP or similar devices, rather than having Windows Server 2003 machines exposed directly to the internet. Even some security and antivirus software vendors are still running IIS 6.0 on public-facing sites, including Panda Security and eScan.

While Microsoft does not officially offer any support beyond the extended support period ("Once a product transitions out of support, no further support will be provided for the product"), reports suggest that some companies who have not migrated in time have arranged to pay millions of dollars for custom support deals.

PCI compliance: Automatic failure

Companies still using unsupported operating systems like Windows Server 2003 in a cardholder data environment should migrate immediately. All organisations and merchants who accept, transmit or store cardholder data must maintain a secure PCI compliant environment.

The Payment Card Industry Data Security Standard (PCI DSS) provides a baseline of technical and operational requirements designed to protect cardholder data and sensitive authentication data. PCI DSS Requirement 6.2 requires all system components and software to be protected from known vulnerabilities by installing vendor-supplied security patches. This will not be possible with Windows Server 2003, as no more security updates will be made available by Microsoft.

Additionally, merchants and service providers who handle a large enough volume of cardholder data must have quarterly security scans by a PCI SSC Approved Scanning Vendor (such as Netcraft) in order to maintain compliance. ASVs are required to record an automatic failure if the merchant's cardholder data environment uses an operating system that is no longer supported.

In some cases, the PCI SSC can allow for risks to be mitigated through the implementation of suitable compensating controls, but these are unlikely to be sufficient for an unsupported web-facing operating system – especially one which will become less secure as time goes by, as new vulnerabilities are discovered.

Consequently, many merchants still using Windows Server 2003 is likely to be noncompliant, and could face fines, increased transaction fees, reputational damage, or other potentially disastrous penalties such as cancelled accounts.

Microsoft advises that any datacenter still using Windows Server 2003 needs to protect its infrastructure by planning and executing a migration strategy. Some possible options suggested by Microsoft include switching to Windows Server 2012 R2, Microsoft Azure or Office 365. To help customers migrate, Microsoft has provided an interactive Windows Server 2003 Migration Planning Assistant, which, incidentally, is hosted on Microsoft Azure.

Finding out more

Netcraft's techniques provide an independent view with a consistent methodology on the number of web-facing computers at each hosting location worldwide. For more information, see our Hosting Provider Server Count, or contact us at sales@netcraft.com for bespoke datasets.

For more information about Netcraft's Automated Vulnerability Scanning for PCI Compliance, please contact us at security-sales@netcraft.com.

Most Reliable Hosting Company Sites in July 2015

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Datapipe Linux 0:00:00 0.004 0.106 0.012 0.026 0.037
2 GoDaddy.com Inc Linux 0:00:00 0.009 0.131 0.009 0.023 0.024
3 Qube Managed Services Linux 0:00:00 0.009 0.109 0.047 0.094 0.094
4 EveryCity SmartOS 0:00:00 0.009 0.067 0.066 0.131 0.131
5 XILO Communications Ltd. Linux 0:00:00 0.013 0.182 0.063 0.128 0.128
6 Anexia Linux 0:00:00 0.013 0.404 0.086 0.173 0.173
7 Bigstep Linux 0:00:00 0.018 0.111 0.062 0.124 0.124
8 LeaseWeb Linux 0:00:00 0.022 0.224 0.025 0.053 0.053
9 ServerStack Linux 0:00:00 0.027 0.080 0.072 0.143 0.143
10 Swishmail FreeBSD 0:00:00 0.031 0.123 0.066 0.132 0.172

See full table

Datapipe had the most reliable company website during July, responding successfully to all but one of Netcraft's requests. This continues their regular appearance in the top 10, making 11 of the past 12 months and every month of 2015 so far. Datapipe offers a 100% Uptime Guarantee, and fulfilled this promise on its own site, with 100% uptime recorded over the past 9 years.

With two failed requests each, GoDaddy, Qube, and EveryCity contend for second place. The ranking between them is decided by the average connection time, putting GoDaddy into second place and Qube into third. GoDaddy recently produced CODE, a documentary about gender bias in the tech industry; the film was selected for the Tribeca Film Festival. Qube is based in London and offers managed private cloud hosting services from datacentres in London, New York and Zurich.

Linux is once again the most common choice of Operating System; 8 out of the top 10 companies used the OS to power their website. The two remaining sites were powered by SmartOS and FreeBSD. This is the first time since December 2014 that no sites powered by Windows have appeared in the top 10.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

July 2015 Web Server Survey

In the July 2015 survey we received responses from 849,602,745 sites and 5,350,323 web-facing computers. This represents a net loss of 13.5 million websites, but a gain of 3,700 additional computers.

One of the most significant changes in July was the net loss of nearly 13,000 web-facing computers powered by Microsoft web server software, accompanied by a decline of more than 29 million hostnames. The loss was predominantly seen for servers running Microsoft IIS 6.0, 7.0 and 7.5. These versions of IIS are used by Windows Server 2003, which is no longer supported, and Windows Server 2008 (including 2008 R2), whose mainstream support ended in January. The latest stable release of IIS (version 8.5) is however continuing to grow, this month increasing by over 9,000 web facing computers.

This month's decline has brought Microsoft's market share of hostnames down by nearly 3 percentage points, increasing Apache's lead. However, Apache's own market share also fell slightly, largely due to gains made by nginx and Tengine.

nginx gained 8.5 million sites this month, but more remarkably, it gained over 14,000 web-facing computers, with the largest gains in the US, China, Germany and the UK. Compounding Microsoft's losses, nearly 1.8 million existing websites switched from using Microsoft IIS to nginx in July.

nginx also fared well amongst the top million websites, where it gained a further 3,771 sites, causing losses for Apache, Microsoft and Google. Nonetheless, Apache is still used by nearly half of the top million sites, with its market share being almost 26 percentage points ahead of nginx.

Tengine now powers more websites than Google's web server software, after the number of sites using it grew by 7 million to a total of more than 25 million this month. The open source Tengine web server is based on nginx, and used extensively by the online marketplace Taobao. It currently supports all features found in nginx 1.6.2, plus several other features required by Taobao that were not able to be implemented as nginx modules. Neither nginx nor Tengine support HTTP/2 yet, but they were both early supporters of Google's SPDY protocol, on which HTTP/2 is based. nginx plans to provide support for HTTP/2 by the end of this year, and so it is likely that Tengine may also follow suit at a later date.

Tengine 2.1.0 is the latest development version of Taobao's nginx fork, but despite being released more than six months ago, only 25,000 websites currently claim to be using it. In contrast, Tengine 1.4.2 — which was released in 2012 and is also a development version — is used by nearly 10 million sites, making it by far the most commonly deployed version. The latest stable release, Tengine 1.5.2, is the second most commonly used version, but accounts for just under 200,000 sites.

But like Apache, more than half of the sites running Tengine do not reveal which version they are running, and so the true distribution of version numbers could vary greatly. For instance, 2.7 million of these version-less Tengine websites are used to host Taobao stores directly under the taobao.com domain (e.g. baobeiit.taobao.com). Given that Tengine was created by Taobao in order to provide the features they need, it is not unreasonable to assume that these sites might be using the latest release, or at least a relatively recent one.

Despite being used by a large number of sites, Tengine was found on only 4,240 web-facing computers in July 2015. Three-quarters of these computers are located in China, while nearly 10% are located in the US.

Total number of websites

Web server market share

DeveloperJune 2015PercentJuly 2015PercentChange
Apache334,731,03538.78%325,696,51438.34%-0.45
Microsoft254,408,17929.48%225,282,71326.52%-2.96
nginx122,965,52214.25%131,460,06315.47%1.23
Google20,130,7322.33%20,255,4242.38%0.05
Continue reading

Most Reliable Hosting Company Sites in June 2015

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Qube Managed Services Linux 0:00:00 0.000 0.102 0.052 0.104 0.104
2 ServerStack Linux 0:00:00 0.000 0.074 0.076 0.152 0.152
3 Bigstep Linux 0:00:00 0.005 0.117 0.060 0.123 0.123
4 iWeb Linux 0:00:00 0.005 0.135 0.078 0.156 0.156
5 Anexia Linux 0:00:00 0.005 0.541 0.085 0.172 0.172
6 GoDaddy.com Inc Linux 0:00:00 0.010 0.121 0.009 0.022 0.023
7 Datapipe Linux 0:00:00 0.015 0.095 0.013 0.027 0.037
8 Netcetera Windows Server 2012 0:00:00 0.015 0.055 0.083 0.165 0.165
9 LeaseWeb Linux 0:00:00 0.025 0.224 0.028 0.061 0.061
10 One.com Linux 0:00:00 0.025 0.166 0.058 0.217 0.218

See full table

Qube Managed Services had the most reliable website during June, responding successfully to all of Netcraft's requests. This is Qube's fourth appearance in the top ten in 2015, continuing its strong showing from 2014 when it placed in the top ten in eleven months, and came first on four occasions. Qube is based in London and offers managed private cloud hosting services from datacentres in London, New York and Zurich.

In second place, ServerStack also successfully responded to all requests in June, placing second only as a result of a slightly slower average connection time. ServerStack provides managed hosting services to enterprises from three datacentres in Amsterdam, New Jersey and San Jose. It has appeared in the top 10 list frequently in the past few years.

Bigstep, iWeb and Anexia also did well this month, each responding to all but one request.

Linux remains the most popular choice of operating system, with 9 of the top 10 companies using the OS to power their website, while the remaining one uses Windows Server 2012.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

June 2015 Web Server Survey

In the June 2015 survey we received responses from 863,105,652 sites and 5,346,650 web-facing computers, representing an increase of 5.2 million websites and 65,000 additional computers.

Microsoft was responsible for the majority of this month's hostname growth, with a gain of 6.6 million sites, but only contributed 11,700 additional web-facing computers. This has caused Microsoft's market share by hostnames to overtake its declining market share by computers, with both standing at just under 30%.

Apache led this month's web-facing computer growth with a net gain of 24,800, while nginx followed closely with 22,800. This has resulted in nginx's market share increasing by 0.28 to 12.4%, and despite showing the largest net growth, Apache's share fell slightly.

Apache, Microsoft and nginx together account for more than 88% of all web-facing computers in the world, making these vendors by far the most popular choices. However, nginx is the only vendor experiencing consistent increases in market share, up by 3 percentage points over the last year while both Apache and Microsoft have seen losses. The next most commonly used server is lighttpd (pronounced "lighty"), which is used by a mere 0.46% of web-facing computers.

nginx's market share has also been steadily increasing within the top million websites. Its share now stands at 21.9%, and although Apache's use within the million busiest sites has been steadily declining this decade, Apache looks likely to retain the lead for at least a few more years.

Three months after the death of Sir Terry Pratchett, approximately 84,000 websites are now serving the X-Clacks-Overhead: GNU Terry Pratchett header in tribute. Invisible to the majority of users, this HTTP header is a reference to the Discworld novel Going Postal, which features a series of communication towers called the clacks.

In the book, a similar header ("GNU John Dearheart"), is transmitted around the clacks after the inventor's son is killed in an accident while working on a clacks tower. The G means send the message on, N means do not log the message, and U means turn the message around at the end of the line and send it back again — this ensures that the message is transmitted indefinitely, allowing his son to be memorialised forever. Similarly, by transmitting Pratchett's name around the internet, the sites participating in this HTTP header tribute hope to keep his legacy alive. After all, as it says in the book, "A man is not dead while his name is still spoken."

One of the most popular sites to use the X-Clacks-Overhead: GNU Terry Pratchett header is www.theguardian.com, which alone reached more than 5 million unique browsers per day in 2014. With each header taking up 40 bytes of an uncompressed HTTP request, all of the sites involved in the tribute could be generating terrabytes of additional bandwidth usage every day.

Total number of websites

Web server market share

DeveloperMay 2015PercentJune 2015PercentChange
Apache336,813,95939.26%334,731,03538.78%-0.48
Microsoft247,784,66828.88%254,408,17929.48%0.59
nginx123,697,64514.42%122,965,52214.25%-0.17
Google20,103,0682.34%20,130,7322.33%-0.01
Continue reading