January 2016 Web Server Survey

In the January 2016 survey we received responses from 906,616,188 sites and 5,753,264 web-facing computers, reflecting a modest increase of less than six million sites, but a significant gain of 174,000 computers.

Microsoft gained 22.5m sites (+9.40%), which has taken its market share up by 2.32 points. Meanwhile, Apache lost 16.4m sites, and nginx fell by 15.6m. Apache's market share is now less than 5 points ahead of Microsoft; this difference was more than twice as large just two months ago.

The web-facing computers metric is typically much more stable, but this month's overall gain of 174,000 computers is unusually large as a result of a 7.6% increase in the number of web-facing computers running Apache.

This large gain comprised of nearly 195,000 Apache computers, and the majority of these are Western Digital My Cloud personal storage devices. These consumer devices run web servers and can be accessed using public hostnames with a format similar to device1000000-a1b2c3d4.wd2go.com. Consumers can remotely access their files via the My Cloud web application, a mobile app, or via third-party applications that make use of the relatively new My Cloud OS 3 platform.

Consumers can remotely access their files via the My Cloud web application (shown), or via a mobile app.

Consumers can remotely access their files via the My Cloud web application (shown), a mobile app, or third-party tools.

More than 240,000 of these wd2go.com hostnames point directly to a variety of consumer broadband connections, which is where the My Cloud devices are physically located.

Network Attached Storage (NAS) devices are rarely exposed to the internet on such a large scale, and so this provides some otherwise invisible insights into the usage of these particular devices. Although consumers do not have to enable the Cloud Access feature, the 240,000+ devices that are directly exposed to the internet are likely to be a fairly representative sample of all similar Western Digital devices.

Nearly half of the My Cloud devices that are exposed directly to the internet are located in the US, while the UK has the next largest share of 13%, and France follows with 6%. This suggests that nearly two-thirds of Western Digital's consumer NAS sales take place in these three countries alone.

As well as the My Cloud devices that are exposed directly to the internet, a further 273,000 wd2go.com hostnames resolve to fewer than 200 IP addresses hosted by Amazon AWS. These hostnames likely represent additional My Cloud devices that have been cloud-enabled using Relay mode. In this mode, requests bound for the device are relayed via the Amazon-hosted web service, which makes it possible for a consumer to gain remote access even when they are not able to set up port forwarding on their router.

However, whilst certainly convenient, exposing a My Cloud device to the internet (either directly or in relay mode) could undermine a consumer's security by revealing the device's internal IP address to the whole world. Each of the 500,000+ My Cloud devices that can be accessed via hostnames like device1070698-xxxxxxxx.wd2go.com also have corresponding DNS entries that reveal their local IP addresses:

$ host device1070698-xxxxxxxx.wd2go.com
device1070698-xxxxxxxx.wd2go.com has address 78.72.xx.x
$ host device1070698-xxxxxxx-local.wd2go.com
device1070698-xxxxxxxx-local.wd2go.com has address 192.168.1.65

These "-local" DNS entries allow a remote attacker to discover the local IP address of a consumer's My Cloud device (in this case, 192.168.1.65), which would make it easier to carry out CSRF attacks against it. Even if the consumer has taken the precaution of changing the device's name so that his browser cannot reach it via the default local address (http://wdmycloud), it could still be reached by browsing directly to its local IP address. Devices that have not been updated recently might still be vulnerable to remote code execution via CSRF attacks.

The local IP address of the My Cloud device can also be used to infer the address of the consumer's broadband router, which may well be vulnerable to similar types of attack. Knowing some likely IP addresses of the router makes CSRF attacks much more feasible – for example, if the My Cloud device has an IP address of 10.10.0.31, the attacker could deduce that the router's IP address might be 10.10.0.1 or 10.10.0.255, rather than any of the other 17+ million IANA-reserved private network addresses. A successful exploit against a vulnerable router could give an attacker full control over the router's settings, which could ultimately lead to data theft or financial losses through pharming attacks.

While the influx of these My Cloud devices has resulted in strong growth for Apache, nginx continued its steady progress by gaining a further 23,300 (+3.0%) web-facing computers. Apache's market share in terms of computers now stands at 47.9% (+2.0), while Microsoft lost 20,600 computers, contributing to its share falling to 27.1%. Despite maintaining the consistent growth it has demonstrated for several years, nginx also suffered a minor loss in share by virtue of Apache's exceptional growth.

Total number of websites

Web server market share

DeveloperDecember 2015PercentJanuary 2016PercentChange
Apache320,676,75935.59%304,271,06133.56%-2.03
Microsoft239,927,01326.63%262,471,88628.95%2.32
nginx157,001,01817.43%141,443,63015.60%-1.82
Google20,362,6782.26%20,799,0872.29%0.03
Continue reading

December 2015 Web Server Survey

In the December 2015 survey we received responses from 901,002,770 sites and 5,579,077 web-facing computers, reflecting a loss of 2.0 million sites, but a gain of 39,900 computers.

Apache suffered the largest loss of 13.4 million sites, followed by Microsoft, which lost 5.0 million. A good part of this month's overall losses were caused by expired .xyz domains, which resulted in nearly 9 million .xyz websites disappearing from the internet. Despite the widespread losses caused by the demise of these websites, nginx managed to gain 7.1 million sites overall, which was the largest growth seen by any web server vendor.

The .xyz top-level domain was made available to the general public on 2 June 2014 and immediately received strong support from Network Solutions, which registered nearly 100,000 .xyz domains during the first ten days of operation. Controversially, Network Solutions gave away many .xyz domains for free to customers who already had the corresponding domain under the .com TLD. This was done on an opt-out basis, and the domains were only free for the first year, leaving some customers surprised when each domain became due for renewal at a cost of $38 this year.

Google's parent company, Alphabet Inc, is one of the most notable users of the .xyz TLD with the domain abc.xyz, while some of the other popular .xyz sites include adult sites and torrent search engines. The .xyz TLD has also proven reasonably popular with fraudsters: Netcraft found phishing sites on 150 .xyz domains throughout November 2015.

This month's changes have caused Apache's leading market share to fall by 1.41 points to 35.6%, while nginx's site share has increased to 17.4%. A little over a year ago, Microsoft was in the lead, but has recently been floating around in second place, currently 9.2 percentage points ahead of nginx, and 9.0 behind Apache.

As well as gaining the largest number of sites this month, nginx also showed the largest growth in terms of web-facing computers, growing by 17,000 to reach a total of 765,000. Despite their site losses, Apache and Microsoft also gained a reasonable number of web-facing computers (10,400 and 6,100), while Lighttpd and Google suffered small losses.

A relatively unknown web server, Safedog, was found serving nearly ten times as many websites as last month, making it now the 7th most commonly used web server software with 6.3 million hostnames. However, the number of web-facing computers with Safedog installed is very low – less than 300 – and nearly all of these are running the deprecated Windows Server 2003 operating system. All websites using this Chinese server software claim to be running Safedog 4.0.0, which appears to be a cloud security system.

2015 has been a turbulent year in terms of hostnames, with the total number of sites rising from 877 million in January, to 901 million in December, but dipping as low as 849 million in April. Apache has continued to lead the market throughout the year, with Microsoft following in second place, getting to within 4.1 percentage points of Apache's share in October. In web-facing computers, nginx has shown remarkably consistent growth in its market share, while both Apache and Microsoft have declined. nginx is now installed on 13.71% of all web-facing computers, compared with 11.03% at the start of the year, and its market share within the top million sites has also grown noticeably from 21.09% to 24.29%.

Total number of websites

Web server market share

DeveloperNovember 2015PercentDecember 2015PercentChange
Apache334,095,10237.00%320,676,75935.59%-1.41
Microsoft244,906,58627.12%239,927,01326.63%-0.49
nginx149,967,73316.61%157,001,01817.43%0.82
Google19,622,6242.17%20,362,6782.26%0.09
Continue reading

November 2015 Web Server Survey

In the November 2015 survey we received responses from 902,997,800 sites and 5,539,129 web-facing computers. This reflects a monthly gain of 24.7 million sites, and 47,200 computers.

This month's website growth was dominated by Apache, which gained nearly 31 million sites – more than eight times as many as nginx, which had the second largest growth amongst the top three. Helped by a loss of 22 million Microsoft-powered websites, Apache's market share has increased to 37%, with its lead over Microsoft more than doubling to 9.9 percentage points.

This sizeable shift in market shares can be mostly attributed to 17 million websites whose domain names became due for renewal. This caused them to be moved from IIS servers to a set of domain holding pages hosted on Apache servers.

Despite Apache also having the greatest growth in web-facing computers this month, with an increase of 23,405 computers, its market share grew by just 0.03 percentage points. In contrast, nginx's similar growth of 21,004 computers increased its market share by 0.27 percentage points.

The number of web-facing computers using each vendor's software serves as a more stable metric, due in part to the cost of provisioning machines. Conversely, website counts are more prone to large fluctuations, as a single computer can serve countless websites at little incremental cost.

Demonstrating this disconnect, Tengine – an nginx fork developed by Alibaba – made a significant contribution to the overall growth in hostnames despite being used on only 5,100 web-facing computers. While the number of sites using this server grew by nearly 30%, rising to 42 million, the number of active sites using Tengine actually fell by 5%.

nginx continues to increase its presence amongst the top million sites. It now powers an additional 2,708 of the top sites, with Apache, Microsoft and Google each losing out to make room. nginx also showed the largest active sites growth in November, growing by 1.6 million (+6.2%) to reach a total of 27.9 million.

Since the launch of Yunjiasu ("fast cloud") in December 2014, more than 2.5 million sites (and 108,000 active sites) are now being served by a modified version of nginx called yunjiasu-nginx, making it the 10th most commonly used web server software by hostnames. Most of this growth has taken place in the last few months, with the total number of sites using this server growing by more than 5x since August.

Yunjiasu is operated by Chinese search engine giant Baidu, in collaboration with CloudFlare, who are responsible for the similar cloudflare-nginx server that is currently used by more than 5 million sites. Baidu's Yunjiasu offers the same features and functionality as CloudFlare (CDN, DNS, DDoS protection, etc.), but it is optimised for performance and regulatory controls within China.

By combining Baidu's network of 17 mainland China data centers with CloudFlare's 47 data centers outside of China, it is possible to start addressing some of the performance issues that have been dampening the appeal of Chinese hosting companies. For example, the largest hosting company in China, Aliyun, only allows its customers to host websites within China, and although it provides its own CDN service, all of the nodes are also within China. Websites that are hosted in China, and available across the combined CloudFlare/Baidu network, will benefit from much greater availability and faster load times from outside of China. Symmetrically, websites that are hosted outside of China will load faster and become much more available within China.

One of the first customers to be served across Baidu's network was TechCrunch, whose local Chinese edition (techcrunch.cn) was previously only available about 50% of the time within mainland China. CloudFlare claims that it now achieves nearly 100% availability, with an average page load time of 2.5 seconds rather than 17. CloudFlare customers must explicitly opt in to enjoy the performance benefits of the China network: To overcome technical, economic and regulatory issues, Baidu operates all services within China, while CloudFlare operates all of those outside, and by default, no CloudFlare customer traffic will pass through the China network.

Total number of websites

Web server market share

DeveloperOctober 2015PercentNovember 2015PercentChange
Apache303,234,89734.53%334,095,10237.00%2.47
Microsoft267,012,32230.40%244,906,58627.12%-3.28
nginx146,229,30716.65%149,967,73316.61%-0.04
Google19,931,8622.27%19,622,6242.17%-0.10
Continue reading

October 2015 Web Server Survey

In the October 2015 survey we received responses from 878,269,546 sites and 5,491,917 web-facing computers. This reflects a drop of 14.5 million sites since last month, while the number of computers rose by 53,800.

nginx grew in all metrics this month – websites, active sites, web-facing computers, and its share of the top million sites. With a gain of 866,000 active sites, nginx has increased its market share in this metric beyond 15% for the first time.

nginx also made an impressive gain of 21,480 web-facing computers, outpacing Apache's increase of 12,629 and Microsoft's 4,606. nginx is now used by 727,000 web-facing computers around the world, but it still has a fair way to go before it encroaches on the dominance of Microsoft and Apache. More than twice as many computers are running Microsoft server software, while Apache is even further ahead with its 2.5 million computers giving it a 46% share of the market.

Increasing native support for HTTP/2

The latest mainline version of nginx (1.9.5) has ditched support for SPDY, replacing it with HTTP/2 via an experimental ngx_http_v2_module. The latest major release in the 2.4 stable branch of Apache also now supports HTTP/2 natively. Apache 2.4.17 was released on 13 October 2015, and includes a donated HTTP/2 implementation in the mod_http2 core module, which has similar configuration options to the existing mod_ssl module. HTTP/2 support was previously available since Apache 2.4.12 via the mod_h2 module, although this required the server source code to be patched.

HTTP/2 is the standardised successor of SPDY, on which it was based. The primary motivation for using either of these protocols is performance – compared with HTTP 1.1, both of the newer protocols offer reduced latency through methods like header compression, prioritisation, and allowing webpage elements to be requested in parallel over a single TCP connection.

However, widespread use of HTTP 1.1 is likely to continue for several more years at least, as most browser vendors only support HTTP/2 over encrypted TLS connections. This means the significantly greater number of non-HTTPS sites currently in existence will carry on using HTTP 1.1, even though the HTTP/2 standard is also defined for HTTP URLs.

Despite the potential performance benefits, less than 5% of all SSL certificates in Netcraft's October SSL Survey were found on web servers that supported SPDY or HTTP/2. However, 29% of SSL sites within the thousand most popular sites currently support SPDY or HTTP/2, while 8% of those within the top million sites do. The busiest sites have the most to gain by optimising their connections, so this distribution is not too surprising.

HTTP/2 is also supported by the latest version of Microsoft Internet Information Services, although with the production version of Windows Server 2016 yet to be released, it is not too surprising that IIS 10.0 was found being used by only 2,200 sites in this month's survey. Several of these sites are hosted by Microsoft, and although publicly accessible, the hostnames suggest they are test servers that mirror the functionality of existing Microsoft sites still running IIS 7.0 and IIS 7.5.

While Windows Server 2016 is likely to become the primary platform for IIS 10.0 on the internet, IIS 10.0 is also included in Windows 10, which is already available and has been offered as a free upgrade to many Windows users. Technical Preview versions of Windows Server 2016 are also currently available for evaluation. Some earlier versions of Windows, including Windows 7 Service Pack 1, can also run IIS 10.0 Express. This is a self-contained version that has all of the core capabilities of IIS 10.0, as well as some additional features to make it easier to develop and test websites.

Total number of websites

Web server market share

DeveloperSeptember 2015PercentOctober 2015PercentChange
Apache312,106,63834.96%303,234,89734.53%-0.43
Microsoft265,010,74629.68%267,012,32230.40%0.72
nginx139,297,80415.60%146,229,30716.65%1.05
Google19,683,0872.20%19,931,8622.27%0.06
Continue reading

September 2015 Web Server Survey

In the September 2015 survey we received responses from 892,743,625 sites and 5,438,101 web-facing computers. Both of these key metrics grew this month, with net gains of 18 million sites and 47,000 computers.

Microsoft made by far the largest gain in hostnames this month, with an additional 33.6 million sites bringing its total up to 265 million. Combined with a 15.9 million loss in Apache-powered sites, the difference between Microsoft's and Apache's market shares has now halved: Microsoft's share went up by 3.22 percentage points to 29.68%, while Apache's fell by 2.55 to 34.96%, reducing Apache's lead to just over five percentage points.

However, September's growth in web-facing computers paints a different picture, with Apache's net gain of 19,800 computers being more than six times higher than Microsoft's. Despite this, both Microsoft and Apache lost market share this month, while nginx – which gained the most web-facing computers in September (+22,100) – grew its share to nearly 13%. With an additional 6.9 million sites bumping its site share up to 15.60%, nginx was the only major server vendor to increase its market share in both sites and computers this month.

Despite no longer being supported by Microsoft, the number of websites using Microsoft IIS 6.0 (which typically runs on Windows Server 2003) has since grown by 19% and accounted for much of the overall Microsoft hostname growth this month. 153 million websites are now using Microsoft IIS 6.0, compared with 129 million in July; however, the number of web-facing computers using IIS 6.0 has fallen by 6%, and the number of active sites fell by 16%.

China accounted for around a third of this month's overall growth in web facing computers, outpacing the growth seen in the United States and Germany by a factor of three. Even so, China was responsible for only a tiny fraction of this month's site growth. Microsoft Windows continues to be the preferred hosting platform for computers in China, where it is currently used by 42% of all web-facing computers and 43% of all sites. Astoundingly, over half of these Windows computers are running Windows Server 2003, and some Chinese hosting providers continue to provide new installations of this deprecated operating system.

Amongst the world's top million websites, nginx has continued to increase its market share and now powers more than twice as many sites as Microsoft. Apache's share has been steadily declining over the past few years mostly as a result of nginx's gains, but it looks set to remain the dominant server vendor within the top million for a while longer, as it is still used by more than twice as many sites as nginx.

Total number of websites

Web server market share

DeveloperAugust 2015PercentSeptember 2015PercentChange
Apache327,985,96837.51%312,106,63834.96%-2.55
Microsoft231,429,14626.47%265,010,74629.68%3.22
nginx132,443,39115.15%139,297,80415.60%0.46
Google19,933,0952.28%19,683,0872.20%-0.07
Continue reading

August 2015 Web Server Survey

In the August 2015 survey we received responses from 874,408,576 sites and 5,391,301 web-facing computers, representing a net gain of 25 million sites and 40,978 web-facing computers since last month.

Microsoft was responsible for much of the growth in web-facing computers this month, reversing the losses seen last month. This month there was an increase of 15,668 web-facing computers powered by Microsoft web server software, accompanied by a gain of 6.1 million sites. Microsoft has recovered some web-facing computer market share as a result of the increase; however, it remains on a gradual declining trend – it now stands at almost 2 percentage points below its share this time last year.

nginx performed well across all metrics again this month, gaining 3,421 sites in the top million sites, 6,491 web-facing computers, and 983,000 sites overall. nginx is the only vendor experiencing consistent increases in market share, and is now used by 22.61% of the top million sites, and 12.68% of web-facing computers.

Apache also made gains this month, with 1,243 additional web-facing computers and 2.3 million additional sites. However, it lost 4,775 sites in the top million sites, where its market share is now 47.78%. Despite the net gain in web-facing computers, Apache has again seen a small loss in its market share, which now stands at 46.26%.

LiteSpeed gained 486,000 sites this month, bringing the total number of sites using LiteSpeed's web server to just over 5 million. LiteSpeed uses the same configuration format as Apache and is designed to be a drop-in replacement.

LiteSpeed was the first major web server vendor to add support for the final version of HTTP/2 after it was standardised in May. HTTP/2, which is based on Google's SPDY protocol, aims to improve the performance of HTTP by changing how it is encoded on the wire. It does not change HTTP's semantics to ease compatibility with existing applications. While the standard defines a cleartext version of the protocol, all major browsers only support HTTP/2 over TLS. Out of the 45,819 SSL sites that negotiated the final version of HTTP/2 over TLS this month, 21,695 (47.35%) were served by LiteSpeed.

An initial patch was released by nginx this month for adding HTTP/2 support. The patch is still in development – full HTTP/2 support in nginx is expected by the end of 2015.

Microsoft IIS 10 is the first release of IIS that provides HTTP/2 support. IIS 10 is included in Windows 10, which was released in July, and Windows Server 2016, which is currently in public beta testing and expected to be released in early 2016.

mod_h2, an Apache module which provides HTTP/2 support, was donated to the Apache Foundation in June and merged into the development version of Apache. mod_h2 will be backported to Apache 2.4, the current stable release branch.

Total number of websites

Web server market share

DeveloperJuly 2015PercentAugust 2015PercentChange
Apache325,696,51438.34%327,985,96837.51%-0.83
Microsoft225,282,71326.52%231,429,14626.47%-0.05
nginx131,460,06315.47%132,443,39115.15%-0.33
Google20,255,4242.38%19,933,0952.28%-0.10
Continue reading