June 2016 Web Server Survey

In the June 2016 survey we received responses from 1,045,534,808 sites and 5,951,685 web-facing computers. This reflects an increase of 12 million sites, along with a modest gain of 4,700 computers.

Apache regained the lead from Microsoft this month, with a large increase of 60 million sites taking its total up to 360 million, while Microsoft lost 24 million. Microsoft enjoyed a brief foray at the top in April and May, thanks to a proliferation of link farming sites, but now stands 18 million sites behind Apache.

Apache's net growth included only 14 million new websites – the remainder consisted of existing websites that switched to Apache after previously using other web server software. Most notably, 52 million sites switched from Tengine to Apache, while 12 million switched from Microsoft. The number of websites using Tengine fell by more than 60%, largely as a result of the migration to Apache. Most of the sites involved in this switch were hosted by OVH in Canada, and not only changed server vendor, but also moved to a different hosting company—Data Foundry—in the United States.

Although the number of Tengine websites fell to 29 million, the number of active sites using Tengine actually increased slightly to 1.8 million. In a similar vein, the number of active sites running Apache fell by 630,000, even though the total number of sites grew by more than 60 million.

Out of the largest vendors, Microsoft has the lowest proportion of active sites, with only 4.8% of its 340 million sites being active, while Tengine's proportion has crept up to 6.3%. Both are significantly lower than Apache's proportion of 22.3% and nginx's 21.6%.

Newer versions of web server software generally attract a much higher proportion of active sites. For example, 56% of sites running on Microsoft IIS 10 (which will be included with Windows Server 2016, but is already available for Windows 10) are classified as active, while 23% of sites running IIS 8.5 are active, along with 14% of IIS 8.0 sites. This proportion dwindles to only 2.8% by the time we consider IIS 6.0, which remains a very popular choice of server in China despite no longer being supported by Microsoft.

nginx also demonstrates this trend, with the largest number of its active sites running on the latest 1.10.x stable branch. More than half of all sites using this version are active.

Across all versions, nginx continues to muscle its way into the market with confidence. This month it gained almost a million active sites, along with an additional 31,000 web-facing computers, giving it the largest growth in these important metrics. Conversely, Apache lost 26,000 computers, while Microsoft lost 4,500.

nginx has also continued to grow its presence amongst the top million websites, where it is now used by 27.6% of sites. Apache continues to lead with a 43.7% share of this market, although its share has generally been on the decline since 2011. If current trends continue, nginx could possibly take the lead from Apache within the next couple of years.

Total number of websites

Web server market share

DeveloperMay 2016PercentJune 2016PercentChange
Apache300,447,47029.06%360,458,01834.48%5.41
Microsoft366,964,00935.50%342,605,66632.77%-2.73
nginx163,902,97115.85%169,316,54716.19%0.34
Google21,567,2522.09%21,662,6732.07%-0.01
Continue reading

May 2016 Web Server Survey

In the May 2016 survey we received responses from 1,033,790,346 sites and 5,946,961 web-facing computers. This reflects a gain of 147,000 computers, coupled with a loss of 49 million sites.

While last month's survey recorded the largest number of sites ever, many of the Chinese sites running Microsoft IIS that appeared last month have since disappeared. Combined with other departures, Microsoft suffered a net loss of 75 million sites this month, which has played a major part in its market share falling by more than 5 percentage points to less than 36%. Nevertheless, it is still the most common server vendor by number of sites, with a total of nearly 370 million hosted on IIS servers.

Despite Microsoft's loss of 75 million sites, the number of active sites using IIS actually grew by 450,000, which is indicative of the low quality of the sites it lost. Most of the lost sites were engaged in link farming activity, with large numbers of these sites being served from relatively few computers. The loss of these sites therefore had little impact on the number of web-facing computers using Microsoft IIS, which grew by 14,000.

Microsoft's closest competitor, Apache, gained 8.4 million sites, with its increased market share of 29.1% putting it within 6.4 percentage points of Microsoft's leading share.

Although it has yet to reach the same level as Microsoft and Apache, nginx made the largest gains, growing by 21 million sites and increasing its market share by 2.6 points to 15.9%.

nginx also showed the strongest growth in the survey's other metrics: it gained nearly 7.5 million active sites, 74,100 web-facing computers, and increased its presence within the top million sites by 16,000. The most significant of these gains was nginx's active site count increasing by a whopping 27%, largely as a result of Tumblr sites now exhibiting the Server: nginx header (in previous months, most Tumblr sites did not reveal which server software they were using).

While Microsoft has shaken off many of its low-quality sites, Alibaba's nginx fork, Tengine, gained around 10 million. Most of the new sites served by Tengine this month make use of domains under the .science gTLD, which has proved popular with many Chinese link farms and webspam sites – most likely due to the sub-dollar registration costs. Tengine suffered a small net loss in active sites this month, which corroborates the low quality of the 10 million new sites.

Only 2.4% of the sites served by Tengine now qualify as active sites, which highlights just how many of them are used for displaying automatically generated content. Microsoft is still also fairly popular with link farm operators (particularly in China), with only 4.6% of its sites showing active content. In contrast, more than 26% of Apache sites, and nearly 22% of nginx sites feature active content.

Total number of websites

Web server market share

DeveloperApril 2016PercentMay 2016PercentChange
Microsoft441,470,89440.75%366,964,00935.50%-5.26
Apache292,043,54826.96%300,447,47029.06%2.10
nginx143,349,43913.23%163,902,97115.85%2.62
Google20,597,6051.90%21,567,2522.09%0.18
Continue reading

April 2016 Web Server Survey

In the April 2016 survey we received responses from 1,083,252,900 sites and 5,800,222 web-facing computers. This reflects a gain of nearly 80 million sites and 18,100 computers.

This is the largest number of sites the survey has ever seen, beating the previous maximum of 1,028,932,208 in October 2014. The number of web-facing computers is also at its largest, although this total has generally risen much more steadily than the number of sites.

Microsoft was the only major vendor to gain sites this month, and so it was solely responsible for this month's total reaching its highest value ever. Apache lost 33 million sites, while nginx and Google suffered much smaller losses. Many of the 124 million additional sites using Microsoft IIS are aimed at a Chinese audience. Several million are served from just a handful of IP addresses, using either IIS 6.0 or 7.5.

However, this proliferation of new Microsoft-powered websites is largely driven by automated processes. Many are "spam" sites that use link farming techniques to attract traffic. Although Microsoft's website count grew by a remarkable 38.9% in April, it lost 12,100 web-facing computers. High quality websites that attract genuine repeat traffic tend to have a very low number of sites per computer compared with the computers that are involved in link farming, which sometimes host millions of automatically-generated sites each. Corroborating this further, Microsoft suffered a loss of 341,000 active sites this month, taking its total down by 2.0%.

Meanwhile, nginx continued its relentless growth. It gained 19,500 web-facing computers this month (+2.4%), was the only major vendor to increase its active sites count, and increased its share within the top-million websites by 0.49 percentage points.

nginx is particularly prominent at Amazon and DigitalOcean, with the two hosting companies accounting for more than 25% of all nginx computers. In particular, nginx is the most commonly used server at DigitalOcean, being used by just under half of its web-facing droplets. At Amazon, despite its large share of all nginx computers, Apache is more than twice as common, with nginx only used on a quarter of EC2 instances.

Total number of websites

Web server market share

DeveloperMarch 2016PercentApril 2016PercentChange
Microsoft317,761,31831.65%441,470,89440.75%9.10
Apache325,285,18532.40%292,043,54826.96%-5.44
nginx143,464,29314.29%143,349,43913.23%-1.06
Google20,790,7672.07%20,597,6051.90%-0.17
Continue reading

March 2016 Web Server Survey

In the March 2016 survey we received responses from 1,003,887,790 sites and 5,782,080 web-facing computers. This reflects a gain of nearly 70 million sites, but a loss of 14,100 computers.

This is the second time the total number of sites has reached more than a billion. This milestone was first reached in September 2014, although it was short-lived: By November 2014, the total fell back below one billion, and had stayed that way until the current month. During the intervening period, the total fell as low as 849 million sites in April 2015.

The total number of websites is typically prone to large fluctuations. Domain holding companies, typo squatters, spammers and link farmers can cause millions of sites to be deployed in a short space of time, without any significant outlay, but these types of site are intrinsically uninteresting to humans. Netcraft's active sites metric counters the effect of these by discounting sites that appear to be automatically generated. This leads to a more-stable metric that better illustrates real, practical use of the web.

The number of active sites currently stands at just 171 million, meaning around 1 in 6 sites are active. The total fell by 764,000 this month, but nginx stands out as being the only major vendor to increase its active site count — by an impressive 699,000. This has increased its active sites share to 16.4%, while Apache's loss of nearly a million active sites took its leading share down to 49.2%.

Typifying nginx's rise amongst active sites, it also showed the only growth in web-facing computers amongst the major server vendors. This month's survey found more than 15,000 additional computers running nginx on the web, while Microsoft's loss of 30,000 computers was the primary cause of the overall loss in this metric. Thankfully, the majority of this decline consisted of Windows Server 2003 computers, which arguably helps improve the safety of the internet — this server software is no longer supported by Microsoft.

China accounts for over 30% of all web-facing computers that run Windows Server 2003, making it the largest user of this obsolete operating system; however, more than half of this month's Windows Server 2003 losses were seen in China, which has helped to bring this share down slightly.

Apache's computer growth was relatively modest at only 447 computers, but Microsoft's large loss caused Apache's market share to increase by 0.12 to 47.9%. nginx's gain of 15,000 computers took its market share up by 0.30 to 14.3%, but Microsoft remains a fair way ahead of nginx with a 26.6% share of the market.

Total number of websites

Web server market share

DeveloperFebruary 2016PercentMarch 2016PercentChange
Apache306,292,55732.80%325,285,18532.40%-0.39
Microsoft278,593,04129.83%317,761,31831.65%1.82
nginx137,459,39114.72%143,464,29314.29%-0.43
Google20,640,0582.21%20,790,7672.07%-0.14
Continue reading

February 2016 Web Server Survey

In the February 2016 survey we received responses from 933,892,520 sites and 5,796,210 web-facing computers.

Microsoft has edged closer towards Apache, with an increase of 16.1 million sites bringing its total up by 6.14% to 279 million. Apache's relatively modest growth of 0.66% has put Microsoft within 3 percentage points of Apache's leading market share of 32.8%.

In terms of web-facing computers, Apache maintains a much clearer lead with a 47.8% share of the market. Microsoft also takes second place by this metric, albeit with a share that is more than 20 percentage points behind Apache. However, both Apache and Microsoft suffered small losses in market share as nginx continues to exhibit strong growth: This month, nginx gained 21,100 computers, increasing its market share by 0.26 points to 13.96%.

nginx 1.9.11 mainline was released on 9th February. This version introduced support for dynamic modules, enabling selective loading of both third-party and some native modules at runtime. In previous versions, nginx modules had to be statically linked into an nginx binary built from source, causing the module to be loaded every time even if it was not going to be used.

The latest version of Microsoft Internet Information Services, IIS 10.0, is still very rare in the wild, as its primary deployment platform (Windows Server 2016) has yet to be released. Fewer than 5,000 websites are currently using IIS 10.0, and these are being served either from technical preview versions of Windows Server 2016, or from Windows 10 machines.

The latest technical preview version of Windows Server 2016 also supports a headless deployment option known as Nano Server. This is a stripped-down version of Windows Server, without a graphical interface and a few other features that are not essential for modern web applications. As a result, it typically requires fewer updates to be installed – and consequently, fewer reboots, too.

Despite losing a small amount of market share, Apache also showed a reasonable growth of 15,600 computers. Similar to last month, a significant proportion of this growth was due to the appearance of more Western Digital My Cloud consumer storage devices.

The total number of My Cloud devices in the survey now stands at 583,400, which is 68,400 more than last month; however, the number of devices that are exposed directly to the internet grew by only 11,100.

Western Digital is using Amazon AWS to host the servers that proxy requests to My Cloud devices in Relay mode. Most of these relay servers have been configured to serve a few thousand devices each, and so the 331,000 devices that are currently using Relay mode contribute fewer than 200 computers towards Apache's total.

Interestingly, while most web-facing My Cloud devices are hosted in the US, more than half of the *.wd2go.com hostnames used by the relay servers are hosted in Amazon's EU regions.

Total number of websites

Web server market share

DeveloperJanuary 2016PercentFebruary 2016PercentChange
Apache304,271,06133.56%306,292,55732.80%-0.76
Microsoft262,471,88628.95%278,593,04129.83%0.88
nginx141,443,63015.60%137,459,39114.72%-0.88
Google20,799,0872.29%20,640,0582.21%-0.08
Continue reading

January 2016 Web Server Survey

In the January 2016 survey we received responses from 906,616,188 sites and 5,753,264 web-facing computers, reflecting a modest increase of less than six million sites, but a significant gain of 174,000 computers.

Microsoft gained 22.5m sites (+9.40%), which has taken its market share up by 2.32 points. Meanwhile, Apache lost 16.4m sites, and nginx fell by 15.6m. Apache's market share is now less than 5 points ahead of Microsoft; this difference was more than twice as large just two months ago.

The web-facing computers metric is typically much more stable, but this month's overall gain of 174,000 computers is unusually large as a result of a 7.6% increase in the number of web-facing computers running Apache.

This large gain comprised of nearly 195,000 Apache computers, and the majority of these are Western Digital My Cloud personal storage devices. These consumer devices run web servers and can be accessed using public hostnames with a format similar to device1000000-a1b2c3d4.wd2go.com. Consumers can remotely access their files via the My Cloud web application, a mobile app, or via third-party applications that make use of the relatively new My Cloud OS 3 platform.

Consumers can remotely access their files via the My Cloud web application (shown), or via a mobile app.

Consumers can remotely access their files via the My Cloud web application (shown), a mobile app, or third-party tools.

More than 240,000 of these wd2go.com hostnames point directly to a variety of consumer broadband connections, which is where the My Cloud devices are physically located.

Network Attached Storage (NAS) devices are rarely exposed to the internet on such a large scale, and so this provides some otherwise invisible insights into the usage of these particular devices. Although consumers do not have to enable the Cloud Access feature, the 240,000+ devices that are directly exposed to the internet are likely to be a fairly representative sample of all similar Western Digital devices.

Nearly half of the My Cloud devices that are exposed directly to the internet are located in the US, while the UK has the next largest share of 13%, and France follows with 6%. This suggests that nearly two-thirds of Western Digital's consumer NAS sales take place in these three countries alone.

As well as the My Cloud devices that are exposed directly to the internet, a further 273,000 wd2go.com hostnames resolve to fewer than 200 IP addresses hosted by Amazon AWS. These hostnames likely represent additional My Cloud devices that have been cloud-enabled using Relay mode. In this mode, requests bound for the device are relayed via the Amazon-hosted web service, which makes it possible for a consumer to gain remote access even when they are not able to set up port forwarding on their router.

However, whilst certainly convenient, exposing a My Cloud device to the internet (either directly or in relay mode) could undermine a consumer's security by revealing the device's internal IP address to the whole world. Each of the 500,000+ My Cloud devices that can be accessed via hostnames like device1070698-xxxxxxxx.wd2go.com also have corresponding DNS entries that reveal their local IP addresses:

$ host device1070698-xxxxxxxx.wd2go.com
device1070698-xxxxxxxx.wd2go.com has address 78.72.xx.x
$ host device1070698-xxxxxxx-local.wd2go.com
device1070698-xxxxxxxx-local.wd2go.com has address 192.168.1.65

These "-local" DNS entries allow a remote attacker to discover the local IP address of a consumer's My Cloud device (in this case, 192.168.1.65), which would make it easier to carry out CSRF attacks against it. Even if the consumer has taken the precaution of changing the device's name so that his browser cannot reach it via the default local address (http://wdmycloud), it could still be reached by browsing directly to its local IP address. Devices that have not been updated recently might still be vulnerable to remote code execution via CSRF attacks.

The local IP address of the My Cloud device can also be used to infer the address of the consumer's broadband router, which may well be vulnerable to similar types of attack. Knowing some likely IP addresses of the router makes CSRF attacks much more feasible – for example, if the My Cloud device has an IP address of 10.10.0.31, the attacker could deduce that the router's IP address might be 10.10.0.1 or 10.10.0.255, rather than any of the other 17+ million IANA-reserved private network addresses. A successful exploit against a vulnerable router could give an attacker full control over the router's settings, which could ultimately lead to data theft or financial losses through pharming attacks.

While the influx of these My Cloud devices has resulted in strong growth for Apache, nginx continued its steady progress by gaining a further 23,300 (+3.0%) web-facing computers. Apache's market share in terms of computers now stands at 47.9% (+2.0), while Microsoft lost 20,600 computers, contributing to its share falling to 27.1%. Despite maintaining the consistent growth it has demonstrated for several years, nginx also suffered a minor loss in share by virtue of Apache's exceptional growth.

Total number of websites

Web server market share

DeveloperDecember 2015PercentJanuary 2016PercentChange
Apache320,676,75935.59%304,271,06133.56%-2.03
Microsoft239,927,01326.63%262,471,88628.95%2.32
nginx157,001,01817.43%141,443,63015.60%-1.82
Google20,362,6782.26%20,799,0872.29%0.03
Continue reading