August 2014 Web Server Survey

In the August 2014 survey we received responses from 992,177,228 sites — four million fewer than last month.

Despite losing more than six million hostnames and its lead over Apache shrinking to 2.13 percentage points, Microsoft managed to retain the top spot it snatched from Apache last month. Many of the lost hostnames that were using Microsoft IIS belonged to a single Chinese link farm; these sites typically last just a few months and so such volatility in the number of hostnames is no surprise. Apache was the only major server vendor to gain hostnames, adding more than 780,000.

Web Server Developer - Market Share of Computers

The number of web-facing computers, on the other hand, is less susceptible to fluctuations, representing the install base of each server vendor.

Web server market share for computers

DeveloperJuly 2014PercentAugust 2014PercentChange
Apache2,321,14147.92%2,338,92747.83%-0.09
Microsoft1,512,93331.24%1,515,67431.00%-0.24
nginx460,7959.51%478,7939.79%0.28

Whilst nginx lost more than six million hostnames — many from link farms of a similar nature to those using IIS albeit on a smaller scale — the number of web-facing computers running the open-source web server grew by almost 18,000. Although nginx had the largest growth this month, Apache was not far behind, growing by almost 17,800 web-facing computers. Despite this growth, Apache's share of web-facing computers fell slightly, though it retains its lead over Microsoft by a comfortable margin (47.8% vs 31%).

Although all of the major server vendors have experienced regular and steady growth in the web-facing computers metric, they have often been outpaced by nginx. Both Apache and IIS have seen a slow decline in market share as a result, losing 0.9 and 2.4 percentage points respectively over the last year. nginx's share of web-facing computers is now just shy of 10%, almost double its share in August 2012.

DigitalOcean opens London data centre

Cloud hosting provider DigitalOcean has seen tremendous growth over the past 18 months, and this month became the 5th largest hosting provider in terms of web-facing computers. More than half of all Digital Ocean droplets are running Apache, and a further 43% use nginx.

In July, DigitalOcean opened a new data centre in London, its third data centre in Europe. If the growth rate of their Singapore data centre is anything to go by, DigitalOcean could soon become one of the largest hosting providers in the UK.

Less than a month after opening their Singapore data centre in February 2014, DigitalOcean became the 8th largest provider in Singapore. Half a year later, DigitalOcean is now the second largest hosting provider in Singapore, with a total 4,900 computers, trailing only Amazon, which has more than 12,000 web-facing computers.

Explosive growth on .xyz

The biggest gain among the new gTLDs was on .xyz, which entered general availability on July 2nd. In just over a month, .xyz has grown by 225,000 hostnames, and is already the largest of the new gTLDs by both hostnames and domains. Almost all the growth appears to be the result of a Network Solutions promotion which offers a free matching .xyz domain with each .com domain purchased. Almost 200,000 of these new hostnames resolve to a single IP address, which shows a Network Solutions domain holding page.

Other large growths among the new gTLDs include .berlin which gained 85,500 hostnames, .club (+9,700), and .events (+5,500).

Microsoft using Brazilian IP address space in US

Microsoft recently opened an Azure data centre in São Paulo, making its first foray into South America. Shortly after the data centre was opened, some Azure users began noticing that their virtual machines were being assigned Brazilian IP addresses despite not using the new Brazilian data centre. Microsoft attributed this behaviour to its dwindling supply of available IPv4 address space, particularly in the United States. This month, almost 9,000 hostnames were found on 1,600 Brazilian IP addresses despite being hosted in one of Microsoft's other Azure data centres.

Total number of websites
 
Web server market share
 
Developer July 2014 Percent August 2014 Percent Change
Microsoft 373,869,026 37.53% 367,805,416 37.07% -0.46
Apache 345,921,550 34.73% 346,702,990 34.94% 0.22
nginx 141,041,852 14.16% 135,037,738 13.61% -0.55
Google 20,511,505 2.06% 20,076,890 2.02% -0.04
Continue reading

July 2014 Web Server Survey

In the July 2014 survey we received responses from 996,106,380 sites.

Microsoft gained 22 million sites this month, helping to increase its market share by 1.18 percentage points. Combined with a 1.78 point loss in Apache's market share, Microsoft is now the new market leader with a 37.5% share of all sites.

In the early days of the web, hostnames were a good indication of actively managed content providing information and services to the Internet community. Today, hostnames are used for a wide range of activities, including holding pages produced at the point of customer acquisition by domain registration or hosting service companies, typo-squatting advertising providers, speculative domain registrants, and search-engine optimisation companies. Where wildcard DNS is used, the vast majority of the hostnames will not receive visitors, and the resources required to run the sites are minimal.

iis-share
The IIS market share growth in hostnames has not been reflected in our other metrics

Our active sites metric shows just how significant an influence automatically generated sites can have on the number of hostnames found by the survey for all server vendors. Apache remains the clear leader by number of active sites, and has been ever since we started using this additional metric in 2000. Over half of the world's active sites use Apache; a total of more than 91 million active sites, compared with Microsoft's 21 million.

Microsoft now leads in hostnames, but Apache is still far ahead in terms of active sites.

Microsoft’s most recent growth in hostnames since mid-2013 has, for the most part, been caused by a large number of Chinese linkfarms (泛站群). The sites in question provide advertising for gambling sites, online product listings, and normally make use of affiliate schemes. Yet they are hosted in the USA, on generic TLDs such as .com and .net to bypass China’s TLD and internet content provider (ICP) license requirements. Unusually, each linkfarm makes use of a reasonably large number of domains and IP addresses, presumably making them harder for search engines to evade. This would normally be cost prohibitive for this kind of activity, however hosting and domain packages can be found advertised on auction sites specifically for this purpose, with packages of (random/unspecified) .com domains available for as little as ¥17 (~ £2 / $3) each, guaranteed to remain yours for at least a month. It is not clear why IIS has been chosen for these sites, however it does have a considerably higher market share (for all of our metrics) in China compared to worldwide - for example 59% of domains hosted in China use IIS compared to just 29% worldwide.

In just over a year IIS has gained over 236 million hostnames (+172%) while only gaining 503k active sites (+2%). The number of web-facing computers running IIS websites has increased by just over 30k (+2%), compared to Apache’s 171k growth (+8%), and nginx’s 159k growth (+53%), resulting in a 2.4 percentage point loss in market share for IIS by this metric.

computer-share
 
 
Developer June 2014 Percent July 2014 Percent Change
Microsoft 352,208,487 36.35% 373,869,026 37.53% 1.18
Apache 353,672,431 36.50% 345,921,550 34.73% -1.78
nginx 133,763,494 13.81% 141,041,852 14.16% 0.35
Google 20,192,595 2.08% 20,511,505 2.06% -0.02
Continue reading

June 2014 Web Server Survey

In the June 2014 survey we received responses from 968,882,453 sites, six million less than last month. The battle between Microsoft and Apache heated up this month, with Apache losing 13 million sites and Microsoft gaining 26 million. The resultant changes in market share have left Apache barely clinging onto the lead — Microsoft is now only 0.15 percentage points behind. This is the closest Microsoft has ever been, giving it a good chance of taking the lead for the first time next month. However, Apache continues to dominate in terms of active sites, i.e. sites which are actively managed by humans rather than being automatically generated for use in activities such as link farming and domain squatting. Under this metric, Apache's losses were less significant, still leaving it with more than half of the market share, and more than 36 percentage points ahead of its closest competitor, nginx. In terms of all websites, nginx suffered the second largest loss of 8.6 million sites. nginx is very often used as a reverse proxy, although other web servers can also fulfill this role. Apache's mod_proxy module allows it to be configured as either a forward or reverse proxy, and Microsoft IIS can be configured to act as a reverse proxy with the URL Rewrite and Application Request Routing modules. Microsoft Azure Web Sites can also achieve the same functionality once the proxy feature has been enabled. Tengine, which is based on nginx, also fell by three million sites this month. This web server software is used extensively by its originators, Taobao, and has been an open source project since 2011. Tengine supports all of the features of nginx 1.4.7, plus some additional features which are not present in the stable releases of nginx 1.4.x or 1.6.x, such as syslog and pipe support. However, the most recent mainline version (nginx 1.7.1), which was released on 27 May, does now allow the error_log and access_log directives to be logged to syslog.

IPv4 addresses nearing total exhaustion

On 20 May, ICANN announced that it had begun the process of allocating the remaining blocks of IPv4 addresses to the five Regional Internet Registries (RIRs). As the total number of available 32-bit IPv4 addresses dwindles, network operators are being encouraged to adopt the use of 128-bit IPv6 addresses, which will allow a significantly larger number of unique addresses: IPv4 can only provide 4.3 billion addresses, whereas IPv6 can provide nearly 8×1028 times as many. Unfortunately, adoption of IPv6 is proving to be a slow process. Only 3% of the hostnames in this month's survey can be resolved to IPv6 addresses, and the total number of IPv6 addresses used by websites has increased by only 18% over the past 12 months. ICANN's statement says the process of allocating the remaining blocks was triggered when Latin America and Caribbean Network Information Centre's (LACNIC) supply of IPv4 addresses dropped to below 8 million. Topically, this month's survey saw ICANN's website at www.icann.org change its Server banner from Apache to BigIP. For the past few years, it had either been "Apache" or "Apache/2.2.3 CentOS", although the operating system has consistently been identified as F5 BIG-IP throughout. Adobe's community forums at forums.adobe.com also switched to BigIP this month, from Apache-Coyote/1.1.
 
 
Developer May 2014 Percent June 2014 Percent Change
Apache 366,262,346 37.56% 353,672,431 36.50% -1.05
Microsoft 325,854,054 33.41% 352,208,487 36.35% 2.94
nginx 142,426,538 14.60% 133,763,494 13.81% -0.80
Google 20,685,165 2.12% 20,192,595 2.08% -0.04
Continue reading

May 2014 Web Server Survey

In the May 2014 survey we received responses from 975,262,468 sites — 16 million more than last month.

Microsoft threatening Apache's market lead

Microsoft gained nine million additional sites this month, increasing its market share by a further 0.37 percentage points. Meanwhile, despite gaining 4.3 million sites, Apache's market share fell by 0.18 points. Although Apache still leads with 37.6% of all sites, Microsoft is now just 4.1 percentage points behind. Apache has been the most commonly used web server for more than 18 years, but this is the closest Microsoft has ever been to threatening this position.

Apache's position is much stronger when considering only Active Sites — it retains an absolute majority of 52.3%, and second place is held by nginx (14.4%), rather than Microsoft (11.3%). By excluding much of the automatically-generated content present on the internet, the Active Sites metric better reflects web server market share amongst human-maintained web sites.

New releases of nginx

The total number of websites using nginx fell by 3.7 million this month; however, the losses were mostly isolated to just a few hosting companies. Nearly two million nginx sites disappeared at cloud hosting company Enzu, followed by 1.2 million at BurstNet. Within the top million sites, however, nginx was the only web server to grow — Microsoft, Apache, and Google all lost market share.

A new stable version of nginx was released on April 24. nginx 1.6.0 features SPDY 3.1 support and various SSL improvements that were originally implemented in the 1.5.x mainline branch of releases. The previous mainline branch has also been superseded by nginx 1.7.0, which added support for backend SSL certificate verification and support for SNI while working with SSL backends.

Seven million new sites using Microsoft IIS

Nearly seven million of this month's new websites are using Microsoft IIS. Around 11 thousand of these new sites are hosted on the Microsoft Azure platform (including a few phishing sites), helping to maintain Microsoft's position as the largest Windows hosting company in terms on web-facing computers. Most of the new IIS sites at Azure are hosted in the US, with more than a third of the total being hosted in the North Central US Azure region alone.

Notably, www.apachelounge.com is one of this month's websites which appears to have switched to Microsoft IIS. The Apache Lounge is a web forum primarily focused on running Apache in Windows environments, and has provided Windows Apache binaries for more than 10 years. The site understandably has a long history of being hosted from an Apache web server, most recently on a Windows Server 2008 platform, but is now using Microsoft IIS 8.5 on Windows Server 2012.

All may not be as it seems, however, as the web server is still sending an X-Powered-By: Apache/2.4.9 (Win64) header. The web server is also reporting X-Powered-By: ARR/2.5, indicating the use of IIS's load-balancing features. It is likely that Apache Lounge is powered by multiple Apache instances which are hidden behind a Microsoft IIS load balancer.

New gTLDs showing promising growth

Several more new generic top-level domains have shown rapid growth in this month's survey. The .berlin gTLD, which was used by only two websites in last month's survey, is now used by 40,000 websites. The domain started its sunrise registration period on February 14, when trademark owners were able to register .berlin domains ahead of the March 16 deadline.

German news magazine Der Spiegel – one of Europe's largest publications of its kind – criticised the availability of .berlin and other new top-level domains, likening them to a housing bubble and suggesting that such domain names are a waste of money. The company behind the .berlin domain, dotBERLIN GmbH & Co. KG, was quick to reply pointing out that not only had Der Spiegel applied for its own gTLD (.spiegel) two years ago, but it had also already registered the domain name spiegel.berlin before its own article was published.

The .email gTLD is another one that has been thriving since the closure of its sunrise period: More than 12 thousand websites are now using .email, compared with only two last month. This gTLD is one of many operated by Donuts Inc, and became generally available on March 19.  Donuts applied to ICANN for more than 300 TLDs in 2012, 65 of which are already generally available for registration, including .support, .domains, .photos, .estate, .guru and .plumbing. A further 66 TLDs are scheduled for general availability between now and September, including .expert, .parts, .media, .toys, .wtf and .fail.





DeveloperApril 2014PercentMay 2014PercentChange
Apache361,853,00337.74%366,262,34637.56%-0.18
Microsoft316,843,69533.04%325,854,05433.41%0.37
nginx146,204,06715.25%142,426,53814.60%-0.64
Google20,983,3102.19%20,685,1652.12%-0.07
Continue reading

April 2014 Web Server Survey

In the April 2014 survey we received responses from 958,919,789 sites — 39 million more than last month.

Microsoft made the largest gain this month, with nearly 31 million additional sites boosting its market share by 1.9 percentage points. IIS is now used by a third of the world's websites. Although this is not Microsoft's largest ever market share (it reached 37% in October 2007), this is the closest it has ever been to Apache's leading market share, leaving Apache only 4.7 points ahead. Although Apache gained 6.9 million sites, this was not enough to prevent its market share falling by 0.87 to 37.7%. nginx, which gained 3.1 million sites, also lost some of its market share.

More than 70% of this month's new IIS-powered websites are hosted in the US, followed by 22% in China. Nearly 20 million of the new IIS sites in the US are hosted by a single company, Nobis Technology Group, which was also responsible for much of Microsoft's growth in February. A smaller amount of Microsoft IIS growth was also seen on the Windows Azure platform (which will be renamed to Microsoft Azure on April 3), where the total number of active sites has grown by 25% since February, when we compared the platform against Amazon AWS. 84% of all active sites hosted on the Azure platform are running Microsoft web server software.

Many of the new IIS sites hosted by Nobis Technology Group feature similar content and form part of a Chinese link farm. Link farming is often an attempt to influence search engine results, and each individual site within a link farm is typically of little interest to a human. Netcraft's active sites metric therefore provides a better idea of how many websites are actively managed rather than being automatically generated en mass, such as link farm content and domain holding pages. Of the 114 million sites hosted by Nobis, only a fifth are counted as active sites.

In terms of active sites, Apache remains in a much stronger position with a 52% share of the market, compared with Microsoft's 11%. A significantly higher proportion of Apache sites are active: 26% of all Apache sites were deemed to be active, whereas only 6% of Microsoft's were. nginx takes a 14% share of the active sites market, putting it 3 points ahead of Microsoft.

Apache also fares well amongst the million busiest sites, where there is intrinsically very little interference from domain holding pages, link farms and other web spam. Here Apache takes a 53% share of the market, while nginx has 18% and Microsoft has 12%. Although only 3% of the top million sites use Google web server software, Google's dominance amongst the very busiest sites give it a presence on 8 of the top 10 sites.

Both Apache and nginx were affected by security vulnerabilities which were resolved during March, whereas Microsoft IIS has yet to be affected by publicly-known security issues this year.

The latest version of Apache (2.4.9) was released on March 17. The Apache Software Foundation describes this as representing fifteen years of innovation by the project, and this major release of the 2.4 stable branch is recommended over all previous releases. Nevertheless, it is still common for many websites to use the legacy 2.2 branch of releases, or even older versions. Apache 2.4.9 is primarily a security and bug fix release, although it also includes the changes introduced in 2.4.8, which was not actually released. A workaround for a bug in older versions of OpenSSL, which prevented the release of 2.4.8, has been included in 2.4.9.

Although Apache 2.4.8 was not released, the development version (Apache/2.4.8-dev) was found on 675 sites during this survey, which ran in March. Nearly all of these sites were running on FreeBSD servers which belonged to various Apache projects, mostly Apache HTTPD and Apache OpenOffice.

The stable branch of nginx was updated twice during March. Two bugs were resolved in nginx 1.4.6, which was released on March 4. nginx 1.4.7 was then released on March 18, addressing another bug and a heap buffer overflow vulnerability. This security vulnerability affected nginx's SPDY module, where a specially crafted request could allow a remote attacker to execute arbitrary code on a vulnerable web server. nginx is notable for its SPDY support, which is used extensively by CloudFlare and also by Automattic, which hosts millions of WordPress blogs and co-sponsored the development of the ngx_http_spdy_module. The same SPDY vulnerability also affected the mainline branch of nginx, which was resolved with the release of nginx 1.5.12.

Many of the new generic top level domains (gTLDs) are starting to appear in Netcraft's Web Server Survey in significant numbers. For example, the previous survey saw only one website using the .guru gTLD, whereas this month's survey (which ran during March) found 36 thousand. Other gTLDs which have shown significant growth since last month's survey include .photography, .today, .tips, .technology, .directory, .land, .gallery, .estate and .singles.

Amongst established TLDs, the number of sites using the .ga country code top level domain grew by 140% this month. The My GA website allows .ga domains to be registered for free from between 1 and 12 months, which has no doubt helped towards their goal of increasing the awareness of Gabon across the globe. The .ga ccTLD is administered by the Agence Nationale des Infrastructures Numériques et des Fréquences (ANINF) in Libreville, Gabon, while the registration process is provided by Freenom, who also provide free domain registrations for the more popular .tk ccTLD. Registered Freenom users are allowed an unlimited number of domain name renewals on both the .ga and .tk d domains, while paying customers can choose to register domains for as long as 10 years in one go and can automatically renew the registration.

Free and easily-registerable domain names are obviously attractive to fraudsters: During February, Netcraft blocked nearly 1,500 unique phishing sites hosted on .ga domains alone, and this figure jumped to more than 2,400 in March. The vast majority of these phishing attacks targeted Chinese companies, particularly the Taobao marketplace and the Alipay online payment escrow service.





DeveloperMarch 2014PercentApril 2014PercentChange
Apache354,956,66038.60%361,853,00337.74%-0.87
Microsoft286,014,56631.10%316,843,69533.04%1.94
nginx143,095,18115.56%146,204,06715.25%-0.31
Google20,960,4222.28%20,983,3102.19%-0.09
Continue reading

March 2014 Web Server Survey

In the March 2014 survey we received responses from 919,533,715 sites — around half a million fewer than last month.

Apache has gained some breathing space this month. Nearly a year of strong market share growth by Microsoft eventually culminated in the gap between Apache and Microsoft being reduced to only 5.4 percentage points. After the gap dropped to its lowest point last month, Microsoft had looked set to usurp Apache as the most common web server within a matter of months. This month's survey saw Microsoft lose 15.8 million sites and Apache gain 3.2 million, however. Apache's market share increased to 38.6%, 7.5 percentage points ahead of Microsoft and bucking the recent trend.

Most of Microsoft's losses this month were seen at Nobis Technology Group, where more than 30 million link-farming sites stopped operating. Nobis is a private holding company, which owns the DarkStar voice communications network and Ubiquity Hosting Solutions, which has seven data centres across the United States.

nginx gained 5 million sites this month, increasing its market share to 15.6%. The latest mainline version of nginx (1.5.10) now supports SPDY 3.1, which extends the flow control features of SPDY 3.0 by allowing different sessions within a single connection to send data at different rates. It is no surprise that SPDY 3.1 is already supported in the Google Chrome web browser; SPDY was primarily developed by Google, and is one of their trademarks. SPDY 3.1 has also been supported in Mozilla Firefox since 4th February 2014.

Content delivery network CloudFlare — which uses its own web server software based on nginx — rolled out SPDY 3.1 support for all of its customers in February. Since last month, Netcraft's SSL Survey has identified a four-fold increase in the number of HTTPS websites supporting SPDY 3.1, most of which are hosted by CloudFlare. A smaller number of these SPDY 3.1 sites are hosted by the owner of the WordPress.com blogging platform, Automattic, which was one of the sponsors of the ngx_http_spdy_module.

Mozilla has been planning to remove SPDY 2 support from Firefox since September 2013, and this looks set to happen with the release of Firefox 28. Some developers asked for SPDY 2 support to be retained, arguing that dropping support for SPDY 2 would effectively drop SPDY support in many SPDY-enabled websites. However, nginx and CloudFlare now supporting SPDY 3.1 allays some of that concern.

LibreOffice — the free open source office suite bundled with Ubuntu Linux — moved its website from an Apache web server to nginx at the end of January, apparently for performance reasons. Incidentally, this further distances LibreOffice from the Apache Software Foundation - LibreOffice was forked from OpenOffice.org in 2010, before the latter was given to the ASF where development continued under the name of Apache OpenOffice.

More than 30 new generic top-level domains (gTLDs) were delegated to the Root Zone during February, making them officially part of the internet. There are now 471 top level domains in total. The new ones added in February included .flights, .wiki, .xyz, .fish and .移动 (xn--6frz82g – Chinese for "mobile").

Many of these new gTLDs were applied for by Donuts Inc, a US domain registry which was founded in 2011. The company's CEO and co-founder, Paul Stahura, previously founded domain name registrar eNom in 1997. Donuts raised more than $100,000,000 in its Series A financing round and applied to ICANN for more than 300 TLDs in 2012. As a registry, Donuts does not sell domain names directly to the public; instead, customers must purchase them from one of its accredited registrars.





DeveloperFebruary 2014PercentMarch 2014PercentChange
Apache351,700,57238.22%354,956,66038.60%0.38
Microsoft301,781,99732.80%286,014,56631.10%-1.69
nginx138,056,44415.00%143,095,18115.56%0.56
Google21,129,5092.30%20,960,4222.28%-0.02
Continue reading