December 2014 Web Server Survey

In the December 2014 survey we received responses from 915,780,262 sites and 5,034,578 web-facing computers.

This is the second month in a row where there has been a large drop in the total number of websites, giving this month the lowest count since January. As was the case in November, the loss has been concentrated at just a small number of hosting companies, with the ten largest drops accounting for over 52 million hostnames. The active sites and web facing computers metrics were not affected by the loss, with the sites involved being mostly advertising linkfarms, having very little unique content. The majority of these sites were running on Microsoft IIS, causing it to overtake Apache in the July 2014 survey. However the recent losses have resulted in its market share dropping to 29.8%, leaving it now over 10 percentage points behind Apache.

Despite losing more than six million hostnames this month, nginx outpaced all other major server vendors by gaining 22,300 web-facing computers. nginx is now used by nearly 11% of all web-facing computers – twice the share that it had two years ago.

Overall, the total number of web-facing computers in our survey increased by just over 40,000 this month, making nginx responsible for more than half the increase. Despite an increase of over 11,000 computers for Apache, and 1,700 for IIS, both continue to lose market share.

Thanks to continued strong growth at Amazon Web Services, Amazon is the largest hosting company by a considerable margin in terms of our web-facing computers metric (which includes web-facing virtual machines, providing that each has its own kernel and IP address). With nearly 300,000 web-facing computers in total, Amazon has just over twice as many as second-place OVH. In October, we reported that DigitalOcean had become the 4th largest hosting company in under 2 years, but it quickly reached third place in November and is continuing to close the gap on OVH.

Cloud growth

Both Amazon Web Services and Microsoft Azure expanded their cloud hosting footprints recently. Amazon opened a new European AWS region in Frankfurt, which augments its existing EU region in Ireland. Besides being able to host services closer to the center of Europe, the new region means that customers can now build multi-region applications with the assurance that their data will stay within the EU. The new Frankfurt region houses two EC2 availability zones and three AWS edge locations.

Microsoft's new Azure "geo" is in Australia, and consists of two geographically redundant regions in New South Wales and Victoria. This will help Microsoft to compete with Amazon's two EC2 availability zones in Sydney.

New TLDs

More new top-level domains showing strong growth in this month's survey include .nyc, which is targeted for use by New Yorkers, and .realtor, which is only allowed to be used by members of the National Association of Realtors or the Canadian Real Estate Association. These have grown from virtually nothing to a total of 40,000 and 80,000 sites respectively.

Ironically, one of this month's fastest growing new top-level domains started off as an April Fool's Day joke in 2009, when the founder of OVH announced the creation of the .ovh TLD – years before such things were actually possible. This joke resulted in over 22,000 requests to register .ovh domains within a few hours, demonstrating the potential demand for such domains. OVH eventually entered into a Registry Agreement with ICANN in January 2014, and the sunrise period for the .ovh TLD began in September. This month's survey saw the number of sites using .ovh domains grow from 6,000 to 63,000, likely due to the first 50,000 .ovh domains being given away for free, and with subsequent growth being fuelled by attractive pricing: new .ovh domains can currently be registered for only EUR 0.99 per year and renewed for EUR 1.99.

Total number of websites

Web server market share

DeveloperNovember 2014PercentDecember 2014PercentChange
Apache350,853,79837.05%358,159,40539.11%2.06
Microsoft306,029,30732.31%272,967,29429.81%-2.51
nginx139,130,99214.69%132,467,76314.47%-0.23
Google19,560,2062.07%20,011,2602.19%0.12
Continue reading

November 2014 Web Server Survey

In the November 2014 survey we received responses from 947,029,805 sites and 4,994,577 web-facing computers.

Despite the dramatic loss of almost 82 million sites since October, the number of web-facing computers increased by over 23 thousand (+0.47%) this month. Automatically generated content and wildcard sites are often used for activities such as domain holding and search-engine results manipulation, where the majority of the sites will not receive human visitors and therefore the resources required to run the sites are minimal. This is clearly demonstrated this month, with over half of the net loss of sites being attributed to a single IP address that was previously hosting parked websites.

The most significant contributor to the increase in web-facing computers was nginx, with almost 16k more computers using the web server this month. Apache and Microsoft both experienced small losses in market share as a result, continuing the trend seen over the past few years.

nginx gained a number of notable high-traffic websites in this month's survey. After several years of using lighttpd, the SourceForge download site (downloads.sourceforge.net, traffic rank #411) has switched to nginx, and gaming news site pcgamer.com (#1339) has also started using nginx with Rackspace's Cloud Load Balancing as a Service product.

Amongst the top million websites, Apache's market share has continued its slow decline. Ever since Netcraft started publishing these figures, Apache has commanded more than half of this market sector. This month, however, Apache has reached its lowest market share ever: 50.01%. Although it still clings on to more than half of the market this month, it looks unlikely to retain it for long. Microsoft and Google have also been losing share amongst the top million sites, all making room for nginx, which has muscled its way up to 20.4%.

New top level domains are continuing to add a modest number of sites to the survey each month. One of this month's fastest growing new TLDs is .audio, which is run by Uniregistry, where it is pitched as a dedicated online space for sound, musicians, engineers and producers. More than 17,000 sites are already using this TLD.

German states and cities are also contributing to the growth in new TLDs. For example, .koeln is now used by 14,000 sites, and .bayern is already used by more than 10,000 websites since it entered general availability on 30 September 2014.

Just in time for this year's Black Friday events on November 28th, the .blackfriday TLD was also launched recently by Uniregistry. More than 10,000 sites have adopted this domain in the space of a month, including amazon.blackfriday and target.blackfriday, which redirect to each company's main US website. Uniregistry also provides another seasonal TLD, .christmas, which was found to be used by 12,000 sites in this month's survey.

NCC Group released its 124-page .trust Technical Policy [pdf] last month, which outlines a stringent set of criteria which must be met by registrants of .trust domains. Many of these policies relate to security, dictating both what must and must not be done – the table of contents alone contains 71 instances of "Do not". The policies go far beyond the typical fraud prevention requirements set by most other registries, with a heavy emphasis on web application security. For example, websites which use .trust domains must serve all content over an encrypted HTTPS connection, must not use inline JavaScript or the eval() function, must provide an appropriate Content Security Policy header, and must be free from open redirects.

NCC Group bought the .trust TLD earlier this year from Deutsche Post, which had originally obtained it from ICANN in 2013. With NCC Group claiming that security compliance is externally verifiable, it anticipates that all sites using the .trust TLD will be recognised as trustworthy.

Total number of websites

Web server market share

DeveloperOctober 2014PercentNovember 2014PercentChange
Apache385,354,99437.45%350,853,79837.05%-0.40
Microsoft345,485,41933.58%306,029,30732.31%-1.26
nginx148,330,19014.42%139,130,99214.69%0.28
Google19,431,0261.89%19,560,2062.07%0.18
Continue reading

October 2014 Web Server Survey

In the October 2014 survey we received responses from 1,028,932,208 sites, which is nearly six million more than last month.

Apache regains the lead

Microsoft lost the lead to Apache this month, as the two giants continue to battle closely for the largest share of all websites. Apache gained nearly 30 million sites, while Microsoft lost 22 million, causing Apache to be thrust back into the lead by more than 36 million sites. In total, 385 million sites are now powered by Apache, giving it a 37.45% share of the market.

A significant contributor to this change was the expiry of domains previously used for link farming on Microsoft IIS servers. The domains used by these link farms were acquired and the sites are now hosted on Apache servers at Confluence-Networks, which display Network Solutions parking notices.

A new major release in the Apache 2.2 legacy branch was announced on 3 September. Apache 2.2.29 also incorporates many changes — including several security fixes — from version 2.2.28, which was not officially released. New versions of nginx stable and mainline were also released during September, which included fixes for an SSL session reuse vulnerability, plus several other bugfixes.

Top million sites

The million busiest websites now represent less than 0.1% of all websites in the survey, but provide an insight into the preferences amongst the sites which are responsible for the great majority of today's web traffic.

Just over half (50.2%) of the top million sites use Apache, which is very similar to its share amongst all active sites; however, nginx's market share is skewed noticeably higher amongst the top million sites, where it powers 20.3% of sites, compared with only 14.3% of all active sites.

Computer growth

The most stable metric is the market share of web-facing computers — hundreds of thousands of websites can easily be served from a single computer (and subsequently disappear all in one go) but it is obviously far less trivial and less desirable to deploy or decommission a significant number of computers. Netcraft's survey is also able to identify distinct computers which use multiple web-facing IP addresses, which adds further stability.

Apache leads in this market with a 47.5% share, and Microsoft also performs well with 30.7%, but both have been gradually falling over the past few years as a result of nginx's strong growth. nginx gained more than 17,000 additional web-facing computers this month, helping to bring its market share up to 10.3%.

New top level domains

The relatively new .xyz domain, which showed tremendous growth over the past couple of months, has started to flatten out slightly after gaining only 33,000 sites this month (+8%). Nonetheless, this is still quite a healthy gain, albeit notably less than last month's growth of 177,000 hostnames which then boosted its total by 78%.

Other promising TLDs include .london, .hamburg and .公司, each of which had fewer than 50 sites in last month's survey, but now have 17,000, 11,000 and 10,000 sites respectively.

The internationalised .公司 (.xn--55qx5d) TLD is delegated to the Computer Network Information Center of Chinese Academy of Sciences. It means "company", making it the Chinese equivalent of .com.

Total number of websites

Web server market share

DeveloperSeptember 2014PercentOctober 2014PercentChange
Apache355,925,98534.79%385,354,99437.45%2.66
Microsoft371,406,90936.31%345,485,41933.58%-2.73
nginx144,717,67014.15%148,330,19014.42%0.27
Google19,499,1541.91%19,431,0261.89%-0.02
Continue reading

September 2014 Web Server Survey

In the September 2014 survey we received responses from 1,022,954,603 sites — nearly 31 million more than last month.

More than a billion websites

This is the first time the survey has exceeded a billion websites, a milestone achievement that was unimaginable two decades ago.

Netcraft's first ever survey was carried out over 19 years ago in August 1995. That survey found only 18,957 sites, although the first significant milestone of one million sites was reached in less than two years, by April 1997.

Fuelled by the dot-com bubble between 1997 and 2000, the survey reached nearly 10 million sites by the start of 2000. The active sites metric was added to our survey shortly afterwards, immediately showing that a significant proportion of websites were automatically generated, displaying identical tag structures, and used for activities such as holding pages, typo-squatting advertising providers, speculative domain registrants, and search-engine optimisation companies.

Rapid hostname growth has continued ever since, with the number of active sites increasing at a far gentler rate. Just under half of the hostnames in our June 2000 survey were active sites, whereas today, less than one in five are active — 178 million active sites in total.

Microsoft, Apache, and nginx

Microsoft and Apache currently take the lion's share of the web server market (just over 71% combined), while Microsoft edged into the lead for the first time in July 2014. Nginx has been steadily gaining share over the last 7 years, and is now used to serve just over 14% of all hostnames.

The view by number of active sites is very different, however. While Microsoft has seen a rapid growth in their hostname market share of around 20 percentage points since September 2011, there has been almost no change in their share of the active sites in this time. Nginx overtook Microsoft in terms of active sites in 2012, and today has a market share of 14.5% – more than 2 points ahead of Microsoft, whose web server software is used by only 11.9% of active sites. However, Apache truly dominates this market, with more than half of all active sites choosing to use Apache software.

Recently nginx has been seeing even greater gains in terms of web facing computers, doubling their market share in the last 2 years to just over 10% this month. Apache and Microsoft are continuing to experience increases in their number of web facing computers, however the growth is often far smaller than that of nginx. This month they gained just 323 and 414 computers respectively, compared to an increase of over 17k for nginx.

New top level domains

Dozens of new TLDs were added to the Root Zone during this month's survey, including .deals, .healthcare, .realtor, .auction, .yandex, .city and .lgbt. Recent additions which have now started to experience growth in the survey include .media, .services, .reisen, .pictures, .exchange and .toys. Each of these TLDs had only two or three sites last month, but all are now in their thousands.

The .xyz domain, which we mentioned last month, has outpaced all of the other new gTLDs after a Network Solutions promotion offering a free matching .xyz domain with each .com domain purchased. This month an additional 177,000 hostnames were found under this TLD, bringing the total number of .xyz sites up by 78% to 403,000. Even faster growth was seen among the .中国 (xn--fiqs8s) internationalised domain name for China, which grew by 181% to a total of 73,000 sites.

Total number of websites

Web server market share

DeveloperAugust 2014PercentSeptember 2014PercentChange
Microsoft367,805,41637.07%371,406,90936.31%-0.76
Apache346,702,99034.94%355,925,98534.79%-0.15
nginx135,037,73813.61%144,717,67014.15%0.54
Google20,076,8902.02%19,499,1541.91%-0.12
Continue reading

August 2014 Web Server Survey

In the August 2014 survey we received responses from 992,177,228 sites — four million fewer than last month.

Despite losing more than six million hostnames and its lead over Apache shrinking to 2.13 percentage points, Microsoft managed to retain the top spot it snatched from Apache last month. Many of the lost hostnames that were using Microsoft IIS belonged to a single Chinese link farm; these sites typically last just a few months and so such volatility in the number of hostnames is no surprise. Apache was the only major server vendor to gain hostnames, adding more than 780,000.

Web Server Developer - Market Share of Computers

The number of web-facing computers, on the other hand, is less susceptible to fluctuations, representing the install base of each server vendor.

Web server market share for computers

DeveloperJuly 2014PercentAugust 2014PercentChange
Apache2,321,14147.92%2,338,92747.83%-0.09
Microsoft1,512,93331.24%1,515,67431.00%-0.24
nginx460,7959.51%478,7939.79%0.28

Whilst nginx lost more than six million hostnames — many from link farms of a similar nature to those using IIS albeit on a smaller scale — the number of web-facing computers running the open-source web server grew by almost 18,000. Although nginx had the largest growth this month, Apache was not far behind, growing by almost 17,800 web-facing computers. Despite this growth, Apache's share of web-facing computers fell slightly, though it retains its lead over Microsoft by a comfortable margin (47.8% vs 31%).

Although all of the major server vendors have experienced regular and steady growth in the web-facing computers metric, they have often been outpaced by nginx. Both Apache and IIS have seen a slow decline in market share as a result, losing 0.9 and 2.4 percentage points respectively over the last year. nginx's share of web-facing computers is now just shy of 10%, almost double its share in August 2012.

DigitalOcean opens London data centre

Cloud hosting provider DigitalOcean has seen tremendous growth over the past 18 months, and this month became the 5th largest hosting provider in terms of web-facing computers. More than half of all Digital Ocean droplets are running Apache, and a further 43% use nginx.

In July, DigitalOcean opened a new data centre in London, its third data centre in Europe. If the growth rate of their Singapore data centre is anything to go by, DigitalOcean could soon become one of the largest hosting providers in the UK.

Less than a month after opening their Singapore data centre in February 2014, DigitalOcean became the 8th largest provider in Singapore. Half a year later, DigitalOcean is now the second largest hosting provider in Singapore, with a total 4,900 computers, trailing only Amazon, which has more than 12,000 web-facing computers.

Explosive growth on .xyz

The biggest gain among the new gTLDs was on .xyz, which entered general availability on July 2nd. In just over a month, .xyz has grown by 225,000 hostnames, and is already the largest of the new gTLDs by both hostnames and domains. Almost all the growth appears to be the result of a Network Solutions promotion which offers a free matching .xyz domain with each .com domain purchased. Almost 200,000 of these new hostnames resolve to a single IP address, which shows a Network Solutions domain holding page.

Other large growths among the new gTLDs include .berlin which gained 85,500 hostnames, .club (+9,700), and .events (+5,500).

Microsoft using Brazilian IP address space in US

Microsoft recently opened an Azure data centre in São Paulo, making its first foray into South America. Shortly after the data centre was opened, some Azure users began noticing that their virtual machines were being assigned Brazilian IP addresses despite not using the new Brazilian data centre. Microsoft attributed this behaviour to its dwindling supply of available IPv4 address space, particularly in the United States. This month, almost 9,000 hostnames were found on 1,600 Brazilian IP addresses despite being hosted in one of Microsoft's other Azure data centres.

Total number of websites
 
Web server market share
 
Developer July 2014 Percent August 2014 Percent Change
Microsoft 373,869,026 37.53% 367,805,416 37.07% -0.46
Apache 345,921,550 34.73% 346,702,990 34.94% 0.22
nginx 141,041,852 14.16% 135,037,738 13.61% -0.55
Google 20,511,505 2.06% 20,076,890 2.02% -0.04
Continue reading

July 2014 Web Server Survey

In the July 2014 survey we received responses from 996,106,380 sites.

Microsoft gained 22 million sites this month, helping to increase its market share by 1.18 percentage points. Combined with a 1.78 point loss in Apache's market share, Microsoft is now the new market leader with a 37.5% share of all sites.

In the early days of the web, hostnames were a good indication of actively managed content providing information and services to the Internet community. Today, hostnames are used for a wide range of activities, including holding pages produced at the point of customer acquisition by domain registration or hosting service companies, typo-squatting advertising providers, speculative domain registrants, and search-engine optimisation companies. Where wildcard DNS is used, the vast majority of the hostnames will not receive visitors, and the resources required to run the sites are minimal.

iis-share
The IIS market share growth in hostnames has not been reflected in our other metrics

Our active sites metric shows just how significant an influence automatically generated sites can have on the number of hostnames found by the survey for all server vendors. Apache remains the clear leader by number of active sites, and has been ever since we started using this additional metric in 2000. Over half of the world's active sites use Apache; a total of more than 91 million active sites, compared with Microsoft's 21 million.

Microsoft now leads in hostnames, but Apache is still far ahead in terms of active sites.

Microsoft’s most recent growth in hostnames since mid-2013 has, for the most part, been caused by a large number of Chinese linkfarms (泛站群). The sites in question provide advertising for gambling sites, online product listings, and normally make use of affiliate schemes. Yet they are hosted in the USA, on generic TLDs such as .com and .net to bypass China’s TLD and internet content provider (ICP) license requirements. Unusually, each linkfarm makes use of a reasonably large number of domains and IP addresses, presumably making them harder for search engines to evade. This would normally be cost prohibitive for this kind of activity, however hosting and domain packages can be found advertised on auction sites specifically for this purpose, with packages of (random/unspecified) .com domains available for as little as ¥17 (~ £2 / $3) each, guaranteed to remain yours for at least a month. It is not clear why IIS has been chosen for these sites, however it does have a considerably higher market share (for all of our metrics) in China compared to worldwide - for example 59% of domains hosted in China use IIS compared to just 29% worldwide.

In just over a year IIS has gained over 236 million hostnames (+172%) while only gaining 503k active sites (+2%). The number of web-facing computers running IIS websites has increased by just over 30k (+2%), compared to Apache’s 171k growth (+8%), and nginx’s 159k growth (+53%), resulting in a 2.4 percentage point loss in market share for IIS by this metric.

computer-share
 
 
Developer June 2014 Percent July 2014 Percent Change
Microsoft 352,208,487 36.35% 373,869,026 37.53% 1.18
Apache 353,672,431 36.50% 345,921,550 34.73% -1.78
nginx 133,763,494 13.81% 141,041,852 14.16% 0.35
Google 20,192,595 2.08% 20,511,505 2.06% -0.02
Continue reading