June 2015 Web Server Survey

In the June 2015 survey we received responses from 863,105,652 sites and 5,346,650 web-facing computers, representing an increase of 5.2 million websites and 65,000 additional computers.

Microsoft was responsible for the majority of this month's hostname growth, with a gain of 6.6 million sites, but only contributed 11,700 additional web-facing computers. This has caused Microsoft's market share by hostnames to overtake its declining market share by computers, with both standing at just under 30%.

Apache led this month's web-facing computer growth with a net gain of 24,800, while nginx followed closely with 22,800. This has resulted in nginx's market share increasing by 0.28 to 12.4%, and despite showing the largest net growth, Apache's share fell slightly.

Apache, Microsoft and nginx together account for more than 88% of all web-facing computers in the world, making these vendors by far the most popular choices. However, nginx is the only vendor experiencing consistent increases in market share, up by 3 percentage points over the last year while both Apache and Microsoft have seen losses. The next most commonly used server is lighttpd (pronounced "lighty"), which is used by a mere 0.46% of web-facing computers.

nginx's market share has also been steadily increasing within the top million websites. Its share now stands at 21.9%, and although Apache's use within the million busiest sites has been steadily declining this decade, Apache looks likely to retain the lead for at least a few more years.

Three months after the death of Sir Terry Pratchett, approximately 84,000 websites are now serving the X-Clacks-Overhead: GNU Terry Pratchett header in tribute. Invisible to the majority of users, this HTTP header is a reference to the Discworld novel Going Postal, which features a series of communication towers called the clacks.

In the book, a similar header ("GNU John Dearheart"), is transmitted around the clacks after the inventor's son is killed in an accident while working on a clacks tower. The G means send the message on, N means do not log the message, and U means turn the message around at the end of the line and send it back again — this ensures that the message is transmitted indefinitely, allowing his son to be memorialised forever. Similarly, by transmitting Pratchett's name around the internet, the sites participating in this HTTP header tribute hope to keep his legacy alive. After all, as it says in the book, "A man is not dead while his name is still spoken."

One of the most popular sites to use the X-Clacks-Overhead: GNU Terry Pratchett header is www.theguardian.com, which alone reached more than 5 million unique browsers per day in 2014. With each header taking up 40 bytes of an uncompressed HTTP request, all of the sites involved in the tribute could be generating terrabytes of additional bandwidth usage every day.

Total number of websites

Web server market share

DeveloperMay 2015PercentJune 2015PercentChange
Apache336,813,95939.26%334,731,03538.78%-0.48
Microsoft247,784,66828.88%254,408,17929.48%0.59
nginx123,697,64514.42%122,965,52214.25%-0.17
Google20,103,0682.34%20,130,7322.33%-0.01
Continue reading

PayPal.com unexpectedly offline

PayPal.com experienced some unexpected outages today, leaving many of its 165 million active customers unable to access the site via its paypal.com domain.

p-3894.652

All of Netcraft's geographically distributed performance monitors have showed outages of at least an hour so far today.

But requests to www.paypal.com continued to work throughout the outage, and customers could successfully log in by browsing directly to that hostname; however, it is likely that many users visit the site by typing only "paypal.com" into their browser's address bar.

The shorter paypal.com address does not serve a homepage; its sole purpose is to redirect visitors directly to the main www.paypal.com website. This shortcut saves the user having to type "www." into his browser, but during the outage, anyone attempting to reach the site via this commonly used method would have faced error messages similar to this (unless the redirect has been cached from a previous visit):

timeout

Even a short outage can have a huge impact on a financial giant like PayPal. In 2014, PayPal handled 4 billion payments with a total payment volume of $235 billion.

The cause of the outage is as yet unknown: The PayPal Notifications website does not show any scheduled maintenance for the live site until tomorrow, and does not currently list any unscheduled events corresponding to today's outage.

status

阿里云的发展壮大已让阿里集团成为中国最大的服务器主机提供商

[Read this article in English]

中国云主机服务提供商阿里云正在以前所未有的速度发展壮大。从今年4月到5月仅一个月的时间,从阿里云直接连接到国际互联网的Web服务器数量就增长了8千多台。这样的迅猛增长已让其母公司阿里巴巴集团一跃成为世界第四大同时也是中国第一大服务器主机提供商。

Continue reading

Most Reliable Hosting Company Sites in May 2015

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 EveryCity SmartOS 0:00:00 0.008 0.079 0.066 0.133 0.133
2 Datapipe Linux 0:00:00 0.017 0.085 0.012 0.024 0.032
3 Anexia Linux 0:00:00 0.017 0.633 0.088 0.176 0.176
4 CWCS Linux 0:00:00 0.017 0.188 0.104 0.186 0.187
5 XILO Communications Ltd. Linux 0:00:00 0.030 0.179 0.064 0.129 0.129
6 INetU Windows Server 2008 0:00:00 0.038 0.098 0.068 0.199 0.404
7 Netcetera Windows Server 2012 0:00:00 0.038 0.056 0.081 0.161 0.161
8 New York Internet FreeBSD 0:00:00 0.047 0.186 0.030 0.062 0.180
9 Hyve Managed Hosting Linux 0:17:02 0.051 0.156 0.064 0.127 0.128
10 Umbee Hosting Linux 0:00:00 0.076 0.072 0.064 0.481 1.231

See full table

EveryCity had the most reliable website during May 2015, responding to all but two of Netcraft's requests. Since Netcraft began monitoring its site in April 2014, EveryCity has maintained a 100% uptime record. Besides SmartOS-based hosting, EveryCity also specialises in Solaris hosting on both SPARC and Intel platforms.

In second place is Datapipe, with just four failed requests to its website during May 2015. Two other companies had the same number of failures: Anexia and CWCS. Datapipe's impressively quick average connection time (0.012 seconds) gives it the edge in the tie-breaker, fending off both Anexia (0.088 seconds) and CWCS (0.104 seconds). Datapipe has data centres in key technology and financial hubs including New York, Silicon Valley, London, and Hong Kong.

Anexia (3rd place) and CWCS (4th place), both with four failed requests and both of which are based in Europe, differ in their focus: Anexia operates 58 data centres across the world, whereas CWCS concentrates on the UK market, owning two "state of the art" British data centres. Anexia announced in early May that it had become an official Debian mirror, providing support to the open-source project.

Making it into the top 10 for the first time since its monitoring began in February 2015, Umbee Hosting's website responded to all but a handful of Netcraft's requests. Umbee Hosting has five data centres, spread across three continents: two in the London region, two in metro New York, and one in Sydney.

Linux remains a popular choice amongst the top 10 hosting company websites, powering six. The use of SmartOS and FreeBSD brings up the total of UNIX-based operating systems to eight. The remaining two are powered by Windows Server 2008 and 2012.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

Aliyun cloud growth makes Alibaba largest hosting company in China

Chinese cloud hosting company Aliyun is growing faster than ever, with more than 8,000 more web-facing computers found in the May 2015 survey than in April's. This growth has launched Aliyun's parent company, Alibaba Group, into position as the 4th largest hosting company in the world, as well as the largest in China.

This reflects a positive change in Aliyun's fortunes, whose earlier impressive growth had started to stagnate towards the end of 2013. However, growth later resumed in earnest, coinciding with Aliyun's partnership with rival cloud computing company Inspur in July 2014. The growth has continued ever since, with the largest absolute growth being seen between April 2015 and May 2015.

aliyun-growth

Aliyun now accounts for 38% of the web-facing computers hosted by Alibaba Group, while 44% are operated by HiChina, which was acquired by Alibaba in 2009. If the latest growth trends continue, Aliyun will soon account for the majority of web-facing computers at Alibaba.

All of the web-facing computers at Aliyun are located in China, which offers significant advantages for the local market. Hosting a website close to its end-users generally results in faster page loads, but increased reliability is the most crucial factor in this case. Connectivity between China and other countries is often slow, unstable or even blocked, making China the most practical location for hosting websites aimed at local consumers.

However, this also means that Aliyun would be a troublesome choice for any company that has a significant user base outside of China. This is exemplified by the following graph, which shows the performance and reliability of www.aliyun.com when accessed from the Netherlands:

uptime

This connectivity problem has so far proven to be insurmountable, and is likely to be a showstopper for most companies with customers outside of China. Unfortunately, this problem only seems to be getting worse: Nearly half of the requests made from the Netherlands over the past 20 days failed, whereas only 4% failed during a similar 20-day period in 2013.

Globally, Amazon continues to dominate the hosting arena with nearly three times as many web-facing computers as Alibaba Group. DigitalOcean recently usurped OVH Net to become the second largest hosting company. Amazon, DigitalOcean and Aliyun are all similar in that they provide relatively low-cost virtual servers, but Aliyun's growth is likely restricted by the impracticalities of using it to serve content outside of China.

Considering this rather significant restriction, it is impressive that Aliyun's current growth rate is almost on par with DigitalOcean's (in fact, Alibaba Group as a whole exceeded DigitalOcean's absolute growth in May 2015). This growth perhaps demonstrates the scale of the Chinese market, and if it were practical to use Aliyun to host websites for a global audience, Aliyun could well give Amazon a run for its money.

Amazon and DigitalOcean are likely to remain ahead for a fair while, particularly as they both provide a variety of hosting locations in several different countries. This gives customers flexibility over where a website can be hosted, providing not just performance benefits, but also regulatory ones — for example, German data protection laws limits where companies can store personal data, in particular making it unappealing to do so outside of Germany or the EU.

Amazon is perhaps still best known for its retail operations, but recently surprised some analysts by announcing that its Amazon Web Services segment is profitable. This segment generated sales of $1.57bn in the first quarter of 2015, and operating income of $265m, demonstrating that it can now operate without having to fall back on Amazon's other revenue streams.

Aliyun operates under a similar safety net, with its parent, Alibaba Group, having significant revenue from other business areas, including business-to-business trading via alibaba.com and an eBay-like consumer-to-consumer marketplace on taobao.com. Both Aliyun and Amazon Web Services have had opportunities to grow in this relatively risk-free environment, where – if necessary – they can be supported by the parent group's other business areas.

With this safety net in place, Aliyun is well placed to continue its growth within China, and could even contemplate adding datacenters abroad. Notably, its ability to invest in new datacenters is not likely to be a problem: Alibaba Group (NYSE:BABA) has a higher market capitalisation than Amazon (NASDAQ:AMZN).

Aliyun has already tried to attract foreign customers by offering support and site content in English, and aliyun.com also mentions that it is "preparing to support more languages to improve user experience". New customers must provide their phone number when creating an Aliyun account, which is verified by text message in the 23 supported countries. Customers in other countries can also register an Aliyun account by following a slower offline registration process.

However, any plans for global expansion could be scuppered unless Aliyun can solve the connectivity issues and also match the low prices offered by DigitalOcean, where the cheapest virtual machine costs only $5/month, including up to 1TB of data transfer. A somewhat-similar Elastic Compute Service instance at Aliyun (1-core, 512MB memory, 20GB storage, bandwidth limited to 3Mbps) costs ¥109/month, making it more than three times as expensive. This instance costs only ¥38/month if the customer chooses Aliyun's pay-as-you-go option for bandwidth (which would cope better with bursts of traffic), but this could work out far more expensive for heavy users — at ¥0.80 per GB of outbound public network traffic, 1TB of traffic would cost over $120 at Aliyun, whereas it is included in the price of DigitalOcean's $5 droplet.

May 2015 Web Server Survey

In the May 2015 survey we received responses from 857,927,160 sites and 5,281,889 computers. The number of sites detected increased by nearly 9M this month after two consecutive months of losses. The number of web facing computers also increased by 54k.

Microsoft had the largest growth of web sites, gaining more than one percentage point of market share. Apache, the current leader with a market share of 39.26%, remained stable. Nginx, however, experienced the largest loss amongst major web server vendors and consequently saw a small loss in market share.

Nginx is performing well within the million busiest sites, and when counting the number of web facing computers using the web server — being responsible for the largest growth in each category. Nginx gained just over 2k of the million busiest sites, giving it a market share of 21.64%. Almost 16% of nginx's market share (or 3.4% of the top million sites) in the top million busiest sites is due to CloudFlare. While nginx is used to serve requests at CloudFlare, they may be proxied to backend servers running other web servers. Nginx contributed over half of the total net gain of web facing computers this month, with an increase of 27,500 computers.

DigitalOcean recently became the second largest hosting company by the number of web facing computers, overtaking OVH. Apache, with 48% of publicly visible DigitalOcean computers, is the leading web server, closely followed by nginx with 45.5%.

As the remaining IPv4 address space dwindles, companies are more often resorting to the emerging transfer market to acquire additional IPv4 ranges — for example, DigitalOcean recently appeared as one of the leading inbound receivers of IPv4 addresses from transfers within the RIPE NCC Service Region. With prices expected to increase as IPv4 becomes a more valuable asset, the adoption of IPv6 could accelerate. Netcraft currently finds 740k IPv6 addresses, an increase of 25% compared to the same time last year.

Total number of websites

Web server market share

DeveloperApril 2015PercentMay 2015PercentChange
Apache333,285,74139.25%336,813,95939.26%0.00
Microsoft236,288,84327.83%247,784,66828.88%1.05
nginx126,274,77814.87%123,697,64514.42%-0.45
Google20,051,4332.36%20,103,0682.34%-0.02
Continue reading