December 2017 Web Server Survey

In the December 2017 survey we received responses from 1,734,290,608 sites across 212,870,632 unique domain names and 7,014,428 web-facing computers. This reflects a gain of 5.34 million domains and 121,000 computers.

Web Server Developers - Market Share of Domains

The number of hostnames in use on the web has been a headline metric since the inception of the Web Server Survey, but it has been subjected to quite large fluctuations in recent years. Netcraft has therefore introduced the number of unique domains as an additional metric that provides a more stable view of the web.

The domains metric is not influenced by wildcarded domains or other large numbers of sites that can be hosted under a single domain name with minimal effort; but unlike the active sites metric, the domains metric still takes account of sites that are still under construction, or running hosting company or domain registrar holding pages.

Web server market share for domains

The noticeable spike in Apache-powered domains in May 2013 was caused by the largest hosting company of the time, GoDaddy, switching a large number of its domains from Microsoft IIS to Apache Traffic Server (ATS) . GoDaddy switched back to using IIS 7.5 a few months later.

Today, Apache still has the largest market share by number of domains, with 81.4 million giving it a market share of 38.2%. It also saw the largest gain this month, increasing its total by 1.53 million. This growth was closely followed by nginx, with a gain of 1.09 million domains increasing its total to 47.5 million. While Microsoft leads by overall number of hostnames, it lags in 3rd position when considering the number of unique domains those sites run on, with a total of 22.8 million.

Web-facing Computers

The number of web-facing computers provides an alternative view that corresponds more closely to the install base of each server vendor.

With 1.63 million web-facing computers, nginx is already 97,800 computers ahead of Microsoft since it took second place in October, but Apache remains much further ahead with a total of 2.98 million. Apache experienced the largest gain of 58,000 computers this month, closely followed by nginx with 49,000, and with Microsoft trailing with an increase of just 22,000.

Web server market share for computers

Web Server Updates

Microsoft's Internet Information Services platform has benefitted from a few improvements since the publication of last month's survey. The newest version of the IIS Administration API (2.2.0) introduced new endpoints that make it easy to monitor the health of a web server, as well as the individual websites and application pools running on it. There is also a new configuration endpoint for the files API, which allows the API's root folders to be configured – this means administrators no longer have to edit a file to configure which sections of the file system can be accessed via the API.

Version 1.0 of the IIS CORS Module, which works on IIS 7.5 or later, was also released in November. This enables support for the Cross-Origin Resource Sharing protocol, which lets webpages make use of resources that are hosted on other websites, such as web fonts and AJAX endpoints. If a website hosts these resources without setting a suitable CORS policy, the default same-origin policy enforced by all browsers would prevent other websites from accessing them.

The latest version of the open source LiteSpeed HTTP server, OpenLiteSpeed 1.4.28 (stable), was released on 8 November. This release adds multithreading APIs for LSIAPI – the API that allows it to support third-party modules. Although there are only 12,400 web-facing computers running LiteSpeed, these computers host 2.42 million domains. It is not clear how many of these computers are already running LiteSpeed 1.4.28, as this server does not expose version information in its headers.

lighttpd 1.4.48 was subsequently released on 11 November. This adds a new mod_authn_sasl module, which provides Simple Authentication and Security Layer (SASL) authentication similar to Apache's libapache2-mod-authn-sasl module. With 20,800 web-facing computers running lighttpd, it has a greater install base than LiteSpeed, but its market share of domains is noticeably smaller with a count of 565,000.

nginx 1.13.7 was released on 21 November, although this addresses several bugs rather than introducing any new features. There are, however, several new features in the latest version of its commercially supported product, NGINX Plus Release 14, which was announced on 12 December. This release features several improvements, including an updated live monitoring dashboard and JSON support in its nginScript scripting language; and there is also a technology preview of its extended clustering support, which lets NGINX Plus instances in a cluster share state information.

Total number of websites

Web server market share

DeveloperNovember 2017PercentDecember 2017PercentChange
Microsoft669,517,17736.80%535,762,81330.89%-5.91
Apache443,521,99524.38%446,418,87825.74%1.36
nginx367,687,48920.21%395,881,69022.83%2.62
Google20,333,6041.12%21,308,0691.23%0.11
Continue reading

Most Reliable Hosting Company Sites in November 2017

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Webair Linux 0:00:00 0.000 0.145 0.048 0.095 0.097
2 Swishmail FreeBSD 0:00:00 0.000 0.138 0.055 0.110 0.156
3 vXtream Ltd Linux 0:00:00 0.005 0.148 0.066 0.132 0.132
4 Hyve Managed Hosting Linux 0:00:00 0.005 0.093 0.068 0.141 0.141
5 Pair Networks FreeBSD 0:00:00 0.005 0.236 0.073 0.144 0.144
6 Bigstep Linux 0:00:00 0.005 0.142 0.075 0.151 0.151
7 Netcetera Linux 0:00:00 0.005 0.096 0.089 0.180 0.180
8 CWCS Linux 0:00:00 0.005 0.217 0.150 0.231 0.231
9 New York Internet FreeBSD 0:00:00 0.010 0.289 0.024 0.048 0.049
10 One.com Linux 0:00:00 0.010 0.187 0.037 0.110 0.110

See full table

Both Webair and Swishmail responded to all of our requests in November 2017, but the faster average connect time for Webair means it takes the top spot. Webair is a US-based provider of fully managed hosting solutions and has appeared in the top ten a total of seven times in 2017, also taking the top spot in May. Swishmail, a provider of email and web hosting solutions using FreeBSD, has appeared in the top ten five times in 2017.

The six websites ranking third to eighth each failed to respond to only a single request in November. vXtream and Hyve (in third and fourth places) have both made their tenth appearances in the top ten in 2017 - vXtream was absent only in March, while Hyve narrowly missed out by ranking 11th in January. London based vXtream recently acquired Qube Managed Services contracts and infrastructure including presence in datacentres in London, New York and Zurich. Hyve places fourth with an average connect time of 68ms, 2ms slower than vXtream, it offers fully managed cloud hosting from datacentres in the UK, US and China.

Pair Networks owns its own datacentre in Pittsburgh and also offers space in the datacentre of a partner in Denver, it is one of the three websites in the top ten to be run on FreeBSD.

Linux is once again the most prevalent operating system in use for the top ten websites, with seven of the ten websites using it. FreeBSD is the operating system of choice for the remaining three.

Netcraft measures and makes available the response times of around thirty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

LinkedIn certificate blunder leaves users LockedOut!

Many LinkedIn users were unable to access the professional networking website today after its administrators failed to renew a TLS certificate before it expired.

Image10

The certificate in question was used by various country-specific LinkedIn websites such as https://uk.linkedin.com and https://de.linkedin.com. It expired at midday today, immediately preventing users from accessing the site via these hostnames.

The expired certificate was issued to us.linkedin.com, but was also valid for – and used by – dozens of other country-specific LinkedIn hostnames. The main site at www.linkedin.com was not affected.

The expired certificate was issued to us.linkedin.com, but was also valid for – and used by – dozens of other country-specific LinkedIn hostnames. The main site at www.linkedin.com was not affected.

The sites were still inaccessible a few hours after the problem manifested itself.

The sites were still inaccessible a few hours after the problem manifested itself.

Ironically, LinkedIn's better-than-average security made the expired certificate even more problematic. Most browsers will allow users to ignore certificate validation warnings — however unwise that may be — but the warnings cannot be ignored on these LinkedIn sites.

LinkedIn is in a minority of sites that make use of a security feature called HTTP Strict Transport Security. This feature protects HTTPS sites against trivial man-in-the-middle attacks, but unfortunately in this case, the additional security made the site completely unreachable for regular users.

Good security requires great care: Strict Transport Security is a good idea, but when a certificate expires, users cannot visit the site because browsers will not allow the warnings to be ignored.

Good security requires great care: Strict Transport Security is a good idea, but when a certificate expires, users cannot visit the site because browsers will not allow the warnings to be ignored when an active HSTS policy is in place.

Many modern browsers, such as Firefox and Chrome, simply do not allow users to add an exception when a site has an HSTS policy in place. LinkedIn's HSTS policy has a validity period of 30 days, which means that anyone who has visited the site within the past month would have been unable to add a certificate exception, and would therefore not be able to visit the site until LinkedIn renewed the certificate.

LinkedIn's expired certificate was renewed shortly before this article was published.

November 2017 Web Server Survey

In the November 2017 survey we received responses from 1,819,412,110 sites and 6,893,323 web-facing computers, reflecting a gain of 4.17M sites and 6,961 computers.

This month’s web server survey saw Microsoft’s market share amongst all sites fall by 12.64 percentage points due to a loss of 228M sites. Despite this, Microsoft still retains its place with the largest market share by this metric of 36.80%, with Apache trailing at 24.38%. The majority of the loss occurred at just one hosting provider where over 190M Microsoft sites were lost.

This change isn’t reflected in the active sites metric which only saw minor changes amongst the main web server vendors. Microsoft lost only 0.03 percentage points of its market share with a drop of 261k active sites. Apache leads in the active sites metric by a considerable margin, increasing its share slightly this month to 44.55%.

Amongst the top million busiest sites Microsoft experienced a small increase in market share, pausing its general decline in this market. nginx experienced the largest growth with an increase of 2,133 of the top million sites.

nginx also saw the largest increase in number of web-facing computers, gaining 25k and pulling 1 percentage point of market share clear of Microsoft, which it overtook last month. Apache also experienced a gain in computers, albeit smaller at just 7k. It remains considerably ahead with a 42.38% market share.

New gTLDs Seen for the First Time

This month the controversial new .search gTLD being run by Google’s Charleston Road Registry subsidiary was found for the first time, with www.nic.search responding to the survey. Google hopes it will be able to run .search as a dotless domain which will automatically redirect users to their search engine of choice. This proposal has been criticised for going against ICANN’s own rules, which prohibits this functionality due to the potential for conflicts with existing names on internal networks. This feature could also cause confusion for users who have come to expect that typing words into their address bar will perform a search query for that term.

It is currently uncertain whether or not Google will be allowed to run the .search TLD as a dotless domain, however with the launch of the first site on this TLD this month Google is one step closer to the provision of this service.

Total number of websites

Web server market share

DeveloperOctober 2017PercentNovember 2017PercentChange
Microsoft897,467,51749.44%669,517,17736.80%-12.64
Apache340,811,23518.78%443,521,99524.38%5.60
nginx333,942,60418.40%367,687,48920.21%1.81
Google21,127,0781.16%20,333,6041.12%-0.05
Continue reading

Major update to Netcraft Anti-Phishing Extension for Firefox

An update to the Netcraft Anti-Phishing Extension for Mozilla Firefox is now available. This release replaces the Toolbar interface with a modern Button interface to sit alongside the browser's address bar.

firefox-extension-cropped

The upcoming Firefox 57 — to be released on the 14th November — represents a major overhaul of the browser, and removes support for legacy XUL extensions. Future versions of Firefox will only support the new cross-browser WebExtensions API.

The Netcraft Anti-Phishing Extension (known then as the Netcraft Toolbar) was first made available for Internet Explorer in December 2004. A Firefox version followed in May 2005. The current button-style Anti-Phishing Extension was released for Google Chrome and the Opera browser in 2012 and 2013 respectively. The new extension enjoys a 4.5 star rating on the Google Chrome Store.

The Extension runs on any operating system supported by the desktop version of Mozilla Firefox and displays the hosting location, country, longevity, popularity, and an abstracted risk rating for each site visited. In particular its key features are:

  • Protection against phishing sites — The Netcraft anti-phishing community is effectively a giant neighbourhood watch scheme, empowering the most alert and most expert members to defend everyone within the community. As soon as the first recipients of a phishing mail report it, we can block it for all users of the extension providing an additional level of protection from Phishing. Netcraft processes reports of fraudulent URLs from a diverse variety of sources and proactively searches for new fraudulent sites.
  • Detailed site reports — simply click the Netcraft logo to access a wealth of information about the sites you visit, helping you to make informed choices about their safety.
  • Risk Ratings — we evaluate the characteristics of the site compared against those depicted by fraudulent sites. The result is a simple visual summary displayed on the site report.
  • Conveniently report suspected phishing & fraudulent sites — At the click of the button you can report suspected web forgeries to Netcraft, helping to protect the community. Netcraft operates an incentive scheme for Phishing site submissions, including iPads, backpacks, mugs, and more... Over 38.4 million phishing sites have been detected and blocked by Netcraft since the anti-phishing service was launched (November 2017).
  • Protection against cross site scripting (XSS) — The extension optionally traps XSS and other suspicious URLs which contain characters highly likely to deceive.
firefox-blocked-url

The Extension is available for download from the Firefox add-ons page and requires no special administrator privileges to install. Users of the existing Netcraft Anti-Phishing Toolbar will be upgraded automatically to the latest version.

Versions of the Extension are available for other browsers on the Google Chrome Store and Opera add-ons page.

Customised versions with corporate branding and navigation are also available.

Most Reliable Hosting Company Sites in October 2017

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 www.viawest.com Linux 0:00:00 0.000 0.273 0.005 0.200 0.201
2 vXtream Ltd Linux 0:00:00 0.000 0.137 0.058 0.117 0.117
3 Hyve Managed Hosting Linux 0:00:00 0.000 0.088 0.060 0.126 0.126
4 ServerStack Linux 0:00:00 0.000 0.119 0.063 0.125 0.125
5 Rackspace F5 0:00:00 0.000 0.161 0.074 0.150 0.150
6 One.com Linux 0:00:00 0.004 0.192 0.038 0.108 0.108
7 XILO Communications Ltd. Linux 0:00:00 0.004 0.216 0.066 0.132 0.132
8 Webair Linux 0:00:00 0.009 0.141 0.052 0.104 0.106
9 Memset Linux 0:00:00 0.009 0.144 0.062 0.234 0.362
10 EveryCity SmartOS 0:00:00 0.009 0.106 0.066 0.305 0.306

See full table

ViaWest had the most reliable hosting company website in October 2017, successfully responding to all of Netcraft's requests with an average connect time of just 5 milliseconds. This is ViaWest's third appearance in the top-ten this year, and its first time in the number-one spot. ViaWest was acquired by Peak 10 Inc. in August, and now operates as the combined company Peak 10 + ViaWest. It boasts 40 data centers connected by a 100 Gigabit core network backbone.

vXtream and Hyve came in second and third respectively. Both sites successfully responded to all of Netcraft's requests, but vXtream clinched second place with a marginally quicker average connect time of 58 milliseconds. Both companies are on a top-ten streak: vXtream appearing every month since April, and Hyve appearing every month since February.

ServerStack and Rackspace also had no failed requests this month, placing fourth and fifth based on their average connect time. ServerStack's site has had a 100% uptime record over the past two years, and has appeared in Netcraft's monthly most reliable hosting company sites more than 40 times since 2012. Serverstack operates three data centers across the US and Europe, and offers a 100% uptime SLA.

Linux remains the most popular choice of operating system, with eight of this month's top-ten hosting companies using it. F5 and SmartOS also make an appearance.

Netcraft measures and makes available the response times of around thirty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.