February 2018 Web Server Survey

In the February 2018 survey we received responses from 1,838,596,056 sites across 214,036,874 unique domain names and 7,290,968 web-facing computers. This reflects a gain of 63,000 computers, and nearly a million additional domains. Overall hostname growth was 33 million, and the number of active sites grew by 9.3 million.

Microsoft dominated this month's domain growth, with an extra 1.1 million domains (+2.0%) taking its market share up by 0.41 percentage points to 26.5%, while Apache lost 115,000 domains. Microsoft also saw the largest hostname growth, with an additional 59 million sites taking its leading hostname share up by 2.65 points to 34.5%.

After excelling in all growth metrics last month, nginx did not sweep the board this time. Although its computer growth continued to be strong – gaining 44,000 this month – it lost 11.2 million hostnames globally. The primary cause of this was more than 15 million sites switching from nginx to Apache. Most of these sites were previously hosted by Raksmart in China, but are now hosted by Data Foundry in the United States.

All four of the largest major vendors – Apache, nginx, Google and Microsoft – gained active sites this month. Apache made the largest gain of 2.1 million, but this was not enough to prevent its active sites share falling by 1.1 points to 42.7%. Nonetheless, Apache continues to lead with more than twice as many active sites as its closest competitor, nginx.

Cloudflare-nginx migration underway

For several years, every edge machine in Cloudflare's content delivery network (CDN) exhibited the Server: cloudflare-nginx HTTP response header. This reflected the fact that its custom software stack was based on nginx – but this month's survey saw a vast swathe of websites switching to a new Server: cloudflare header.

Although nginx remains part of Cloudflare's stack, it plays less of a role than it once did. The code that handles HTTP requests on Cloudflare's machines now goes far beyond the capabilities of nginx alone, and Cloudflare also hints that it will most likely end up writing its own caching software instead of using nginx.

In line with Cloudflare's reasoning, Netcraft's survey treats the new cloudflare server as a distinct product to nginx, and this has taken a chunk out of nginx's share in the top million sites. The new cloudflare server already accounts for 5.2% of the top million sites, and this share will undoubtedly increase next month.

The transition to the new cloudflare server banner started on 18 December 2017, and the February 2018 survey found nearly 60% of Cloudflare's sites using the new banner. It is likely that they will all use the new banner by next month's survey.

Other web server news

The developers of nginx have added support for HTTP/2 Server Push, which is likely to be made available in the next release. This feature allows web servers to send resources such as images and stylesheets before they are requested by the browser, which can make some webpages load faster.

After coming to prominence last month, DPS is now the 9th largest server by domains. It continues to be used almost exclusively by GoDaddy to host sites created with its Website Builder tool, and it is still being regularly updated. The current version in use at the time of writing is DPS 1.2.1, whereas a month ago it was 1.1.20.

OpenLiteSpeed 1.5.0 RC1 was released on 2 February, adding new Multi-Thread APIs and a module developer guide. This followed the 10 January release of 1.4.29, which is currently the latest stable version of the freely available open source server. Just over 2.5 million domains are currently using OpenLiteSpeed or the commercially available LiteSpeed Web Server product. Both products use the same LiteSpeed server banner and do not reveal version numbers.

Finally, NGINX Unit saw a few new releases since the last survey. NGINX Unit 0.4 was released on 15 January. This was the first release of the lightweight web application server to be compatible with DragonFly BSD, but it was mostly a bugfix release that eliminated some significant regressions in the previous version. NGINX Unit 0.5 was then released on 8 February, adding a Perl application module that allows it to run applications like Bugzilla; however, this release was not announced, as it contained a serious regression that could cause the main process to die. This was rectified in NGINX Unit 0.6, which was announced on 9 February.

Total number of websites

Web server market share

DeveloperJanuary 2018PercentFebruary 2018PercentChange
Microsoft575,026,64831.85%634,359,41934.50%2.65
Apache491,259,91827.21%504,701,56027.45%0.24
nginx458,386,42325.39%447,224,45624.32%-1.07
Google21,657,7961.20%22,022,6331.20%-0.00
Continue reading

Most Reliable Hosting Company Sites in January 2018

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 ServerStack Linux 0:00:00 0.000 0.126 0.060 0.118 0.118
2 vXtream Ltd Linux 0:00:00 0.006 0.135 0.053 0.107 0.107
3 XILO Communications Ltd. Linux 0:00:00 0.006 0.184 0.059 0.118 0.118
4 Memset Linux 0:00:00 0.006 0.146 0.067 0.133 0.133
5 Pair Networks FreeBSD 0:00:00 0.006 0.249 0.067 0.135 0.135
6 CWCS Linux 0:00:00 0.006 0.203 0.123 0.201 0.201
7 Bigstep Linux 0:00:00 0.011 0.155 0.059 0.121 0.121
8 www.arvixe.com Linux 0:00:00 0.011 0.145 0.081 0.167 0.167
9 ReliableServers.com Linux 0:00:00 0.017 0.312 0.009 0.160 0.160
10 Hyve Managed Hosting Linux 0:00:00 0.017 0.092 0.056 0.116 0.116

See full table

ServerStack had the most reliable hosting company website in January 2018, with the only site to respond successfully to each of Netcraft's requests. Its website appeared in the top 10 five times in 2017, and a total of 46 times since its inclusion in 2012. The hosting company, which operates three data centres in the US and Europe, provides managed hosting and was co-founded by brothers Moisey and Ben Uretsky, who later went on to start cloud provider DigitalOcean.

The next five hosting company sites each had only a single failed request, with the tie being broken by average connect time. Of these five, vXtream's Qube site had the quickest average connect time (53ms), earning it second place. vXtream, which recently acquired Qube, offers managed hosting, colocation, and cloud-based solutions from data centres in Zurich, London and New York. Its website, qubenet.net, has appeared in the top 10 a total of ten times in 2017. The sites belonging to XILO, Memset, pair Networks, and CWCS were ranked 3rd, 4th, 5th, and 6th according to their average connect times.

Hyve Managed Hosting's site continues its top 10 streak, now having been amongst the top 10 most reliable hosting company sites for 12 consecutive months. It claims 10th place this month, with three failed requests but a fast response time. Hyve's website has sustained a 100% uptime record since Netcraft began monitoring it in 2016.

Linux is the most popular operating system this month, being used by nine of the top ten hosting companies. FreeBSD also makes an appearance as Pair Networks's OS of choice.

Netcraft measures and makes available the response times of around thirty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

The hidden “well-known” phishing sites

Thousands of phishing sites have been finding homes in special hidden directories on compromised web servers.

In the past month alone, over 400 new phishing sites were found hosted within directories named /.well-known/; but rather than being created by fraudsters, these special directories are already present on millions of websites.

A Microsoft Excel Online phishing site hosted in the /.well-known/ directory on a compromised web server. The phishing site piggybacks on the trust instilled by the compromised site's existing SSL certificate, which has not been revoked.

A Microsoft Excel Online phishing site hosted in the /.well-known/ directory on a compromised web server. The phishing site piggybacks on the trust instilled by the compromised site's existing SSL certificate, which has not been revoked.

The /.well-known/ directory acts as a URI path prefix for "well-known locations", as defined by IETF RFC 5785, and provides a way for both humans and automated processes to discover a website's policies and other information.

One of the most common legitimate uses of the /.well-known/ directory is to prove control over a domain. When a secure website uses the Automatic Certificate Management Environment (ACME) protocol to manage its SSL certificate, the issuer will verify ownership by checking for a unique token in /.well-known/acme-challenge/ or /.well-known/pki-validation/. Consequently, most of the phishing attacks that make use of the /.well-known/ directory have been deployed on sites that support HTTPS, using certificates issued by ACME-driven certificate authorities like Let's Encrypt and cPanel.

Due to the success of Let's Encrypt and ACME, millions of websites now have a /.well-known/ directory in their web root, although many website administrators may be oblivious to its presence – particularly if they did not create the directory themselves. The directory can also easily be overlooked, as a bare ls command will treat files or directories that start with a "." as hidden. These factors make /.well-known/ an ideal place to smuggle phish onto a compromised web server.

Around 3% of these phishing sites are mistakenly deployed in a /well-known/ directory, without a leading "." character. This mistake could stem from file system name limitations if the phishing kit was created on a Windows computer. This screenshot shows a phishing kit that would be installed in a /well-known/ directory when unzipped.

Around 3% of these phishing sites are mistakenly deployed in a /well-known/ directory, without a leading "." character. This mistake could stem from file system name limitations if the phishing kit was created on a Windows computer. This screenshot shows a Bank of America phishing kit that would be installed in a /well-known/ directory when unzipped.

Shared hosting platforms are particularly vulnerable to misuse if the file system permissions on the /.well-known/ directories are overly permissive, allowing one website to place content on another customer's website. Some of the individual servers involved in these attacks were hosting "well-known" phishing sites for multiple hostnames, which lends weight to this hypothesis.

Other well-known URIs

In addition to pki-validation and acme-challenge, there are 30 other widely recognised well-known URI suffixes defined by the IETF, W3C and others. For example, the EFF came up with the dnt-policy.txt suffix, which allows websites to announce their compliance with user opt-outs from tracking. The EFF's own Do Not Track Compliance Policy can be viewed at https://www.eff.org/.well-known/dnt-policy.txt.

Where multiple resources may be required, the well-known URI suffix is a directory rather than a file. For example, the IETF's Enrollment over Secure Transport RFC defines a set of resources that can be found under the /.well-known/est/ path.

Despite there being several other well-known URI directory suffixes, only pki-validation and acme-challenge have been used to host recent phishing sites. In fact, more than half of the phishing sites found under the /.well-known/ directory were planted within the subdirectories created by ACME clients (i.e. /.well-known/pki-validation/ and /.well-known/acme-challenge/), possibly making them even less likely to be noticed by the website administrators.

An Alibaba phishing site. More than half of all "well-known" phishing sites are installed in the directories used by ACME clients.

An Alibaba phishing site. More than half of all "well-known" phishing sites are installed in the directories used by ACME clients, although this does not necessarily mean the ACME clients are to blame.

The possible route of compromise is not always apparent in the aforementioned cases, but if there are any glaring security misconfigurations, a proposed new well-known URI suffix, security.txt, could come in handy. By placing contact details and disclosure policies in /.well-known/security.txt, website administrators can make it safer and easier for security researchers to reach out and report any problems they find.

January 2018 Web Server Survey

In the January 2018 survey we received responses from 1,805,260,010 sites across 213,053,157 unique domain names and 7,228,005 web-facing computers. This reflects a gain of 214,000 computers, but only 183,000 domains. Overall hostname growth was 71 million, although the number of active sites fell slightly, by 311,000.

DPS powering GoDaddy's Website Builder

While the total number of domains across all web server vendors grew slightly, 1.5 million fewer domains used a Microsoft web server in the January 2018 survey. Its share of domain names has fallen by 0.74 points to 26.1%. Contributing to that loss were more than 985,000 unique domains hosted by GoDaddy, which are now using a lesser-known web server called DPS.

DPS (Data Protection Server) is now the 10th largest server by domains, and it is used exclusively by GoDaddy to host customer sites that have been created with its Website Builder tool. The DPS server appears to be frequently updated: sites using it currently return the Server: DPS/1.1.20 header, but these sites were using version 1.1.19 when the data was collected for the January 2018 survey. In the December 2017 survey, the sites were using version 1.1.16, and 1.1.10 in November 2017.

Cloud balancing with Pepyaka and F5 BIG-IP

Another lesser-known server, Pepyaka, also saw massive domain growth at a single hosting company this month. The Israeli web development platform Wix uses Pepyaka to host its customers' sites in the Amazon Web Services cloud, but many of these sites did not identify which server software they were using during the previous survey, causing a temporary absence. The number of domains using Pepyaka at AWS is now back up to more than 1.8 million, making it the 6th largest server by domains.

Nearly all of the Wix sites hosted at AWS use Pepyaka 1.11.3, which is likely based on the July 2016 mainline release of nginx 1.11.3; but it looks like Wix is in the process of rolling out an updated version: This month saw the appearance of 22 sites using Pepyaka 1.13.4, which most likely corresponds to the August 2017 mainline release of nginx 1.13.4.

Last month's temporary absence of Pepyaka could have been indicative of wider scale experimentation by Wix. Many of Wix's sites were served from machines that exhibited the TCP/IP characteristic of F5 BIG-IP, whereas this month, those sites are back to using Pepyaka running on Linux.

Wix has been a long-time user of nginx, and originally moved all user traffic to the commercial NGINX Plus product to future-proof its load balancing needs. The temporary appearance of F5 BIG-IP demonstrates that Wix may have been testing the waters with a different load balancing setup.

For most of its life, F5 BIG-IP has only been available on specialist hardware devices, such as BIG-IP appliances or VIPRION chassis; but F5's Virtual Editions make it possible to run BIG-IP software on commodity hardware in the cloud. F5 offers several BIG-IP Virtual Edition Amazon Machine Images (AMIs) in the AWS Marketplace, with pay-as-you-go licensing costs ranging from $0.33 to $4.40 per hour.

In May 2017, F5 also announced new public cloud solutions for Azure and Google Cloud, as well as a private cloud solution for the OpenStack cloud platform. This month's survey found more than 13 million domains being served from F5 BIG-IP devices, with Apache being the most commonly seen Server header.

Apache leads in most metrics, but nginx dominates in growth

Across the entire market, Apache remains in the lead with a 38.2% share of domains, but the ongoing trend makes it likely that both Apache and Microsoft could be overtaken by nginx in the next few years. nginx has continued to steadily increase its domain share, with a 0.21 point gain to 20.5% this month, while Apache has been experiencing a general decline of market share in recent years.

nginx's persistent growth has also manifested itself in every other metric this month, with it gaining the largest number of sites, active sites and web-facing computers, as well as increasing its presence amongst the top million sites. nginx is now used by 23.5% of all web-facing computers and 30.5% of the top million sites, but Apache still has the largest number of active sites, computers, domains and top-million sites.

The only metric graphed below in which Apache does not take the lead is hostnames, where Microsoft has a total of 575 million sites; but this metric is prone to fluctuations and is less indicative of market success. Microsoft has the second largest number of domains in the survey, but has been ranked third in web-facing computers since it was overtaken by nginx in October 2017.

While 1.5 million web-facing computers currently run Microsoft web server software, a slightly larger number – 1.8 million – run Windows operating systems. The bulk of the difference is made up of Windows computers that either run Apache or reverse-proxy traffic from backend Apache servers. The most commonly used Windows version is Windows Server 2008, followed by 2012 and then the aging, unsupported Windows Server 2003. Windows Server 2016 accounts for only 3.7% of all Windows web-facing computers at the moment, but it is steadily growing – this month, the number of Windows Server 2016 computers grew by 14% to 66,800.

Total number of websites

Web server market share

DeveloperDecember 2017PercentJanuary 2018PercentChange
Microsoft535,762,81330.89%575,026,64831.85%0.96
Apache446,418,87825.74%491,259,91827.21%1.47
nginx395,881,69022.83%458,386,42325.39%2.56
Google21,308,0691.23%21,657,7961.20%-0.03
Continue reading

Brazilian government providing warm waters for shoals of phish

Security holes in Brazilian government websites are still rife, with no fewer than eight different gov.br sites being compromised within the past week to host phishing attacks and hacking scripts. The situation does not seem to have improved much since two years ago, when we noticed a similar spate of phishing sites and malware hosted on gov.br domains, with evidence of some sites suffering repeated security compromises.

In one of this week's attacks, a gov.br domain was compromised to such an extent that the fraudsters were able to set up their own custom hostname, which was also configured to use HTTPS. The website, at account-verification-redirect-center.[redacted].gov.br, was then used to host a PayPal phishing site, which is still present at the time of writing.

Despite its rather dubious hostname, Let's Encrypt automatically issued an SSL certificate to account-verification-redirect-center.[redacted].gov.br earlier this week. Such foreseeable misuse evidently still does not prevent certificates being issued to phishing sites; but worse still, the fraudulent certificate has not yet been revoked.

The PayPal phishing site makes use of a ready-made phishing kit provided by SHADOW Z118. It includes several comprehensive "antibots" PHP scripts to avoid detection by search engines and enforcement agencies.

The PayPal phishing site makes use of a ready-made phishing kit provided by SHADOW Z118. It includes several comprehensive "antibots" PHP scripts to avoid detection by search engines and enforcement agencies.

To make matters worse, Netcraft found PHP shells on a few of the recently compromised gov.br sites. These backdoors provide fraudsters with almost complete access to the compromised web servers and make it easy for malware and phishing content to be uploaded at any time.

If the PHP shells are not removed, additional phishing sites are likely to appear on the affected sites, or they could even become infested with other PHP shells that will make the clean-up job much harder: If just one shell is overlooked, it can be used to replace all phishing content, malware and backdoors that the web server administrators had already deleted.

PayPal is still the most commonly targeted organisation in the latest attacks hosted by the Brazilian government, but other targets include Microsoft, Naver, Dropbox and the online dating site Match.com.

This OneDrive phishing site can steal Google, Outlook, AOL, Yahoo, Office 365, and other email credentials. The next form will steal the victim's phone number and backup email address.

This OneDrive phishing site can steal Google, Outlook, AOL, Yahoo, Office 365, and other email credentials. A second form steals the victim's phone number and backup email address.

Some of the phishing sites impersonate Microsoft's OneDrive service, using it as a convenient excuse to target Google, Outlook, AOL, Yahoo and other types of accounts from just a single attack. This particular attack could be rather harmful to businesses, as it gives victims the opportunity to log in with an Organizational Google Apps Account, which could result in the fraudster gaining access to sensitive company secrets.

Ironically, after the victim has been phished, he will be redirected to a PDF file on Google Drive entitled "The Business Owner's Guide to Wealth Management".

Ironically, after the victim has been phished, he will be redirected to a PDF file on Google Drive entitled "The Business Owner's Guide to Wealth Management".

All of the aforementioned phishing attacks were added to Netcraft's Phishing Site Feed, which is used by major web browsers and many leading anti-virus, content-filtering and web hosting companies.

Most Reliable Hosting Company Sites in December 2017

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 One.com Linux 0:00:00 0.000 0.165 0.037 0.110 0.110
2 Swishmail FreeBSD 0:00:00 0.000 0.121 0.055 0.110 0.156
3 Bigstep Linux 0:00:00 0.000 0.135 0.071 0.147 0.147
4 Multacom Linux 0:00:00 0.000 0.156 0.090 0.180 0.316
5 www.viawest.com Linux 0:00:00 0.005 0.249 0.008 0.189 0.189
6 New York Internet FreeBSD 0:00:00 0.005 0.278 0.022 0.047 0.047
7 ServerStack Linux 0:00:00 0.005 0.105 0.063 0.125 0.125
8 Pair Networks FreeBSD 0:00:00 0.005 0.225 0.066 0.134 0.134
9 Hyve Managed Hosting Linux 0:00:00 0.005 0.076 0.067 0.140 0.140
10 www.dinahosting.com Linux 0:00:00 0.005 0.186 0.090 0.181 0.181

See full table

One.com had the most reliable hosting company website in December 2017, successfully responding to all requests made by Netcraft. This is their third time claiming the top spot in 2017, with a total of nine appearances in the top ten in 2017. Founded in 2002, One.com has since established a global presence with offices in eleven countries around the world and services offered in fourteen languages.

Swishmail also responded to all of Netcraft's requests, but came in second due to a slightly slower average connect time. Swishmail made six appearances in the top ten in 2017, including a first place ranking in August. Swishmail offers business email solutions in a variety of plans, all of which come with a 30 day unconditional guarantee.

Hyve Managed Hosting had the most appearances in the top ten in 2017. They placed every month except January, and are currently on an impressive eleven-month streak. Hyve's site has had a 100% uptime record since Netcraft started monitoring it in 2016.

Linux is the most popular choice of operating system this month with seven of the top-ten hosting companies using it. FreeBSD is second most popular with three appearances.

Netcraft measures and makes available the response times of around thirty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.