April 2021 Web Server Survey
30th April, 2021
In the April 2021 survey we received responses from 1,212,139,815 sites across 264,469,666 unique domains and 10,939,637 web facing computers. This is an increase of 24,611,866 sites, 1,114,050 domains and 91,955 computers.
nginx gained the largest number of sites this month increasing by 12.5 million sites to 432,167,302. This also increases its market share to 35.65%, up 0.32 percentage points. Microsoft last month lost its place as third largest web server developer to OpenResty, this month it continued to lose sites (-3.6M) and market share, dropping 0.42pp to a market share of 5.54%. OpenResty gained 4.1 million sites and 0.21pp market share, while Apache gained 5.4 million sites but lost 0.08pp market share.
Looking at domains Apache and OpenResty gained the largest amount, with LiteSpeed and Oracle also seeing increases. OpenResty increased by 467k unique domains (+1.2%), this growth is primarily fueled by its increased use at both Google Cloud and Amazon AWS. Apache gained a similar amount with 426k domains (+0.6%), LiteSpeed gained 52k (+1.0%), and Oracle gained 27k (+3.6%). In contrast, nginx, Microsoft and Google each lost domains, 483k (-0.6%), 331k (-2.2%) and 233k (-9.2%) respectively.
In terms of web facing computers nginx gained the largest number with an increase of 59.6k (+1.6%) extending its market lead over Apache to 3.32 percentage points. Apache saw a smaller increase in the number of web facing computers making use of it (8k, +0.2%), this resulted in Apache losing 0.20pp market share. Microsoft lost both absolute numbers of computers, -15.3k (-1.1%), and market share, -0.25pp.
Focusing in on the million busiest sites Cloudflare gained 2,721 sites, the only significant gain this month, it is now used by 16.70% of the million busiest sites. nginx and Microsoft saw the largest losses, 1,978 and 1,806 sites respectively, with Apache dropping 608.
Other vendor and hosting news
- nginx version 1.20.0 was released on April 20th; this incorporates the features from the last year of development on the 1.19 mainline branch into a stable release that will not receive further feature updates. Prior to the release of version 1.20.0 the mainline 1.19 branch received a bug fix update on March 30th, and a minor update with features related to keepalives on April 13th.
- Apache Tomcat major versions 8, 9 and 10 were all updated on April 6th to pick up binaries built with OpenSSL 1.1.1k. Version 7 received a bug fix release on April 26th.
| Developer | March 2021 | Percent | April 2021 | Percent | Change |
|---|---|---|---|---|---|
| nginx | 419,637,923 | 35.34% | 432,167,302 | 35.65% | 0.32 |
| Apache | 308,509,042 | 25.98% | 313,948,741 | 25.90% | -0.08 |
| OpenResty | 77,819,490 | 6.55% | 81,935,391 | 6.76% | 0.21 |
| Microsoft | 70,826,342 | 5.96% | 67,182,740 | 5.54% | -0.42 |
Most Reliable Hosting Company Sites in March 2021
2nd April, 2021
| Rank | Performance Graph | OS | Outage hh:mm:ss |
Failed Req% |
DNS | Connect | First byte |
Total |
|---|---|---|---|---|---|---|---|---|
| 1 | GoDaddy.com Inc | Linux | 0:00:00 | 0.000 | 0.348 | 0.006 | 0.031 | 0.033 |
| 2 | Rackspace | Linux | 0:00:00 | 0.000 | 0.421 | 0.007 | 0.017 | 0.017 |
| 3 | CWCS Managed Hosting | Linux | 0:00:00 | 0.000 | 0.285 | 0.081 | 0.161 | 0.161 |
| 4 | Swishmail | Linux | 0:00:00 | 0.006 | 0.212 | 0.094 | 0.188 | 0.188 |
| 5 | New York Internet (NYI) | FreeBSD | 0:00:00 | 0.011 | 0.519 | 0.055 | 0.110 | 0.110 |
| 6 | Hyve Managed Hosting | Linux | 0:00:00 | 0.011 | 0.124 | 0.076 | 0.151 | 0.151 |
| 7 | ServerStack | Linux | 0:00:00 | 0.011 | 0.205 | 0.094 | 0.188 | 0.188 |
| 8 | Pair Networks | Linux | 0:00:00 | 0.017 | 0.336 | 0.110 | 0.219 | 0.219 |
| 9 | Multacom | Linux | 0:00:00 | 0.017 | 0.381 | 0.124 | 0.250 | 0.250 |
| 10 | www.flexential.com | Linux | 0:00:00 | 0.023 | 0.230 | 0.099 | 0.199 | 0.199 |
In March 2021 GoDaddy had the most reliable hosting company site, with no failed requests and the fastest average connection time amongst the top 10 of 6ms. GoDaddy provides services that allow customers to build their own web presence, which include hosting solutions, domain registration, and a website builder focused on ease of use. In February, GoDaddy acquired Poynt to accelerate its strategy to provide a complete suite of commerce and payment services.
The top three sites each had no failed requests and were separated by average connection time. Rackspace came in second place with an average connection time of 7ms, just 1ms slower than GoDaddy. Rackspace provides a wide variety of cloud services from its global network of over 50 locations in five continents. CWCS Managed Hosting wraps up the podium places, in third. CWCS supplies a wide range of hosting services from their ISO 27001, ISO 9001 accredited and Cyber Essentials certified UK data centres, powered by 100% renewable energy.
March 2021 Web Server Survey
29th March, 2021
In the March 2021 survey we received responses from 1,187,527,949 sites across 263,355,616 unique domains and 10,847,682 web-facing computers. This reflects a loss of 16,724,462 sites, but a gain of 313,561 domains and 81,076 computers.
nginx gained 3.7 million sites this month and holds 35.3% of the market with a total of 419.6 million sites. By contrast, Apache lost 8.5 million sites and accounts for just over a quarter of all sites with 308.5 million. Microsoft lost 9.6% (-7.5M) of its sites this month and ceded third place to OpenResty which in turn gained 1.2 million (+1.6%).
OpenResty is a web platform based on nginx which integrates Lua-based modules and has been the third-largest server by domains for several months. Despite this, it trails the competition in terms of web-facing computers, with only 105,800 computers compared to Microsoft’s 1.4 million.
nginx, Google, OpenResty, and LiteSpeed all acquired significant numbers of domains this month. nginx gained just over a million domains (+1.3%), while Google, OpenResty, and LiteSpeed gained 250,000 (+11.0%), 212,000 (+0.6%), and 68,600 (+1.3%). nginx’s domain growth came primarily from Freenom with 1.3 million domains using the server, while OpenResty’s growth came from its increased use on Google Cloud. Meanwhile, Apache and Microsoft lost -540,000 (-0.8%) and -585,000 (-3.7%) domains.
nginx and Apache both gained web-facing computers this month with nginx gaining a substantial 74,000 additional computers and a gain of 0.4 percentage points of market share and Apache gaining 3,300 - though losing 0.2 percentage points of market share due to nginx’s comparative higher growth. Other vendors also saw market share losses, with Microsoft losing 24,200 computers (-0.3 pp) and OpenResty losing just over 200 computers (-0.01 pp) despite its gains in sites and domains.
Looking at which web servers power the million busiest sites, only Cloudflare saw its count increase this month with a gain of 3,200 sites (+0.3 pp). Cloudflare’s growth came at the expense of nginx which lost the most with 1,570 fewer sites (-0.2 pp), along with Apache and Microsoft which both lost around 250 sites. The top spot remains hotly contested between Apache and nginx - Apache leads, but less than 2.5 percentage points separate the two.
Other vendor and hosting news
- A major fire at OVH’s Strasbourg datacenters resulted in around 3.6 million websites across 464,000 domains being taken offline at the start of March. While this was not captured by this month’s Web Server Survey, additional investigation by Netcraft found that nearly 20% of the IP addresses attributed to OVH stopped responding during the incident. One of the four data centers at the site, SBG2, was completely destroyed, and OVH is now provisioning thousands of new servers to replace those lost.
- Windows Server 2022 is now in preview and will be made generally available later in 2021. The features added in this release focus on adding new layers of security, integrating more tightly with Microsoft’s Azure platform, and improving Windows Containers. The current major release, Windows Server 2019, was made generally available nearly two and a half years ago in October 2018.
- nginx version 1.19.8 and njs version 0.5.2 were released on the 9th March. Both updates add minor new features and bug fixes.
- OpenLiteSpeed, the open-source variant of LiteSpeed Enterprise, received several updates through February and March, with versions 1.5.12, 1.6.20, and 1.7.9 containing primarily security updates and bug fixes.
- Apache Tomcat was updated to versions 9.0.44 and 10.0.4. Both updates include a variety of fixes, including improvements to asynchronous error handling.
| Developer | February 2021 | Percent | March 2021 | Percent | Change |
|---|---|---|---|---|---|
| nginx | 415,900,479 | 34.54% | 419,637,923 | 35.34% | 0.80 |
| Apache | 316,992,638 | 26.32% | 308,509,042 | 25.98% | -0.34 |
| OpenResty | 76,623,440 | 6.36% | 77,819,490 | 6.55% | 0.19 |
| Microsoft | 78,331,379 | 6.50% | 70,826,342 | 5.96% | -0.54 |
Flurry of reboots signal Exchange Server patching
15th March, 2021
Over 100,000 Outlook Web Access servers have been rebooted since Microsoft released security updates for the ProxyLogon remote code execution vulnerability. The subsequent flurry of reboot activity is likely indicative of many Microsoft Exchange servers being restarted after having security updates applied.
Last reboot dates of Outlook Web Access servers as at 14 March 2021.
Around half of all servers running Outlook Web Access (a service included with Microsoft Exchange Server) were rebooted in the five days after the emergency patch was released. Some of these have since been rebooted again, so will appear later in the above graph. Rebooted machines are likely to have been updated, but the absence of a reboot after 2 March does not necessarily indicate vulnerability. Anecdotally, most servers have requested a reboot after being updated, but some may only require services to be restarted – although administrators may have opted to reboot the servers anyway.
Microsoft’s original fixes can only be applied to servers that already have the latest cumulative updates of Exchange Server already installed; however, amidst mass exploitation of the vulnerabilities, Microsoft also released a set of security updates that can be applied to older and unsupported Exchange servers that do not—or cannot—have the latest cumulative updates installed.
The alternative security update path is intended as a temporary measure to protect vulnerable machines. Crucially, installing a later cumulative update that does not include the March 2021 security fixes will make the server vulnerable again, and any machine that uses the alternative security update path must be rebooted even if not prompted. In these cases, the servers will certainly not be protected until after the reboot.
Some of the more recent reboots may have been prompted by Microsoft’s 9 March “Patch Tuesday” collection of software updates, which also includes fixes for the remote code execution vulnerabilities in Microsoft Exchange.
On 6 March, four days after the original security updates were released, Netcraft found more than 99,000 Outlook Web Access servers were still running versions flagged as definitely vulnerable by Kevin Beaumont. However, applying Microsoft’s updates even in a timely fashion could have been like shutting the barn door after the horse had bolted, as more than 10% of all visited Outlook Web Access installations were already compromised with attackers' web shells installed. These provide the criminal with continued administrative access to the compromised servers after the security updates had been applied.
3.6 million websites taken offline after fire at OVH datacenters
10th March, 2021
Around 3.6 million websites across 464,000 distinct domains were taken offline after the major fire at an OVHcloud datacenter site in Strasbourg overnight.
More than 18% of the IP addresses attributed to OVH in Netcraft’s most recent Web Server Survey — which took place two weeks ago — were no longer responding at 06:00-07:15 UTC this morning.
A load monitoring graph of a server that was running at one of OVH’s Strasbourg datacenters.
It was last updated at 01:13 UTC today, indicating when it became inaccessible during the fire.
Thankfully, everybody is safe; but OVH said the fire in its SBG2 datacenter was not controllable and no data is likely to be recoverable. Part of its SBG1 datacenter has also been destroyed. Firefighters were protecting SBG3 throughout the night, and although there was no direct fire impact on SBG4, it was also unavailable due to the whole site being isolated. Consequently, all services in SGB1-4 have been offline.
Websites that went offline during the fire included online banks, webmail services, news sites, online shops selling PPE to protect against coronavirus, and several countries' government websites.
Examples of the latter included websites used by the Polish Financial Ombudsman; the Ivorian DGE; the French Plate-forme des achats de l’Etat; the Welsh Government’s Export Hub; and the UK Government’s Vehicle Certification Agency website, which got a new SSL certificate by 10am and is now back online with a UK hosting company.
Banking websites have also been hit by the fire.
Unsurprisingly for a French hosting company, the most affected country code top-level domain (ccTLD) is .fr, which had 184,000 knocked-out websites spread across 59,600 distinct domain names – these account for 1.9% of all .fr domains in the world. In comparison, there were only 24,100 .uk websites hosted in the affected datacenters, across just 8,700 unique domains. Most of the affected websites use the generic .com top-level domain, amounting to 880,000 websites across 180,000 domains.
Feeding Frenzy as criminal groups stake their claim on Outlook Web Access servers
8th March, 2021
This weekend, several days after Tuesday 2nd March when Microsoft released fixes for the ProxyLogon vulnerability, Netcraft found more than 99,000 unpatched Outlook Web Access servers accessible on the internet — of which several thousand have clear evidence of one or more web shells installed.
Outlook Web Access (OWA) provides remote access to on-premises Microsoft Exchange mailboxes. While a treasure trove of corporate email is a tempting enough target itself, it can also act as a jumping-off point for deeper network access. Vulnerable versions allow unfettered remote access to the mail server. Originally attributed to the Hafnium group, the variety of different web shells and file naming conventions found by Netcraft suggest that the shells belong to multiple groups who have been spurred into action since Microsoft’s announcement by the scale of the opportunity.
Vulnerable OWA installations as at 6 March 2021, based on passive observation of version numbers. Source: Netcraft survey.
Netcraft has established that at least 10% of all visited OWA installations are now infested with web shell backdoors that do not use randomised filenames, and so could plausibly be guessed by anybody. These implants allow continued administrative access to the server, long after the underlying vulnerability has been patched.
One of the backdoor scripts, disguised as an innocuous variable dump in a file named supp0rt.aspx. The active component of the backdoor is ‘hidden’ near the middle of the file.
All of the backdoors hide in plain sight on the web server’s file system but are disguised as benign scripts or information dumps in order to avoid detection. There are several different variants of the backdoor script, but all have the same common feature in that they pass the hacker’s commands to the JScript Eval command, allowing arbitrary code to be executed directly on the web server.
Most of the backdoor scripts accept the criminals' arbitrary commands via a specially named GET or POST parameter, while others require the commands to be Base64 encoded first, and some only accept them via a POST parameter.
Some variants of the backdoor script generate a runtime error if the secret variable name does not appear in the request. This makes it possible to detect their presence regardless.
Netcraft has also seen several different variants of these backdoor scripts being uploaded to individual websites, likely in an attempt to preserve unauthorised access to the compromised web server. Unless all of the backdoor scripts are found and removed, the hackers will still be able to get in and create more.
The web shell when viewed in a browser. There is no obvious indication of its malicious functionality.
While some of the backdoor variants are wildly different in appearance, they all function in a similar way and require the user to know a secret variable name before any commands can be executed on the server. The variable name effectively acts as a password and provides the only security mechanism to ensure that the backdoor can only be used by the person or persons responsible for uploading it.
However, some of the shells use easily guessable variable names like “o” and “orange”, which could plausibly allow them to be misused by other hackers if they can find the scripts and guess the correct variable names. This presents an even more dangerous situation where other fraudsters could then upload their own web shells to secure a foothold on the server. Such a situation could escalate quickly… new battlegrounds could erupt where rival fraudsters try to delete each others' web shells and upload more of their own in a race to secure access and decide how best to monetize their exploits, all long after the initial OWA vulnerabilities have been resolved.