3.6 million websites taken offline after fire at OVH datacenters
10th March, 2021
Around 3.6 million websites across 464,000 distinct domains were taken offline after the major fire at an OVHcloud datacenter site in Strasbourg overnight.
More than 18% of the IP addresses attributed to OVH in Netcraft’s most recent Web Server Survey — which took place two weeks ago — were no longer responding at 06:00-07:15 UTC this morning.
A load monitoring graph of a server that was running at one of OVH’s Strasbourg datacenters.
It was last updated at 01:13 UTC today, indicating when it became inaccessible during the fire.
Thankfully, everybody is safe; but OVH said the fire in its SBG2 datacenter was not controllable and no data is likely to be recoverable. Part of its SBG1 datacenter has also been destroyed. Firefighters were protecting SBG3 throughout the night, and although there was no direct fire impact on SBG4, it was also unavailable due to the whole site being isolated. Consequently, all services in SGB1-4 have been offline.
Websites that went offline during the fire included online banks, webmail services, news sites, online shops selling PPE to protect against coronavirus, and several countries' government websites.
Examples of the latter included websites used by the Polish Financial Ombudsman; the Ivorian DGE; the French Plate-forme des achats de l’Etat; the Welsh Government’s Export Hub; and the UK Government’s Vehicle Certification Agency website, which got a new SSL certificate by 10am and is now back online with a UK hosting company.
Banking websites have also been hit by the fire.
Unsurprisingly for a French hosting company, the most affected country code top-level domain (ccTLD) is .fr, which had 184,000 knocked-out websites spread across 59,600 distinct domain names – these account for 1.9% of all .fr domains in the world. In comparison, there were only 24,100 .uk websites hosted in the affected datacenters, across just 8,700 unique domains. Most of the affected websites use the generic .com top-level domain, amounting to 880,000 websites across 180,000 domains.
Feeding Frenzy as criminal groups stake their claim on Outlook Web Access servers
8th March, 2021
This weekend, several days after Tuesday 2nd March when Microsoft released fixes for the ProxyLogon vulnerability, Netcraft found more than 99,000 unpatched Outlook Web Access servers accessible on the internet — of which several thousand have clear evidence of one or more web shells installed.
Outlook Web Access (OWA) provides remote access to on-premises Microsoft Exchange mailboxes. While a treasure trove of corporate email is a tempting enough target itself, it can also act as a jumping-off point for deeper network access. Vulnerable versions allow unfettered remote access to the mail server. Originally attributed to the Hafnium group, the variety of different web shells and file naming conventions found by Netcraft suggest that the shells belong to multiple groups who have been spurred into action since Microsoft’s announcement by the scale of the opportunity.
Vulnerable OWA installations as at 6 March 2021, based on passive observation of version numbers. Source: Netcraft survey.
Netcraft has established that at least 10% of all visited OWA installations are now infested with web shell backdoors that do not use randomised filenames, and so could plausibly be guessed by anybody. These implants allow continued administrative access to the server, long after the underlying vulnerability has been patched.
One of the backdoor scripts, disguised as an innocuous variable dump in a file named supp0rt.aspx. The active component of the backdoor is ‘hidden’ near the middle of the file.
All of the backdoors hide in plain sight on the web server’s file system but are disguised as benign scripts or information dumps in order to avoid detection. There are several different variants of the backdoor script, but all have the same common feature in that they pass the hacker’s commands to the JScript Eval command, allowing arbitrary code to be executed directly on the web server.
Most of the backdoor scripts accept the criminals' arbitrary commands via a specially named GET or POST parameter, while others require the commands to be Base64 encoded first, and some only accept them via a POST parameter.
Some variants of the backdoor script generate a runtime error if the secret variable name does not appear in the request. This makes it possible to detect their presence regardless.
Netcraft has also seen several different variants of these backdoor scripts being uploaded to individual websites, likely in an attempt to preserve unauthorised access to the compromised web server. Unless all of the backdoor scripts are found and removed, the hackers will still be able to get in and create more.
The web shell when viewed in a browser. There is no obvious indication of its malicious functionality.
While some of the backdoor variants are wildly different in appearance, they all function in a similar way and require the user to know a secret variable name before any commands can be executed on the server. The variable name effectively acts as a password and provides the only security mechanism to ensure that the backdoor can only be used by the person or persons responsible for uploading it.
However, some of the shells use easily guessable variable names like “o” and “orange”, which could plausibly allow them to be misused by other hackers if they can find the scripts and guess the correct variable names. This presents an even more dangerous situation where other fraudsters could then upload their own web shells to secure a foothold on the server. Such a situation could escalate quickly… new battlegrounds could erupt where rival fraudsters try to delete each others' web shells and upload more of their own in a race to secure access and decide how best to monetize their exploits, all long after the initial OWA vulnerabilities have been resolved.
February 2021 Web Server Survey
26th February, 2021
In the February 2021 survey we received responses from 1,204,252,411 sites across 263,042,054 unique domains and 10,766,606 web-facing computers. This reflects a gain of 6,270,052 sites, 92,829 domains, and 116,789 computers.
nginx is top of the charts when it comes to total count of sites as well as number of unique domains and web-facing computers. 34.5% of all sites run on nginx, 30.4% of domains, and 35.0% of web-facing computers. Apache comes in at second place in these metrics, with a 26.3% market share of sites, a similar 26.4% share of domains, and 32.7% of web-facing computers.
In terms of domains, OpenResty and Cloudflare come in at third and fourth place to make up an additional 14.4% and 7.1% of the market respectively. OpenResty is a web application server that is built upon the technology of nginx, but, strictly speaking, is not an nginx fork. Cloudflare historically based their server stack around nginx, but transitioned towards using more in-house developed technologies over time. As of this month, these web server vendors are tracked individually in the monthly Web Server Survey charts.
Although nginx leads the wider market, Apache still has a small lead when it comes to the top one million busiest sites, with a 25.6% market share – 2.4pp ahead of nginx. Apache increased its share of the top million by 0.54pp in February. Although OpenResty takes a sizable chunk of the wider market, it is not nearly as common amongst the top million, taking only a 1.6% share. This disparity can be explained through GoDaddy’s extensive use of OpenResty for domain parking.
Apache also holds a more significant lead in terms of Netcraft’s active sites metric, which favours sites with unique content. Apache serves 25.5% of active sites, whereas nginx serves 19.8%. Google accounts for a reasonably large 9.9% share of active sites, owing to its popular Blogger service.
Microsoft’s server software market share remains in decline. Microsoft’s figures took a significant drop in 2020 in favour of OpenResty, and Microsoft now only has 6.5% (-1.0pp) of the site market and 6.0% (-0.3pp) of domains as of February 2021. OpenResty also looks set to overtake Microsoft as the third largest vendor in terms of sites and active sites.
Other vendor and hosting news
Nginx has pushed out its first product updates for 2021 – nginx version 1.19.7 and NGINX Unit 1.22.0. Lighttpd also released version 1.4.59 of its web server, which now enables HTTP version 2 by default.
| Developer | January 2021 | Percent | February 2021 | Percent | Change |
|---|---|---|---|---|---|
| nginx | 399,330,927 | 33.33% | 415,900,479 | 34.54% | 1.20 |
| Apache | 316,046,149 | 26.38% | 316,992,638 | 26.32% | -0.06 |
| Microsoft | 89,781,136 | 7.49% | 78,331,379 | 6.50% | -0.99 |
| OpenResty | 74,385,487 | 6.21% | 76,623,440 | 6.36% | 0.15 |
January 2021 Web Server Survey
28th January, 2021
In the January 2021 survey we received responses from 1,197,982,359 sites across 262,949,225 unique domains and 10,649,817 web-facing computers. This reflects a gain of 95,900 computers, but a loss of 30.13 million sites and 465,000 domains.
Apache, nginx and Microsoft all lost sites this month. Apache suffered the largest losses with 16.4 million fewer sites, a significant number of which came from a single hosting provider, Enzu. This was followed by nginx which lost 11.9 million sites and Microsoft which lost 7.8 million, both seeing losses across a number of hosting providers. nginx continues to lead in market share with 33.3% (-0.15 pp), ahead of Apache at 26.4% (-0.7 pp).
The number of domains powered by these web servers also fell this month. Microsoft observed the largest drop of 2.2 million domains, while nginx and Apache lost 903,000 and 303,000. This resulted in a small loss of market share for all three, the largest drop being seen by Microsoft which fell 0.8 percentage points to 6.3%. nginx dropped 0.3 percentage points of domain market share, though still powers 30% of all domains. The majority of the domains lost by these largest web server vendors remain in the survey, with individual hosting providers switching large numbers to other server vendors. One such movement involved around 1.2 million domains hosted by Cogeco Cable Canada that previously identified as nginx but now respond with a DOSarrest server banner.
Despite these losses, the total number of web-facing computers for both Apache and nginx grew this month. nginx gained 68,000 computers, increasing its market share by 0.3 percentage points to 34.7%, while Apache’s small gain of 6,200 resulted in a slight drop in market share to 33% (-0.2 pp).
Although both only have a small amount of web-facing computer market share, LiteSpeed and OpenResty both saw good relative growth this month. LiteSpeed gained 3,400 computers (+7.7% compared to last month) and OpenResty gained 5,700 computers (+6.8%).
The number of the top million sites powered by Apache, Microsoft and LiteSpeed all grew this month. Apache observed the highest growth with 4,043 sites, increasing its market share to 25% (+0.4 pp) and maintaining its lead over nginx which lost 105 sites this month. LiteSpeed gained 1,700 new sites, an increase of 9.4% over December.
Other vendor and hosting news
- Lighttpd 1.4.58 was released on 27 December, after the release of 1.4.57 on 17 December. These updates include a number of bug fixes.
- Microsoft Azure added support for Availability Zones in its South Central US datacenter region. Availability zones are multiple interconnected data centers within the same region, providing resiliency in the event of a data center outage. This addition means that Microsoft now has 15 public Azure regions with Availability Zones.
- A number of Google services suffered from a partial outage on 14 December, caused by a failure of Google’s authentication service. As well as affecting a number of Google services such as Gmail and YouTube, the issue also prevented users from authenticating to the Google Cloud Console and managing their infrastructure. Google Cloud Platform services themselves were largely unaffected by the outage.
| Developer | December 2020 | Percent | January 2021 | Percent | Change |
|---|---|---|---|---|---|
| nginx | 411,191,213 | 33.48% | 399,330,927 | 33.33% | -0.15 |
| Apache | 332,420,092 | 27.07% | 316,046,149 | 26.38% | -0.69 |
| Microsoft | 97,532,495 | 7.94% | 89,781,136 | 7.49% | -0.45 |
| 46,924,883 | 3.82% | 46,190,660 | 3.86% | 0.03 |
December 2020 Web Server Survey
22nd December, 2020
In the December 2020 survey we received responses from 1,228,111,563 sites across 263,413,876 unique domains and 10,553,965 web-facing computers. The reflects a loss of 1.84 million sites, 374,000 domains, and 10,600 computers.
Overall, nginx lost the largest number of sites this month (-3.15 million), while Apache gained most (3.04 million); yet in terms of active sites, nginx gained 411,000, while Apache gained only 100,000.
Microsoft, Apache and nginx each suffered losses in their total number of domains, although nginx's loss was small enough that its market share increased slightly. 30.3% of the world's domains are now powered by nginx, compared with 26.4% powered by Apache. Despite losses affecting each major webserver vendor, the causes were independent in each case; for example nginx’s 34,000 loss resulting from a drop of 387,000 domains at Freenom.
OpenResty is continuing to show strong growth, with GoDaddy's use of the web server for its parked domains. It now powers 71.3 million sites across 36.9 million domains and 84,680 web-facing computers.
The number of web-facing computers running nginx, Apache and Microsoft web server software also fell this month. The largest loss was 38,600 web-facing computers for nginx, which took its total down to 3.63 million and its share down by 0.33 percentage points to 34.4%, leaving it just over one percentage point ahead of Apache. Microsoft lost 14,700 computers, while Apache lost 5,820.
Improvements to the million busiest sites methodology has resulted in a number of market share movements. The biggest drop was seen for Apache, with its share down by 4.3 percentage points. Although it continues to lead this market with a share of 24.6%, nginx is now much closer behind with a 23.2% share of the top million, despite also experiencing a drop, and losing 2.0 percentage points. Cloudflare experienced the largest increase, up 1.4 percentage points to reach a 15.2% market share.
Other vendor and hosting news
- The first release candidate of Caddy 2.3 was announced on 12 December.
- nginx 1.19.6 mainline was released on 15 December. This version includes four bugfixes, including one for a bug that was introduced in the previous release, where a segmentation fault could have occurred when HTTPS was used.
- njs 0.5.0 was released on 1 December. This is the latest version of the JavaScript-based scripting language that can be used to extend nginx functionality.
- Apache Tomcat 7.0.107 was released on 23 November, followed by the releases of Tomcat 8.5.61, 9.0.41 and 10.0.0 (beta) on 8 December.
- OpenLiteSpeed 1.7.7 was released on 9 December.
- Microsoft has added new capabilities to its Azure Government cloud, expanded its Azure Government Secret cloud, and announced a new Azure Government Top Secret cloud to handle top secret classified data.
| Developer | November 2020 | Percent | December 2020 | Percent | Change |
|---|---|---|---|---|---|
| nginx | 414,338,895 | 33.69% | 411,191,213 | 33.48% | -0.21 |
| Apache | 329,384,399 | 26.78% | 332,420,092 | 27.07% | 0.29 |
| Microsoft | 97,283,716 | 7.91% | 97,532,495 | 7.94% | 0.03 |
| 45,597,737 | 3.71% | 46,924,883 | 3.82% | 0.11 |
November 2020 Web Server Survey
30th November, 2020
In the November 2020 survey we received responses from 1,229,948,224 sites across 263,787,870 unique domains and 10,564,577 web-facing computers. This reflects a gain of 24.2 million sites and 115,000 computers, but a loss of 310,000 domains.
Apache now powers 28.9% of the world's top million websites, where its presence has grown by 6,431 sites since last month. This has increased its lead over its closest competitor, nginx, which lost 2,563 sites from the top million and is now 3.72 percentage points behind Apache's leading share.
nginx continues to dominate in terms of web-facing computers, where its total rose by 88,700 to 3.67 million (+2.48%) computers and its share has grown by 0.47 percentage points to 34.7%. Although Apache gained 24,400 computers this month, its own share has now fallen to less than a third.
nginx also maintains its lead in two other metrics, with a total of 414 million sites (33.7% share) and 79.8 million domains (30.3%), while Apache still has the largest share of active sites (25.9%).
The number of domains powered by Microsoft web server suffered another noticeable fall this month, dropping by 473,000 to 19.1 million (-2.41%), reducing its share to 7.25%. Some of this decline is still being driven by GoDaddy's parked domains being moved to OpenResty web servers.
More than 36 million unique domains are served by OpenResty, making it the third most commonly used server by that metric, and putting it over 15 million domains ahead of Cloudflare. OpenResty is a scalable web platform based on NGINX and LuaJIT, which is a just-in-time compiler for the Lua language.
Some of the most commonly visited websites powered by OpenResty include Tumblr, Firefox Monitor, Basecamp and a few adult video sites. The 36.6 million domains powered by OpenResty are served from just 81,900 computers.
Other vendor and hosting news
- nginx 1.19.5 mainline was released on 24 November.
- NGINX Unit 1.21.0 was released on 19 November.
- Apache Tomcat 7.0.107, 8.5.60, 9.0.40 and 10.0.0.M10 were released on 17 November.
- OpenLiteSpeed 1.7.6 was released on 9 November, followed by OpenLiteSpeed 1.6.18 Stable on 24 November.
- Microsoft claims it will become the first major cloud provider to track hourly energy consumption and renewable energy matching in a commercial product using the Vattenfall 24/7 Matching solution for its new datacenter regions in Sweden, which will be available next year.
- DigitalOcean has introduced storage-optimized Droplets with NVMe SSDs, and reduced the price of its memory-optimized Droplets.
- Dozens of apps, websites and other online services including Coinbase, Flickr, Roku and The Washington Post were affected by an outage at Amazon's us-east-1 geographic region on 25 November.
| Developer | October 2020 | Percent | November 2020 | Percent | Change |
|---|---|---|---|---|---|
| nginx | 412,851,940 | 34.24% | 414,338,895 | 33.69% | -0.55 |
| Apache | 326,050,973 | 27.04% | 329,384,399 | 26.78% | -0.26 |
| Microsoft | 92,405,675 | 7.66% | 97,283,716 | 7.91% | 0.25 |
| 46,583,411 | 3.86% | 45,597,737 | 3.71% | -0.16 |