Flurry of reboots signal Exchange Server patching
15th March, 2021
Over 100,000 Outlook Web Access servers have been rebooted since Microsoft released security updates for the ProxyLogon remote code execution vulnerability. The subsequent flurry of reboot activity is likely indicative of many Microsoft Exchange servers being restarted after having security updates applied.
Last reboot dates of Outlook Web Access servers as at 14 March 2021.
Around half of all servers running Outlook Web Access (a service included with Microsoft Exchange Server) were rebooted in the five days after the emergency patch was released. Some of these have since been rebooted again, so will appear later in the above graph. Rebooted machines are likely to have been updated, but the absence of a reboot after 2 March does not necessarily indicate vulnerability. Anecdotally, most servers have requested a reboot after being updated, but some may only require services to be restarted – although administrators may have opted to reboot the servers anyway.
Microsoft’s original fixes can only be applied to servers that already have the latest cumulative updates of Exchange Server already installed; however, amidst mass exploitation of the vulnerabilities, Microsoft also released a set of security updates that can be applied to older and unsupported Exchange servers that do not—or cannot—have the latest cumulative updates installed.
The alternative security update path is intended as a temporary measure to protect vulnerable machines. Crucially, installing a later cumulative update that does not include the March 2021 security fixes will make the server vulnerable again, and any machine that uses the alternative security update path must be rebooted even if not prompted. In these cases, the servers will certainly not be protected until after the reboot.
Some of the more recent reboots may have been prompted by Microsoft’s 9 March “Patch Tuesday” collection of software updates, which also includes fixes for the remote code execution vulnerabilities in Microsoft Exchange.
On 6 March, four days after the original security updates were released, Netcraft found more than 99,000 Outlook Web Access servers were still running versions flagged as definitely vulnerable by Kevin Beaumont. However, applying Microsoft’s updates even in a timely fashion could have been like shutting the barn door after the horse had bolted, as more than 10% of all visited Outlook Web Access installations were already compromised with attackers’ web shells installed. These provide the criminal with continued administrative access to the compromised servers after the security updates had been applied.
3.6 million websites taken offline after fire at OVH datacenters
10th March, 2021
Around 3.6 million websites across 464,000 distinct domains were taken offline after the major fire at an OVHcloud datacenter site in Strasbourg overnight.
More than 18% of the IP addresses attributed to OVH in Netcraft’s most recent Web Server Survey — which took place two weeks ago — were no longer responding at 06:00-07:15 UTC this morning.
A load monitoring graph of a server that was running at one of OVH’s Strasbourg datacenters.
It was last updated at 01:13 UTC today, indicating when it became inaccessible during the fire.
Thankfully, everybody is safe; but OVH said the fire in its SBG2 datacenter was not controllable and no data is likely to be recoverable. Part of its SBG1 datacenter has also been destroyed. Firefighters were protecting SBG3 throughout the night, and although there was no direct fire impact on SBG4, it was also unavailable due to the whole site being isolated. Consequently, all services in SGB1-4 have been offline.
Websites that went offline during the fire included online banks, webmail services, news sites, online shops selling PPE to protect against coronavirus, and several countries’ government websites.
Examples of the latter included websites used by the Polish Financial Ombudsman; the Ivorian DGE; the French Plate-forme des achats de l’Etat; the Welsh Government’s Export Hub; and the UK Government’s Vehicle Certification Agency website, which got a new SSL certificate by 10am and is now back online with a UK hosting company.
Banking websites have also been hit by the fire.
Unsurprisingly for a French hosting company, the most affected country code top-level domain (ccTLD) is .fr, which had 184,000 knocked-out websites spread across 59,600 distinct domain names – these account for 1.9% of all .fr domains in the world. In comparison, there were only 24,100 .uk websites hosted in the affected datacenters, across just 8,700 unique domains. Most of the affected websites use the generic .com top-level domain, amounting to 880,000 websites across 180,000 domains.
Feeding Frenzy as criminal groups stake their claim on Outlook Web Access servers
8th March, 2021
This weekend, several days after Tuesday 2nd March when Microsoft released fixes for the ProxyLogon vulnerability, Netcraft found more than 99,000 unpatched Outlook Web Access servers accessible on the internet — of which several thousand have clear evidence of one or more web shells installed.
Outlook Web Access (OWA) provides remote access to on-premises Microsoft Exchange mailboxes. While a treasure trove of corporate email is a tempting enough target itself, it can also act as a jumping-off point for deeper network access. Vulnerable versions allow unfettered remote access to the mail server. Originally attributed to the Hafnium group, the variety of different web shells and file naming conventions found by Netcraft suggest that the shells belong to multiple groups who have been spurred into action since Microsoft’s announcement by the scale of the opportunity.
Vulnerable OWA installations as at 6 March 2021, based on passive observation of version numbers. Source: Netcraft survey.
Netcraft has established that at least 10% of all visited OWA installations are now infested with web shell backdoors that do not use randomised filenames, and so could plausibly be guessed by anybody. These implants allow continued administrative access to the server, long after the underlying vulnerability has been patched.
One of the backdoor scripts, disguised as an innocuous variable dump in a file named supp0rt.aspx. The active component of the backdoor is ‘hidden’ near the middle of the file.
All of the backdoors hide in plain sight on the web server’s file system but are disguised as benign scripts or information dumps in order to avoid detection. There are several different variants of the backdoor script, but all have the same common feature in that they pass the hacker’s commands to the JScript Eval command, allowing arbitrary code to be executed directly on the web server.
Most of the backdoor scripts accept the criminals’ arbitrary commands via a specially named GET or POST parameter, while others require the commands to be Base64 encoded first, and some only accept them via a POST parameter.
Some variants of the backdoor script generate a runtime error if the secret variable name does not appear in the request. This makes it possible to detect their presence regardless.
Netcraft has also seen several different variants of these backdoor scripts being uploaded to individual websites, likely in an attempt to preserve unauthorised access to the compromised web server. Unless all of the backdoor scripts are found and removed, the hackers will still be able to get in and create more.
The web shell when viewed in a browser. There is no obvious indication of its malicious functionality.
While some of the backdoor variants are wildly different in appearance, they all function in a similar way and require the user to know a secret variable name before any commands can be executed on the server. The variable name effectively acts as a password and provides the only security mechanism to ensure that the backdoor can only be used by the person or persons responsible for uploading it.
However, some of the shells use easily guessable variable names like “o” and “orange”, which could plausibly allow them to be misused by other hackers if they can find the scripts and guess the correct variable names. This presents an even more dangerous situation where other fraudsters could then upload their own web shells to secure a foothold on the server. Such a situation could escalate quickly… new battlegrounds could erupt where rival fraudsters try to delete each others’ web shells and upload more of their own in a race to secure access and decide how best to monetize their exploits, all long after the initial OWA vulnerabilities have been resolved.
Netcraft Extension adds credential leak detection
28th August, 2020
The Netcraft Browser Extension now offers credential leak detection for extra protection against shopping site skimmers.
With brick-and-mortar shops around the world closed due to COVID-19, consumers turned to online businesses to fulfil their shopping needs. According to Adobe’s Digital Economy Index report, US online spending in June was $73 billion, up 76% from $42 billion last year. Even with restrictions lifted, research commissioned by Visa suggests that 74% of Britons who shopped online more often during the lockdown will continue to do so.
Now more than ever it is important to protect against JavaScript skimmers. These are snippets of malicious code which criminals upload to compromised shops. Unbeknownst to the store owner or the user, they transmit entered card details directly to the criminal. Unlike scams such as phishing, which can often be avoided by a vigilant internet user, skimmers are invisible to the human eye without a tool such as the Netcraft Extension to expose them.
Netcraft currently blocks over 6,000 shopping sites which contain skimmers, and even large companies such as British Airways, Ticketmaster and Puma have fallen prey to these attacks in the past.
The Netcraft Extension identifying and blocking a skimmer on an online shop
When you visit a shopping site, the Netcraft extension will evaluate all requests made by the web page. If a request is found to be sending credentials to a different domain, the extension will block the request to prevent your data from being stolen. A block screen will notify you about the request and provide information about the malicious behaviour that was detected. Only card number leaks are currently blocked, but other types of credentials may be enabled in future updates.
For example, if you check out using your credit card on exampleshoppingsite.com but your card details are sent to examplebadsite.com, the extension will block the request. This checking is done locally and securely in your browser – no sensitive information is sent to Netcraft.
The extension will also block pages which make requests to malicious domains that are part of JavaScript attacks.
In addition to shopping site skimmers, the Netcraft Extension also protects against other malicious JavaScript, phishing and fake shops, including those related to coronavirus. The extension is available for Chrome, Firefox, Opera and the new Microsoft Edge based on Chromium.
If you already have the Netcraft Extension installed, your browser will update it automatically.
Thousands of shop, bank, and government websites shut down by EV revocation
13th July, 2020
More than two thousand sites using Extended Validation certificates stopped working this weekend and remain inaccessible today (Monday), including those run by banks, governments, and online shops. The EV certificates used by these sites were revoked on Saturday, and have yet to be replaced. Most visitors using modern web browsers are completely locked out: this certificate error cannot be bypassed in Chrome, Firefox, Safari, or Microsoft Edge.
Chrome’s unbypassable revoked certificate interstitial on online.anz.com. ANZ is one of the"big four" Australian banks.
The first batch of revocations happened this weekend. While most of the certificates revoked on Saturday 11th July have been correctly replaced and reinstalled, many have not.
On Monday morning, Netcraft found 3,800 sites still using EV certificates issued by the affected sub-CAs. Of these 3,800, more than 2,300 were still using a revoked EV certificate, completely disabling the sites for users in modern browsers, which handle EV revocation more robustly than other types of certificate. The remainder are yet to be revoked.
Many organisations appear to have been caught unawares, continuing to use revoked EV certificates, including The State Bank of India, Rackspace, Authorize.net, ANZ Bank, and Telegram.
Authorize.net using a revoked EV certificate
The New Zealand government using a revoked EV certificate
Wirecard, the beleaguered German payment processor, briefly had its main site, www.wirecard.com, displaying a certificate warning early on Monday, but the certificate has since been replaced with a working non-EV certificate. There are still a number of Wirecard domains with revoked certificate warnings.
Protect against Shopping Site Skimmers and Fake Shops with the Netcraft Extension
12th July, 2020
The current coronavirus pandemic has resulted in the closure of many pubs, restaurants, and brick-and-mortar retail stores. Many purchases that would previously have been made in person now take place online. In research commissioned by Visa, 89% of Britons have shopped online since the UK’s lockdown restrictions began, with 31% buying items online for the first time during this period. This increase in online shopping activity benefits criminal groups in that: smaller businesses newly reliant on online transactions provide attackers with a stream of inadequately-defended shopping sites to exploit, and buyers are far more likely to be driven to these compromised shops or to fake shops compared to before the pandemic.
JavaScript skimmers run on compromised shopping sites. When shoppers enter their payment details, the skimmer secretly sends a copy to the attacker - potentially even if the customer does not complete the transaction. Even the most careful of users can be victims of these attacks, as they appear on compromised but otherwise well-intentioned shops with no visual indication of their presence.
Fake shops are another threat. Shoppers seeking bargains may unknowingly find themselves on a fake shop which claims to offers the products they want at a highly discounted price, but the victim will subsequently only receive counterfeit goods, no goods at all, or have the transaction aborted after entering credentials which is equivalent to a phishing attack.
Fake shops also take advantage of the pandemic by offering goods in high demand due to coronavirus, such as N95 masks. The FBI has released a Public Service Announcement about an increase in online shopping scams involving the sale of counterfeit healthcare products such as Personal Protective Equipment (PPE). To date, Netcraft has blocked over a thousand such coronavirus-themed fake shops, 80,000 other fake shops selling all sorts of counterfeit goods, and around 3,500 compromised shops hosting JavaScript skimmers.
The Netcraft browser extension and mobile apps provide protection against fake shops as well as legitimate shopping sites that have been compromised with JavaScript skimmers. When an extension or app user visits one of these dangerous shops, Netcraft will block access to the shop and alert them:
Visiting a fake shop without the Netcraft extension
Visiting a fake shop with the Netcraft extension