February 2019 Web Server Survey

In the February 2019 survey we received responses from 1,477,803,927 sites, 229,586,773 unique domains, and 8,366,753 web-facing computers. This reflects a loss of 40.4 million sites, and gains of 979k domains, and 157k web-facing computers.

Microsoft experienced the largest gain in domains this month, with a net increase of just over one million. Despite several months of relatively small fluctuating gains and losses, the number of domains running Microsoft server software is on a general upward trend. Its total now stands at 59.3 million domains, up by just under 2.6 million (+4.6%) since this time last year. However, this strong domain growth was not reflected in any other metric this month – Microsoft gained only 478 web-facing computers, made losses both in active sites and within the top million sites, and suffered a sizable loss of 65 million hostnames.

On 20 February, Microsoft published a security advisory regarding a potential denial of service vulnerability in IIS. It can be exploited by sending specially crafted HTTP/2 requests to a Microsoft IIS web server, causing CPU usage to spike to 100% until the malicious connections are killed by IIS. Microsoft has addressed this issue in February's "non-security" update by providing the ability to define limits on the number of HTTP/2 settings parameters allowed over a connection.

nginx's growth in the domains metric was some way behind Microsoft's this month, with an increase of 622k. nginx's current total of 52.6 million domains represents a growth of over 8.6 million domains (+19.1%) since February 2018 – over three times that of Microsoft – however, its market share has remained just under 3 percentage points behind Microsoft's since April 2018. In terms of web-facing computers, nginx experienced the largest increase (+102k), continuing its steady gains in market share. It now holds a 29.0% share of the web-facing computer market with a total of 2.4 million.

The latest version of nginx (1.15.9 mainline) was released on 26 February, with some small changes including two new features and two bug fixes. Another product in the nginx family, NGINX Unit 1.7.1, was also released in February to address a security vulnerability in its router process. NGINX Unit is a lightweight web application server that can serve sandboxed Go, Perl, PHP, Python, Ruby and – soon – Java applications.

Apache is still losing domains, with a decrease of 216k this month, and 7.3 million domains over the past year. Apache is also gradually losing market share in terms of web-facing computers, despite an overall increase in the number of public web servers using it: this month, Apache gained 32.9k web-facing computers, taking its total up to 3.2 million, while competitor growth caused its market share to fall to 38.04%. The latest version, Apache 2.4.38, was released on 22 January. This release in the 2.4.x stable branch is regarded as the best available version of Apache, and includes three security fixes and multiple bug fixes.

Total number of websites

Web server market share

DeveloperJanuary 2019PercentFebruary 2019PercentChange
Microsoft485,175,88531.96%419,972,03928.42%-3.54
Apache323,398,86621.30%386,522,26626.16%4.85
nginx375,594,87524.74%374,544,60625.34%0.61
Google23,982,0331.58%23,912,2501.62%0.04
Continue reading

95% of HTTPS servers vulnerable to trivial MITM attacks

Only 1 in 20 HTTPS servers correctly implements HTTP Strict Transport Security, a widely-supported security feature that prevents visitors making unencrypted HTTP connections to a server.

The remaining 95% are therefore vulnerable to trivial connection hijacking attacks, which can be exploited to carry out effective phishing, pharming and man-in-the-middle attacks. An attacker can exploit these vulnerabilities whenever a user inadvertently tries to access a secure site via HTTP, and so the attacker does not even need to spoof a valid TLS certificate. Because no crypto-wizardry is required to hijack an HTTP connection, these attacks are far easier to carry out than those that target TLS, such as the recently announced DROWN attack.

Background

The growth of HTTPS has been a mostly positive step in the evolution of the internet, enabling encrypted communications between more users and websites than ever before. Many high profile sites now use HTTPS by default, and millions of TLS certificates are currently in use on the web. With companies like Let's Encrypt offering free certificates and automated management tools, it is also easier than ever to deploy an HTTPS website that will be trusted by all modern browsers.

The primary purpose of a TLS certificate is to allow a browser to verify that it is communicating with the correct website. For example, if https://www.example.com uses a valid TLS certificate, then a man-in-the-middle attacker would not be able to hijack a browser's connection to this site unless he is also able to obtain a valid certificate for that domain.

A man-in-the-middle attack like this is generally not possible if the customer uses HTTPS.

A man-in-the-middle attack like this is generally not possible if the initial request from the customer uses HTTPS.

It would be extremely difficult for the attacker to obtain a valid certificate for a domain he does not control, and using an invalid certificate would cause the victim's browser to display an appropriate warning message. Consequently, man-in-the-middle attacks against HTTPS services are hard to pull off, and often not very successful. However, there are plenty of realistic opportunities to use the unencrypted HTTP protocol to attack most HTTPS websites.

HTTP Strict Transport Security (HSTS)

Encrypted communications are an essential requirement for banks and other financial websites, but HTTPS alone is not sufficient to defend these sites against man-in-the-middle attacks. Astonishingly, many banking websites lurk amongst the 95% of HTTPS servers that lack a simple feature that renders them still vulnerable to pharming and man-in-the-middle attacks. This missing feature is HTTP Strict Transport Security (HSTS), and only 1 in 20 secure servers currently make use of it, even though it is supported by practically all modern browsers.

Each secure website that does not implement an HSTS policy can be attacked simply by hijacking an HTTP connection that is destined for it. This is a surprisingly feasible attack vector, as there are many ways in which a user can inadvertently end up connecting via HTTP instead of HTTPS.

Manually typed URLs often result in an initial insecure request, as most users do not explicitly type in the protocol string (http:// or https://). When no protocol is given, the browser will default to HTTP – unless there is an appropriate HSTS policy in force.

To improve accessibility, most secure websites also run an HTTP service to redirect users to the corresponding HTTPS site – but this makes them particularly prone to man-in-the-middle attacks if there is no HSTS policy in force. Not only would many users be accustomed to visiting the HTTP site first, but anyone else who visits the site via an old bookmark or search engine result might also initially access the site via an insecure HTTP address. Whenever this happens, the attacker can hijack the initial HTTP request and prevent the customer being redirected to the secure HTTPS website.

This type of attack can be automated with the sslstrip tool, which transparently hijacks HTTP traffic on a network and converts HTTPS links and redirects into HTTP. This type of exploit is sometimes regarded as a protocol downgrade attack, but strictly speaking, it is not: rather than downgrading the protocol, it simply prevents the HTTP protocol being upgraded to HTTPS.

NatWest's online banking website at www.nwolb.com lacks an HSTS policy and also offers an HTTP service to redirect its customers to the HTTPS site. This setup is vulnerable to the type of man-in-the-middle attack described above.

NatWest's online banking website at www.nwolb.com lacks an HSTS policy and also offers an HTTP service to redirect its customers to the HTTPS site. This setup is vulnerable to the type of man-in-the-middle attack described above.

Vulnerable sites can be attacked on a massive scale by compromising home routers or DNS servers to point the target hostname at a server that is controlled by the attacker (a so-called "pharming" attack). Some smaller scale attacks can be carried out very easily – for example, if an attacker sets up a rogue Wi-Fi access point to provide internet access to nearby victims, he can easily influence the results of their DNS lookups.

Even if a secure website uses HTTPS exclusively (i.e. with no HTTP service at all), then man-in-the-middle attacks are still possible. For example, if a victim manually types www.examplebank.com into his browser's address bar—without prefixing it with https://—the browser will attempt to make an unencrypted HTTP connection to http://www.examplebank.com, even if the genuine site does not run an HTTP service. If this hostname has been pharmed, or is otherwise subjected to a man-in-the-middle attack, the attacker can hijack the request nonetheless and eavesdrop the connection as it is relayed to the genuine secure site, or serve phishing content directly to the victim.

In short, failing to implement an HSTS policy on a secure website means attackers can carry out man-in-the-middle attacks without having to obtain a valid TLS certificate. Many victims would fall for these attacks, as they can be executed over an unencrypted HTTP connection, thus avoiding any of the browser's tell-tale warnings about invalid certificates.

Implementing HSTS: A simple one-liner

The trivial man-in-the-middle attacks described above can be thwarted by implementing an appropriate HSTS policy. A secure website can do this simply by setting a single HTTP header in its responses:

    Strict-Transport-Security: max-age=31536000;

This header can only be set over an HTTPS connection, and instructs compatible browsers to only access the site over HTTPS for the next year (31,536,000 seconds = 1 year). This is the most common max-age value, used by nearly half of all HTTPS servers. After this HSTS policy has been applied, even if a user manually prefixes the site's hostname with http://, the browser will ignore this and access the site over HTTPS instead.

The combination of HSTS and HTTPS therefore provides a good defence against pharming attacks, as the attacker will not be able to redirect and intercept plaintext HTTP traffic when a client obeys the HSTS policy, nor will he be able to present a valid TLS certificate for the site he is impersonating.

The attacker cannot even rely on a small proportion his victims unwisely ignoring the use of an invalid certificate, as browsers must regard this situation as a hard fail when an HSTS policy is in force. The browser will simply not let the victim access the site if it finds an invalid certificate, nor will it allow an exception to be added.

When Google Chrome encounters an invalid certificate for a site that has an effective HSTS policy, the victim is not allowed to bypass the browser's warning message or add an exception.

When Google Chrome encounters an invalid certificate for a site that has an effective HSTS policy, the victim is not allowed to bypass the browser's warning message or add an exception.

To prevent other types of attack, it is also wise to add the includeSubDomains directive to ensure that every possible subdomain of a site is protected by HSTS. This mitigates cookie injection and session fixation attacks that could be executed by impersonating an HTTP site on a non-existent subdomain such as foo.www.example.com, and using it to set a cookie which would be sent to the secure site at https://www.example.com. This directive can be enabled like so:

    Strict-Transport-Security: max-age=31536000; includeSubDomains

However, some thought is required before taking the carte blanche approach of including all subdomains in an HSTS policy. The website's administrators must ensure that every single one of its subdomains supports HTTPS for at least the duration specified by the max-age parameter, otherwise users of these subdomains risk being locked out.

Setting an HSTS policy will also protect first time visitors who habitually use search bars or search engines to reach their destination. For example, typing "paypal" into Google's HTTPS search engine will yield a link to https://www.paypal.com, because Google will always link to the HTTPS version of a website if an appropriate HSTS policy exists.

HSTS preloading

HSTS is clearly an important security feature, but there are several circumstances under which its benefits will not work. Because HSTS directives are delivered via an HTTP header (over an HTTPS connection), HSTS can only instruct a browser to only use HTTPS after the browser's first visit to a secure website.

Men-in-the-middle can therefore still carry out attacks against users who have:

  • Never before visited the site.
  • Recently reinstalled their operating system.
  • Recently reinstalled their browser.
  • Switched to a new browser.
  • Switched to a new device (e.g. mobile phone).
  • Deleted their browser's cache.
  • Not visited the site within the past year (or however long the max-age period lasts).

These vulnerabilities can be eliminated by using HSTS Preloading, which ensures that the site's HSTS policy is distributed to supported browsers before the customer's first visit.

Website administrators can use the form at https://hstspreload.appspot.com/ to request for domains to be included in the HSTS Preload list maintained by Google. Each site must have a valid certificate, redirect all HTTP traffic to HTTPS, and serve all subdomains over HTTPS. The HSTS header served from each site must specify a max-age of at least 18 weeks (10,886,400 seconds) and include the preload and includeSubdomains directives.

It can take several months for domains to be reviewed and propagated to the latest stable versions of Firefox, Safari, Internet Explorer, Edge and Chrome. When domains are added to the preload list, all users of these browsers will benefit from the security offered by HSTS, even if they have never visited the sites before.

Conclusions

HSTS is widely supported, but not widely implemented. Nearly all modern browsers obey HSTS policies, including Internet Explorer 11, Microsoft Edge, Firefox, Chrome, Safari and Opera – yet less than 5% of secure websites enable this important security feature.

Secure websites that do not use HSTS are trivial to attack if the attacker can hijack a victim's web traffic, but it is even easier to defeat such attacks by implementing an HSTS policy. This begs the question of why so few websites are using HSTS.

The HSTS specification (RFC 6797) was published in 2012, and so it can hardly be considered a new technology any more. Nonetheless, many website administrators might still be unaware of its existence, or may not yet feel ready to commit to running an HTTPS-only website. These are probably the most significant reasons for its low uptake.

Some website administrators have even disabled HSTS by explicitly setting a max-age of 0 seconds. This has the effect of switching off any previously established HSTS policies, but this backpedalling can only take proper effect if every client revisits the secure site after the max-age has been set to zero. When a site implements an HSTS policy, it is effectively committed to maintaining its HTTPS service for as long as the largest max-age it has ever specified, otherwise it risks denying access to infrequent visitors. Nearly 4% of all HTTPS servers that use the Strict-Transport-Security header currently set a max-age of zero, including Twitter's t.co URL-shortener.

Browser support for HSTS can also introduce some privacy concerns. By initiating requests to several distinct hostnames (some of which enable HSTS), a hostile webpage can establish a "supercookie" to uniquely identify the client browser during subsequent visits, even if the user deletes the browser's conventional cookies. The browser will remember which pattern of hostnames had HSTS enabled, thus allowing the supercookie to persist. However, this privacy concern only affects clients and does not serve as an excuse for websites to avoid implementing their own HSTS policies.

Implementing an HSTS policy is very simple and there are no practical downsides when a site already operates entirely over HTTPS. This makes it even more surprising to see many banks failing to use HSTS, especially on their online banking platforms. This demonstrates poor security practices where it matters the most, as these are likely to be primary targets of pharming attacks.

Netcraft offers a range of services that can be used to detect and defeat large-scale pharming attacks, and security testing services that identify man-in-the-middle vulnerabilities in web application and mobile apps. Contact security-sales@netcraft.com for more information.

January 2019 Web Server Survey

In the January 2019 survey we received responses from 1,518,207,412 sites, 228,607,903 unique domains, and 8,209,715 web-facing computers. This reflects a loss of 138 million sites, and gains of 768.9k domains, and 61.9k web-facing computers.

The vast majority of the loss of sites this month was seen for those using Microsoft web server software — dropping by 203 million sites (-29%). While much of this loss was concentrated at a single hosting provider and made up of automatically generated sites, Microsoft suffered losses in all metrics this month, albeit with much smaller reductions elsewhere. While Microsoft lost more than two hundred million sites in a single month, the net loss of domains was just 89k. Much of this drop can be attributed to IIS 6.0 and 7.5, while the latest version 10.0 saw a gain of 89k domains, and IIS 8.5 gained 164k.

nginx continues to gain market share in the web-facing computer metric, with the largest increase of 47k computers this month being more than double that of Apache's 21k gain. In the domains market nginx remains at somewhat of a plateau, with a loss of 144k domains taking its market share down to 22.7% — just 0.2pp higher than its share in April 2018. The largest factor this month — a single domain parking company, Bodis, moving all of its parked sites over to OpenResty. While the survey tracks these two products separately, OpenResty makes uses of the nginx core, integrating it with additional Lua-based modules.

Apache experienced a large gain in the domains metric this month with an increase of 245k. However, even with the losses seen for both Microsoft and nginx, it was not enough to boost Apache's market share which remains at 32.5%. The largest gain in domains came again for Cloudflare with an increase of 500k. Cloudflare predominantly uses its own server software originally based on nginx. OpenResty also saw greater gains than the market leader Apache, increasing by 388k with help from the movement of sites from nginx at Bodis.

Apache has released an update to httpd 2.4.38 to address an "important" remote DoS vulnerability, which allowed remote attackers to trigger an infinite loop through client-initiated renegotiation on servers using httpd 2.4.37 and OpenSSL version 1.1.1 or later. This is the first "important" level security issue in Apache HTTP Server since July 2017.

Total number of websites

Web server market share

DeveloperDecember 2018PercentJanuary 2019PercentChange
Microsoft688,039,05641.53%485,175,88531.96%-9.58
nginx358,383,16921.63%375,594,87524.74%3.11
Apache313,736,73918.94%323,398,86621.30%2.36
Google23,810,3361.44%23,982,0331.58%0.14
Continue reading

Google’s POODLE affects oodles

97% of SSL web servers are likely to be vulnerable to POODLE, a vulnerability that can be exploited in version 3 of the SSL protocol. POODLE, in common with BEAST, allows a man-in-the-middle attacker to extract secrets from SSL sessions by forcing the victim's browser into making many thousands of similar requests. As a result of the fallback behaviour in all major browsers, connections to web servers that support both SSL 3 and more modern versions of the protocol are also at risk.

The Secure Sockets Layer (SSL) protocol is used by millions of websites to protect confidential data in transit across the internet using strong cryptography. The protocol was designed by Netscape in the mid 1990s and was first released to the public as SSL 2 in February 1995. It was quickly replaced by SSL 3 in 1996 after serious security flaws were discovered. SSL 3 was replaced by the IETF-defined Transport Layer Security (TLS) version 1.0 in January 1999 with relatively few changes. Since TLS 1's release, TLS 1.1 and TLS 1.2 have succeeded it and should be used in its place wherever possible.

sslv3-vulnerable6

POODLE's bark may be worse than its bite

Unlike Heartbleed, POODLE can be used to attack client-server connections and is inherent to the protocol itself, rather than any one implementation such as OpenSSL or Microsoft's SChannel. In order to exploit it, an attacker must modify the victim's network traffic, know how the targeted secret information is structured (such as where a session cookie appears) and be able to force the victim into making a large number of requests.

Each SSL connection is split up into a number of chunks, known as SSL records. When using a block cipher, such as Triple DES in CBC mode, each block is mixed in with the next block and the record then padded to be a whole number of blocks long (8-bytes in the case of Triple DES). An attacker with network access can carefully manipulate the ordering of the cipher-blocks within a record to influence the decryption and exploit the padding oracle. If the attacker has been lucky (there's a 1 in 256 chance), she will have matched the correct value for the padding length in her manipulated record and correctly guessed the value of a single byte of the secret. This can be repeated to reveal the entire targeted secret.

SSL 3's padding is particularly easy to exploit as it relies on a single byte at the end of the padding, the padding length. Consequently an attacker must force the victim to make only 256×n requests for n bytes of secret to be revealed. TLS 1.0 changed this padding mechanism, requiring the padding bytes themselves to have a specific value making the attack far less likely to succeed.

The POODLE vulnerability makes session hijacking attacks against web applications reasonably feasible for a correctly-positioned attacker. For example, a typical 32-byte session cookie could be retrieved after eavesdropping just over 8,000 HTTPS requests using SSL 3. This could be achieved by tricking the victim into visiting a specially crafted web page which uses JavaScript to send the necessary requests.

Use of SSL v3

Within the top 1,000 SSL sites, SSL 3 remained very widely supported yesterday, with 97% of SSL sites accepting an SSL 3 handshake. CitiBank and Bank of America both support SSL 3 exclusively and presumably are vulnerable.

move-to-tls1

A number of SSL sites have already reacted to this vulnerability by disabling support for SSL 3, including CloudFlare and LinkedIn. On Tuesday 14th, the most common configuration within the top 1,000 SSL sites was to support SSL 3.0 all the way through to TLS 1.2, with almost two-thirds of popular sites taking this approach. One day later, this remains the most popular configuration; however, TLS 1.0 is now the minimum version for 11%.

Microsoft Internet Explorer 6 does not support TLS 1.0 or greater by default and may be the most notable victim of disabling SSL 3 internet-wide. Now 13 years old, IE6 was the default browser released with Windows Server 2003 and Windows XP in 2001 and will remain supported in Windows Server 2003 until July 2015. Despite its age and the end of Microsoft's support for Windows XP, IE6 remains popular, accounting for more 3.8% of web visits worldwide, and 12.5% in China. This vulnerability may ring the death knell for IE6 and Windows XP.

However, unless SSL 3 is completely disabled on the server side, a client supporting SSL 3 may still be vulnerable even if the server supports more recent versions of TLS. An attacker can take advantage of browser fallback behaviour to force otherwise secure connections to use SSL 3 in place of TLS version 1 or above.

SSL version negotiation

At the start of an SSL connection, servers and clients mutually agree upon a version of SSL/TLS to use for the remainder of the connection. The client's first message to the server includes its maximum supported version of the protocol, the server then compares the client's maximum version against its own maximum version to pick the highest mutually supported version.

While this mechanism protects against version downgrade attacks in theory, most browsers have an additional fallback mechanism that retries a connection attempt with successively lower version numbers until it succeeds in negotiating a connection or it reaches the lowest acceptable version. This additional fallback mechanism has proven necessary for practical interoperability with some TLS servers and corporate man-in-the-middle devices which, rather than gracefully downgrading when presented with a non-supported version of TLS, they instead terminate the connection prematurely.

An attacker with appropriate network access can exploit this behaviour to force a TLS connection to be downgraded by forging Handshake Alert messages. The browser will take the Handshake Alert message as a signal that the remote server (or some intermediate device) has version negotiation bugs and the browser will retry the connection with a lower maximum version in the initial Client Hello message.

handshake-alert

Operation of a forced downgrade to SSL 3 against a modern browser.

The fallback mechanism was previously not a security issue as it never results in the use of a protocol version that neither the client nor server will accept. However, those with clients that have not yet been updated to disable support for SSL 3 are relying on the server to have disabled SSL 3. What remains is a chicken and egg problem, where modern clients support SSL 3 in order to retain support for legacy servers, and modern servers retain support for SSL 3 for legacy clients.

There is, however, a proposed solution in the form of an indicator (an SCSV) in the fallback connection to inform compatible servers that this connection is a fallback and to reject the connection unless the fallback was expected. Google Chrome and Google's web sites already support this SCSV indicator.


Firefox 32

Chrome 40

IE 11

Opera 25

Safari 7.1
TLS 1.2 TLS 1.2 x 3 TLS 1.2 TLS 1.2 x 3 TLS 1.2
TLS 1.1 TLS 1.1 TLS 1.1
TLS 1.0 TLS 1.0 TLS 1.0 TLS 1.0 TLS 1.0
SSL 3.0 SSL 3.0 SSL 3.0 SSL 3.0 SSL 3.0

Comparison of browser fallback behaviour

We tested five major browsers with an attack based on the forged Handshake Alert method outlined above, and found that each browser has a variant of this fallback behaviour. Both Chrome and Opera try TLS 1.2 three times before trying to downgrade the maximum supported version, whereas the remainder immediately started downgrading. Curiously, Internet Explorer and Safari both skip TLS 1.1 and jump straight from TLS 1.2 to TLS 1.0.

Mitigation

Mitigation can take many forms: the fallback SCSV, disabling SSL 3 fallback, disabling SSL 3 in the client side, disabling SSL 3 in the server side, and disabling CBC cipher suites in SSL version 3. Each solution has its own problems, but the current trend is to disable SSL 3 entirely.

Disabling only the CBC cipher suites in SSL 3 leaves system administrators with a dilemma: RC4 is the only other practical choice and it has its fair share of problems making it an undesirable alternative. The SCSV requires support from both clients and servers, so may take some time before it is widely deployed enough to mitigate this vulnerability; it will also likely not be applied to legacy browsers such as IE 6.

Apache httpd can be configured to disable SSL 3 as follows:

SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2 -SSLv2 -SSLv3
Microsoft IIS and nginx can also be configured to avoid negotiating SSL version 3.

Firefox can be configured to disable support for SSL 3 by altering security.tls.version.min from 0 (SSL 3) to 1 (TLS 1) in about:config.

firefox-disable

Internet Explorer can also be configured to disable support using the Advanced tab in the Internet Options dialogue (found in the Control Panel). In a similar way, IE 6 users can also enable support for TLS 1.0.

internet-options-disable

Chrome can be configured to not use SSL 3 using a command line flag, --ssl-version-min=tls1.

Site Report

You can check which SSL sites are still using SSL 3 using the Netcraft Site Report:

Netcraft site report
URL:

The hidden “well-known” phishing sites

Thousands of phishing sites have been finding homes in special hidden directories on compromised web servers.

In the past month alone, over 400 new phishing sites were found hosted within directories named /.well-known/; but rather than being created by fraudsters, these special directories are already present on millions of websites.

A Microsoft Excel Online phishing site hosted in the /.well-known/ directory on a compromised web server. The phishing site piggybacks on the trust instilled by the compromised site's existing SSL certificate, which has not been revoked.

A Microsoft Excel Online phishing site hosted in the /.well-known/ directory on a compromised web server. The phishing site piggybacks on the trust instilled by the compromised site's existing SSL certificate, which has not been revoked.

The /.well-known/ directory acts as a URI path prefix for "well-known locations", as defined by IETF RFC 5785, and provides a way for both humans and automated processes to discover a website's policies and other information.

One of the most common legitimate uses of the /.well-known/ directory is to prove control over a domain. When a secure website uses the Automatic Certificate Management Environment (ACME) protocol to manage its SSL certificate, the issuer will verify ownership by checking for a unique token in /.well-known/acme-challenge/ or /.well-known/pki-validation/. Consequently, most of the phishing attacks that make use of the /.well-known/ directory have been deployed on sites that support HTTPS, using certificates issued by ACME-driven certificate authorities like Let's Encrypt and cPanel.

Due to the success of Let's Encrypt and ACME, millions of websites now have a /.well-known/ directory in their web root, although many website administrators may be oblivious to its presence – particularly if they did not create the directory themselves. The directory can also easily be overlooked, as a bare ls command will treat files or directories that start with a "." as hidden. These factors make /.well-known/ an ideal place to smuggle phish onto a compromised web server.

Around 3% of these phishing sites are mistakenly deployed in a /well-known/ directory, without a leading "." character. This mistake could stem from file system name limitations if the phishing kit was created on a Windows computer. This screenshot shows a phishing kit that would be installed in a /well-known/ directory when unzipped.

Around 3% of these phishing sites are mistakenly deployed in a /well-known/ directory, without a leading "." character. This mistake could stem from file system name limitations if the phishing kit was created on a Windows computer. This screenshot shows a Bank of America phishing kit that would be installed in a /well-known/ directory when unzipped.

Shared hosting platforms are particularly vulnerable to misuse if the file system permissions on the /.well-known/ directories are overly permissive, allowing one website to place content on another customer's website. Some of the individual servers involved in these attacks were hosting "well-known" phishing sites for multiple hostnames, which lends weight to this hypothesis.

Other well-known URIs

In addition to pki-validation and acme-challenge, there are 30 other widely recognised well-known URI suffixes defined by the IETF, W3C and others. For example, the EFF came up with the dnt-policy.txt suffix, which allows websites to announce their compliance with user opt-outs from tracking. The EFF's own Do Not Track Compliance Policy can be viewed at https://www.eff.org/.well-known/dnt-policy.txt.

Where multiple resources may be required, the well-known URI suffix is a directory rather than a file. For example, the IETF's Enrollment over Secure Transport RFC defines a set of resources that can be found under the /.well-known/est/ path.

Despite there being several other well-known URI directory suffixes, only pki-validation and acme-challenge have been used to host recent phishing sites. In fact, more than half of the phishing sites found under the /.well-known/ directory were planted within the subdirectories created by ACME clients (i.e. /.well-known/pki-validation/ and /.well-known/acme-challenge/), possibly making them even less likely to be noticed by the website administrators.

An Alibaba phishing site. More than half of all "well-known" phishing sites are installed in the directories used by ACME clients.

An Alibaba phishing site. More than half of all "well-known" phishing sites are installed in the directories used by ACME clients, although this does not necessarily mean the ACME clients are to blame.

The possible route of compromise is not always apparent in the aforementioned cases, but if there are any glaring security misconfigurations, a proposed new well-known URI suffix, security.txt, could come in handy. By placing contact details and disclosure policies in /.well-known/security.txt, website administrators can make it safer and easier for security researchers to reach out and report any problems they find.

Hackers still exploiting eBay’s stored XSS vulnerabilities in 2017

Fraudsters are still exploiting eBay's persistent cross-site scripting vulnerabilities to steal account credentials, years after a series of similar attacks took place. Worse still, many of the listings that exploited these vulnerabilities remained on eBay's website for more than a month before they were eventually removed.

All of the attacks stem from the fact that eBay allowed fraudsters to include malicious JavaScript in auction descriptions. Previous attacks exploited this vulnerability to place malicious redirect code on high-value vehicle listings, with the intention of stealing login credentials from other eBay members, whose accounts could then be used to list even more fraudulent vehicle listings.

But fraudsters are now using malicious scripts on a wide variety of lower-value items, including legitimate listings that had already been posted from reputable eBay accounts. Fraudsters have seemingly compromised these accounts and appended additional information to many of the members' existing listings – and this is where the malicious JavaScript is placed.

As can be seen below, the cybercriminals even used listings of dental tools to extract credentials from their victims, bypassing eBay's toothless listing policies in a similar way to the attacks that took place a few years ago.

A compromised listing for a dental tool from a Chinese seller as it appeared in eBay search results.

A compromised listing for a dental tool from a Chinese seller as it appeared in eBay search results.

Clicking on the above listing took the user to the following page, which included malicious JavaScript that had been injected by the fraudster:

The malicious listing is displayed for only a split second

The malicious listing is displayed for only a split second

But the malicious code in this listing executes as soon as the page has loaded, which causes it to be displayed for only a split second. In the blink of an eye — and without any further interaction — the victim is redirected to a spoofed login form:

In the blink of an eye, the victim is redirected to a very-convincing spoof login form.

In the blink of an eye, the victim is redirected to a very-convincing spoof login form.

Victims are unlikely to expect a phishing form to appear as a result of clicking on an eBay search result, and so the efficacy of these attacks is likely to be far greater than the average phishing scam. Allowing listings to include arbitrary JavaScript not only facilitates this type of fraud, but also allows fraudsters to capitalize on the trust instilled by the eBay website.

In this particular example, the malicious code injected by the attacker was obfuscated to make its purpose less apparent – possibly to get around any text-based content filters implemented by eBay. The obfuscated script is used to load a much larger JavaScript payload from an external location at user54631.vs.easily.co.uk/v.js (this script, which was hosted by Easily, has since been removed).

Lightly-obfuscated malicious JavaScript as it appeared in an eBay listing

Lightly-obfuscated malicious JavaScript as it appeared in an eBay listing

The externally-hosted script redirected victims to a data URI, which is another trick sometimes used by cybercriminals: The Base64-encoded address makes it difficult for victims to report such attacks, as by this point, the page is ostensibly not hosted anywhere.

When the victim submits his username and password, the credentials are transmitted to a script at daviddouglas.co.uk/session.php?/ws/eBayISAPI.dll?co_partnerId=2&siteid=3&UsingSSL=1 (which has also since been taken down). This PHP script receives the victim's credentials and then immediately redirects the victim to a page on the genuine eBay website, giving the impression that the listing that the victim originally attempted to visit is no longer available:

The victim is redirected to a non-existent listing after his credentials have been stolen.

The victim is redirected to a non-existent listing after his credentials have been stolen.

The victim may not realise it — as his browser never showed the address of any externally hosted websites — but at this point, his credentials will have already been stolen by the fraudster's PHP script.

The fraudsters behind these attacks can attempt to monetize these stolen credentials by selling them to other fraudsters, or use them to propagate malicious code into even more listings. In the dental tool example, malicious JavaScript was added to the listing on 8 December 2016, and remained there until late January 2017, giving the fraudster more than a month and a half to exploit the vulnerability.

The malicious script (not visible) was added on 8 December 2016, and eBay continued to serve it for a month and a half.

The malicious script (not visible) was added on 8 December 2016, and eBay continued to serve it for a month and a half.

The compromised seller account involved in the above attack had over a thousand of its listings infected with malicious JavaScript, many of which flew under eBay's radar for more than a month, despite having obvious malicious intentions. The only deterrent is eBay's JavaScript policy, which disallows the use of JavaScript redirects – but this is evidently not entirely effective, as it failed to prevent it being exploited for extended periods, and fraudsters will obviously not care about breaking policies that are not proactively enforced.

These latest listings were reported to Netcraft by "Jaco Bustero". Although this pseudonym is very similar to "Buster Jack" — who discovered a series of related scams in 2014 — they are, in fact, different people in the UK. Both hide behind pseudonyms because of valid concerns about their own safety – for instance, Buster Jack's efforts to combat vehicle fraud have earned him several death threats from the perpetrators of these crimes.

But fortunately, the end of script-based attacks may soon be in sight on eBay. In an effort to make its listings mobile-friendly, eBay plans to limit the use of active content (such as JavaScript) at some point in 2017, before eventually blocking it altogether. If this is implemented as a technical control (for example, by using iframes with Content Security Policy and sandbox restrictions), then such attacks should become impossible to carry out against modern browsers.

The most recent attacks have taken place over the past 12 months, after eBay had responded to 'previous reports' of JavaScript-based attacks, when it claimed not to have found any fraudulent activity stemming from these cross-site scripting vulnerabilities.

In some cases, it could be that eBay is simply unaware of the fraud it is facilitating. When one customer phoned eBay Trust & Safety to report these redirect attacks, the eBay handler was unable to see the redirection due to security settings on their internal systems. Consequently, reporting such vulnerabilities to eBay can prove frustrating, as well as fruitless: When Jaco posted a similar warning to the eBay Motors community forum, he claims his message was quickly deleted.

A year ago, we predicted that it would be difficult to prevent this type of fraud when listings are still able to include arbitrary JavaScript. With these recent attacks proving eBay's interim measures are still insufficient to prevent abuse, only technically-enforced controls on the execution of JavaScript will finally put a stop to this fraud.