October 2018 Web Server Survey

In the October 2018 survey we received responses from 1,673,535,333 sites, 225,042,277 unique domains, and 7,927,795 web-facing computers. This reflects a gain of 31 million sites, 2.4 million domains and 142 thousand web-facing computers.

Nginx contributed the majority of this month's growth in terms of both domains and web-facing computers. Over 1.6 million additional domains were running nginx this month, taking its total up to 52 million. Combined with a loss of 444 thousand domains using Microsoft web server software, nginx now stands only 5 million domains behind Microsoft's second place spot. Apache remains in the lead, a further 18 million domains ahead. Despite regular gains in absolute numbers, including an increase of 200 thousand this month, Apache is continuing to experience a steady decline in market share. In terms of web-facing computers, nginx's gain of 70 thousand was almost twice that of Apache and Microsoft combined, which increased by 19 thousand and 16 thousand computers respectively.

Apache's active sites market share took a large hit this month, dropping over 8 percentage points down to 30%. The loss of 13 million active sites was concentrated at the Skyrock blogging platform, where sites stopped including a server banner for HTTP requests this month, meaning they are included in the "Other" category. However, the HTTPS version of these sites still appear to be running on Apache web servers returning 'Server: Apache' headers.

OpenResty gained 160 thousand domains (+6%), ratcheting up its total to 3 million, overtaking LiteSpeed at 2.9 million domains and becoming the fifth largest web server by number of domains. The OpenResty web platform integrates the standard nginx core with a LuaJIT compiler and a number of Lua libraries and third-party modules to run powerful web applications within the web server itself.

Netcraft's Web Server Survey continues to see strong growth among many of the new gTLDs. The .app and .icu gTLDs have shown the greatest gains among new TLDs launching in the last 6-months. The number of domains responding in this month's survey increased by 11 thousand since September for Google's .app domain, to over 236 thousand domains. Registrations for .icu domains (short for "I See You") increased by 10 thousand since September, a 30% increase to 42 thousand domains. This growth for .icu has also seen the number of active sites more than double, to just over 6 thousand.

Total number of websites

Web server market share

DeveloperSeptember 2018PercentOctober 2018PercentChange
Microsoft585,892,92735.67%656,395,38839.22%3.55
Apache384,521,18923.41%384,514,94422.98%-0.44
nginx319,330,26319.44%330,074,97419.72%0.28
Google22,210,0931.35%23,620,5551.41%0.06
Continue reading

Hackers still exploiting eBay’s stored XSS vulnerabilities in 2017

Fraudsters are still exploiting eBay's persistent cross-site scripting vulnerabilities to steal account credentials, years after a series of similar attacks took place. Worse still, many of the listings that exploited these vulnerabilities remained on eBay's website for more than a month before they were eventually removed.

All of the attacks stem from the fact that eBay allowed fraudsters to include malicious JavaScript in auction descriptions. Previous attacks exploited this vulnerability to place malicious redirect code on high-value vehicle listings, with the intention of stealing login credentials from other eBay members, whose accounts could then be used to list even more fraudulent vehicle listings.

But fraudsters are now using malicious scripts on a wide variety of lower-value items, including legitimate listings that had already been posted from reputable eBay accounts. Fraudsters have seemingly compromised these accounts and appended additional information to many of the members' existing listings – and this is where the malicious JavaScript is placed.

As can be seen below, the cybercriminals even used listings of dental tools to extract credentials from their victims, bypassing eBay's toothless listing policies in a similar way to the attacks that took place a few years ago.

A compromised listing for a dental tool from a Chinese seller as it appeared in eBay search results.

A compromised listing for a dental tool from a Chinese seller as it appeared in eBay search results.

Clicking on the above listing took the user to the following page, which included malicious JavaScript that had been injected by the fraudster:

The malicious listing is displayed for only a split second

The malicious listing is displayed for only a split second

But the malicious code in this listing executes as soon as the page has loaded, which causes it to be displayed for only a split second. In the blink of an eye — and without any further interaction — the victim is redirected to a spoofed login form:

In the blink of an eye, the victim is redirected to a very-convincing spoof login form.

In the blink of an eye, the victim is redirected to a very-convincing spoof login form.

Victims are unlikely to expect a phishing form to appear as a result of clicking on an eBay search result, and so the efficacy of these attacks is likely to be far greater than the average phishing scam. Allowing listings to include arbitrary JavaScript not only facilitates this type of fraud, but also allows fraudsters to capitalize on the trust instilled by the eBay website.

In this particular example, the malicious code injected by the attacker was obfuscated to make its purpose less apparent – possibly to get around any text-based content filters implemented by eBay. The obfuscated script is used to load a much larger JavaScript payload from an external location at user54631.vs.easily.co.uk/v.js (this script, which was hosted by Easily, has since been removed).

Lightly-obfuscated malicious JavaScript as it appeared in an eBay listing

Lightly-obfuscated malicious JavaScript as it appeared in an eBay listing

The externally-hosted script redirected victims to a data URI, which is another trick sometimes used by cybercriminals: The Base64-encoded address makes it difficult for victims to report such attacks, as by this point, the page is ostensibly not hosted anywhere.

When the victim submits his username and password, the credentials are transmitted to a script at daviddouglas.co.uk/session.php?/ws/eBayISAPI.dll?co_partnerId=2&siteid=3&UsingSSL=1 (which has also since been taken down). This PHP script receives the victim's credentials and then immediately redirects the victim to a page on the genuine eBay website, giving the impression that the listing that the victim originally attempted to visit is no longer available:

The victim is redirected to a non-existent listing after his credentials have been stolen.

The victim is redirected to a non-existent listing after his credentials have been stolen.

The victim may not realise it — as his browser never showed the address of any externally hosted websites — but at this point, his credentials will have already been stolen by the fraudster's PHP script.

The fraudsters behind these attacks can attempt to monetize these stolen credentials by selling them to other fraudsters, or use them to propagate malicious code into even more listings. In the dental tool example, malicious JavaScript was added to the listing on 8 December 2016, and remained there until late January 2017, giving the fraudster more than a month and a half to exploit the vulnerability.

The malicious script (not visible) was added on 8 December 2016, and eBay continued to serve it for a month and a half.

The malicious script (not visible) was added on 8 December 2016, and eBay continued to serve it for a month and a half.

The compromised seller account involved in the above attack had over a thousand of its listings infected with malicious JavaScript, many of which flew under eBay's radar for more than a month, despite having obvious malicious intentions. The only deterrent is eBay's JavaScript policy, which disallows the use of JavaScript redirects – but this is evidently not entirely effective, as it failed to prevent it being exploited for extended periods, and fraudsters will obviously not care about breaking policies that are not proactively enforced.

These latest listings were reported to Netcraft by "Jaco Bustero". Although this pseudonym is very similar to "Buster Jack" — who discovered a series of related scams in 2014 — they are, in fact, different people in the UK. Both hide behind pseudonyms because of valid concerns about their own safety – for instance, Buster Jack's efforts to combat vehicle fraud have earned him several death threats from the perpetrators of these crimes.

But fortunately, the end of script-based attacks may soon be in sight on eBay. In an effort to make its listings mobile-friendly, eBay plans to limit the use of active content (such as JavaScript) at some point in 2017, before eventually blocking it altogether. If this is implemented as a technical control (for example, by using iframes with Content Security Policy and sandbox restrictions), then such attacks should become impossible to carry out against modern browsers.

The most recent attacks have taken place over the past 12 months, after eBay had responded to 'previous reports' of JavaScript-based attacks, when it claimed not to have found any fraudulent activity stemming from these cross-site scripting vulnerabilities.

In some cases, it could be that eBay is simply unaware of the fraud it is facilitating. When one customer phoned eBay Trust & Safety to report these redirect attacks, the eBay handler was unable to see the redirection due to security settings on their internal systems. Consequently, reporting such vulnerabilities to eBay can prove frustrating, as well as fruitless: When Jaco posted a similar warning to the eBay Motors community forum, he claims his message was quickly deleted.

A year ago, we predicted that it would be difficult to prevent this type of fraud when listings are still able to include arbitrary JavaScript. With these recent attacks proving eBay's interim measures are still insufficient to prevent abuse, only technically-enforced controls on the execution of JavaScript will finally put a stop to this fraud.

HTTP Public Key Pinning: You’re doing it wrong!

HTTP Public Key Pinning (HPKP) is a security feature that can prevent fraudulently issued TLS certificates from being used to impersonate existing secure websites.

Our previous article detailed how this technology works, and looked at some of the sites that have dared to use this powerful but risky feature. Notably, very few sites are making use of HPKP: Only 0.09% of the certificates in Netcraft's March 2016 SSL Survey are served with HPKP headers, which equates to fewer than 4,100 certificates in total.

But more surprisingly, around a third of these sites are using the HPKP header incorrectly, which effectively disables HPKP. Consequently, the total number of certificates that are actually using HPKP is effectively less than 3,000.

Firefox's developer console reveals that this site has failed to include a backup pin, and so its HPKP policy is ignored by the browser.Failing to include a backup pin is the most common type of mistake made by sites that try to use HPKP.

Firefox's developer console reveals that this site has failed to include a backup pin, and so its HPKP policy is ignored by the browser.
Failing to include a backup pin is the most common type of mistake made by sites that try to use HPKP.

HPKP is the best way of protecting a site from being impersonated by mis-issued certificates, but it is easy for this protection to backfire with severe consequences. Fortunately, most misconfigurations simply mean that a site's HPKP policy will be ignored by browsers. The site's administrators might not realise it, but this situation is essentially the same as not using HPKP at all.

How can it go wrong?

Our previous article demonstrated a few high-profile websites that were using HPKP to varying degrees. However, plenty of other sites have bungled HPKP to the extent that it simply does not work.

Zero max-age

Every HPKP policy must specify a max-age directive, which suggests how long a browser should regard the website as a "Known Pinned Host". The most commonly used max-age value is 5184000 seconds (60 days). Nearly 1,200 servers use this value, while around 900 use 2592000 seconds (30 days).

But around 70 sites feature pointlessly short max-age values, such as 5 or 10 seconds. These durations are far too short to be effective, as a victim's browser would rapidly forget about these known pinned hosts.

Additionally, a few sites explicitly specify a max-age of zero along with their public key pins. These sites are therefore not protected by HPKP, and are in some cases needlessly sending this header to every client request. It is possible that they are desperately trying to remove a previously set HPKP policy, but this approach obviously cannot be relied upon to remove cached pins from browsers that do not visit the site in the meantime. These sites would therefore have to continue using a certificate chain that conforms to their previous HPKP policy, or run the risk of locking out a few stragglers.

One of the sites that sets a zero max-age is https://vodsmarket.com. Even if this max-age were to be increased, HPKP would still not be enabled because there is only one pinned public key:

Public-Key-Pins: pin-sha256="sbKjNAOqGTDfcyW1mBsy9IOtS2XS4AE+RJsm+LcR+mU="; max-age=0;

Another example can be seen on https://wondershift.biz, which pins two certificates' public keys. Again, even if the max-age were to be increased, this policy would still not take effect because there are no backup pins specified (both of the pinned keys appear in the site's certificate chain):

Public-Key-Pins: pin-sha256="L7mpy8M0VvQcWm7Yyx1LFK/+Ao280UZkz5U38Qk5G5g=";
    pin-sha256="EohwrK1N7rr3bRQphPj4j2cel+B2d0NNbM9PWHNDXpM=";
    includeSubDomains;
    max-age=0;
    report-uri="https://yahvehyireh.com/incoming/hpkp/index.php"

Wrong pin directives

Each pinned public key must be specified via a separate pin-sha256 directive, and each value must be a SHA256 hash; but more than 1% of servers that try to use HPKP fail to specify these pins correctly.

For example, the Department of Technology at Aichi University of Education exhibits the following header on https://www.auetech.aichi-edu.ac.jp:

Public-Key-Pins: YEnyhAxjrMAeVokI+23XQv1lzV3IBb3zs+BA2EUeLFI=";
    max-age=5184000;
    includeSubDomains

This header appears to include a single public key hash, but it omits the pin-sha256 directive entirely. No browser will make any sense of this attempted policy.

In another example, the Fast Forward Imaging Customer Interface at https://endor.ffwimaging.com does something very peculiar. It uses a pin-sha512 directive, which is not supported by the RFC – but in any case, the value it is set to is clearly not a SHA512 hash:

Public-Key-Pins: pin-sha512="base64+info1="; max-age=31536000; includeSubDomains

Some sites try to use SHA1 public key hashes, which are also unsupported:

Public-Key-Pins: pin-sha1='ewWxG0o6PsfOgu9uOCmZ0znd8h4='; max-age=2592000; includeSubdomains

This one uses pin-sha instead of pin-sha256:

Public-Key-Pins: pin-sha="xZ4wUjthUJ0YMBsdGg/bXHUjpEec5s+tHDNnNtdkwq8=";
    max-age=5184000; includeSubDomains

And this one refers to the algorithm "SHA245", which does not exist:

Public-Key-Pins: pin-sha245="pyCA+ftfVu/P+92tEhZWnVJ4BGO78XWwNhyynshV9C4=";
    max-age=31536000; includeSubDomains

The above example was most likely a typo, as is the following example, which specifies a ping-sha256 value:

Public-Key-Pins: ping-sha256="5C8kvU039KouVrl52D0eZSGf4Onjo4Khs8tmyTlV3nU=";
    max-age=2592000; includeSubDomains

These are careless mistakes, but it is notable that these types of mistake alone account for more than 1% of all certificates that set the Public-Key-Pins header. The net effect of these mistakes is that HPKP is not enabled on these sites.

Only one pinned public key

As we emphasised in our previous article, it is essential that a secure site should specify at least two public key pins when deploying HPKP. At least one of these should be a backup pin, so that the website can recover from losing control of its deployed certificate. If the website owner still possesses the private key for one of the backup certificates, the site can revert to using one of the other pinned public keys without any browsers refusing to connect.

But 25% of servers that use HPKP specify only one public key pin. This means that HPKP will not be enabled on the sites that use these certificates.

To prevent sites from inadvertently locking out all of their visitors, and to force the use of backup pins, browsers should only cache a site's pinned public keys if the Public-Key-Pins header contains two or more hashes. At least one of these must correspond to a certificate that is in the site's certificate chain, and at least one must be a backup pin (if a hash cannot be found in the certificate chain, then the browser will assume it is a backup pin without verifying its existence).

https://xcloud.zone is an example of a site that only sets one public key pin:

Public-Key-Pins: pin-sha256="DKvbzsurIZ5t5PvMaiEGfGF8dD2MA7aTUH9dbVtTN28=";
    max-age=2592000; includeSubDomains

This single pin corresponds to the subscriber certificate issued to xcloud.zone. Despite the 30-day max-age value, this lonely public key hash will never be cached by a browser. Consequently, HPKP is not enabled on this site, and the header might as well be missing entirely.

No pins at all

As well as the 1,000+ servers that only have one pinned public key, some HPKP headers neglect to specify any pins at all, and a few try to set values that are not actually hashes (which has the same effect as not setting any pins at all). For example, the Hide My Ass! forum at https://forum.hidemyass.com sets the following:

Public-Key-Pins: pin-sha256="<Subject Public Key Information (SPKI)>";
    max-age=2592000; includeSubDomains

The ProPublica SecureDrop site at https://securedrop.propublica.org also made a subtle mistake last month by forgetting to enclose its pinned public key hashes in double-quotes:

Public-Key-Pins: max-age=86400;
    pin-sha256=rhdxr9/utGWqudj8bNbG3sEcyMYn5wspiI5mZWkHE8A=
    pin-sha256=lT09gPUeQfbYrlxRtpsHrjDblj9Rpz+u7ajfCrg4qDM=

The HPKP RFC mandates that the Base64-encoded public key hashes must be quoted strings, so the above policy would not have worked. ProPublica has since fixed this problem, as well as adding a third pin to the header.

ProPublica is an independent newsroom that produces investigative journalism in the public interest. It provides a SecureDrop site to allow tips or documents to be submitted securely; however, until recently the HPKP policy on this site was ineffectual.

ProPublica is an independent newsroom that produces investigative journalism in the public interest. It provides a SecureDrop site to allow tips or documents to be submitted securely; however, until recently the HPKP policy on this site was ineffectual.

If companies that specialise in online privacy and secure anonymous filesharing are making these kinds of mistake on their own websites, it's not surprising that so many other websites are also getting it wrong.

At least two pins, but no backup pins

A valid HPKP policy must specify at least two pins, and at least one of these must be a backup pin. A browser will assume that a pin corresponds to a backup certificate if none of the certificates in the site's certificate chain correspond to that pin.

The Samba mailing list website fails to include any backup pins. Consequently, its HPKP policy is not enforced.

The Samba mailing list website fails to include any backup pins. Consequently, its HPKP policy is not enforced.

The Samba mailing lists site at https://lists.samba.org specifies two pinned public key hashes, but both of these appear in its certificate chain. Consequently, a browser will not apply this policy because there is no evidence of a backup pin. HPKP is effectively disabled on this site.

Incidentally, the Let's Encrypt Authority X1 cross-signed intermediate certificate has the most commonly pinned public key in our survey. More than 9% feature this in their set of pins, although it should never be pinned exclusively because Let's Encrypt is not guaranteed to always use their X1 certificate. Topically, just a few days ago, Let's Encrypt started to issue all certificates via its new Let's Encrypt Authority X3 intermediate certificate in order to be compatible with older Windows XP clients; but fortunately, the new X3 certificate uses the same keys as the X1 certificate, and so any site that had pinned the public key of the X1 certificate will continue to be accessible when it renews its subscriber certificate, without having to change its current HPKP policy.

The next most common pin belongs to the COMODO RSA Domain Validation Secure Server CA certificate. This pin is used by more than 6% of servers in our survey, all of which – despite the use of HPKP – could be vulnerable to man-in-the-middle attacks if Comodo were to be hacked again.

Pinning only the public keys of subscriber certificates would offer the best security against these kinds of attack, but it is fairly common to also pin the keys of root and intermediate certificates to reduce the risk of "bricking" a website in the event of a key loss. This approach is very common among Let's Encrypt customers, as the default letsencrypt client software generates a new key pair each time a certificate is renewed. If the public key of the subscriber certificate were to be pinned, the pinning would no longer be valid when it is renewed.

Setting HPKP policies over HTTP

Some sites set HPKP headers over unencrypted HTTP connections, which is also ineffectual. For example, the Internet Storm Center website at www.dshield.org sets the following header on its HTTP site:

Public-Key-Pins: pin-sha256="oBPvhtvElQwtqQAFCzmHX7iaOgvmPfYDRPEMP5zVMBQ=";
    pin-sha256="Ofki57ad70COg0ke3x80cbJ62Tt3c/f3skTimJdpnTw=";
    max-age=2592000; report-uri="https://isc.sans.org/badkey.html"

The Public Key Pinning Extension for HTTP RFC states that browsers must ignore HPKP headers that are received over non-secure transport, and so the above header has no effect other than to consume additional bandwidth.

2.2.2.  HTTP Request Type
  Pinned Hosts SHOULD NOT include the PKP header field in HTTP
  responses conveyed over non-secure transport.  UAs MUST ignore any
  PKP header received in an HTTP response conveyed over non-secure
  transport.

One very good reason for ignoring HPKP policies that are set over unencrypted connections is to prevent "hostile pinning" by man-in-the-middle attackers. If an attacker were to inject a set of pins that the site owner does not control—and if the browser were to blindly cache these values—he would be able to create a junk policy on behalf of that website. This would prevent clients from accessing the site for a long period, without the attacker having to maintain his position as a man-in-the-middle.

If a visitor instead browses to https://www.dshield.org (using HTTPS), an HSTS policy is applied which forces future requests to use HTTPS. The HTTPS site also sets an HPKP header which is then accepted and cached by compatible browsers. However, as the HTTP site does not automatically redirect to the HTTPS site, it is likely that many visitors will never benefit from these HSTS or HPKP polices, even though they are correctly implemented on the HTTPS site.

In another bizarre example, HPKP headers are set by the HTTP site at http://www.msvmgroup.com, even though there is no corresponding HTTPS website (it does accept connections on port 443, but does not present a subscriber certificate that is valid for this hostname).

Not quite got round to it yet...

A few sites that use the Public-Key-Pins header have not quite got around to implementing it yet, such as https://justamagic.ru, which sets the following value:

Public-Key-Pins: TODO

Using HPKP headers to broadcast skepticism

One security company's website – https://websec-test.com – uses the Public-Key-Pins header to express its own skepticisms over the usefulness of HPKP:

Public-Key-Pins: This is like the most useless header I have ever seen.
    Preventing MITM, c'mon, whoever can't trust his own network shouldn't
    enter sensitive data anywhere.

Violation reports that will never be received

The Public-Key-Pins header supports an optional report-uri directive. In the event of a pin validation failure, the user's browser should send a report to this address, in addition to blocking access to the site. These reports are obviously valuable, as they will usually be the first indication that something is wrong.

However, if the report-uri address uses HTTPS, and is also known pinned host, the browser must also carry out pinning checks on this address when the report is sent. This makes it foolish to specify a report-uri that uses the same hostname as the site that is using HPKP.

An example of this configuration blunder can be seen on https://yahvehyireh.com, which sets the following Public-Key-Pins header:

Public-Key-Pins: pin-sha256="y+PfuAS+Dx0OspfM9POCW/HRIqMqsa83jeXaOECu1Ns=";
    pin-sha256="klO23nT2ehFDXCfx3eHTDRESMz3asj1muO+4aIdjiuY=";
    pin-sha256="EohwrK1N7rr3bRQphPj4j2cel+B2d0NNbM9PWHNDXpM=";
    includeSubDomains; max-age=0;
     report-uri="https://yahvehyireh.com/incoming/hpkp/index.php"

This header instructs the browser to send pinning validation failure reports to https://yahvehyireh.com/incoming/hpkp/index.php. However, if there were to be a pinning validation failure on yahvehyireh.com, then the browser would be unable to send any reports because the report-uri itself would also fail the pinning checks by virtue of using the same hostname.

Incidentally, Chrome 46 introduced support for a newer header, Public-Key-Pins-Report-Only, which instructs the browser to perform identical pinning checks to those specified by the Public-Key-Pins header, but it will never block a request when no pinned keys are encountered; instead, the browser will send a report to a URL specified by a report-uri parameter, and the user will be allowed to continue browsing the site. This mechanism would make it safe for site administrators to test the deployment of HPKP on their sites, without inadvertently introducing a denial of service.

Summary

The proportion of secure servers that use HPKP headers is woefully low at only 0.09%, but to make matters worse, many of these few HPKP policies have been implemented incorrectly and do not work as intended.

Without delving into developer settings, browsers offer no visible indications that a site has an invalid HPKP policy, and so it is likely that many website administrators have no idea that their attempts at implementing HPKP have failed. Around a third of the sites that attempt to set an HPKP policy have got it wrong, and consequently behave as if there was no HPKP policy at all. Every response from these servers will include the unnecessary overhead of a header containing a policy that will ultimately be ignored by all browsers.

But there is still hope for the masses: A more viable alternative to HPKP might arise from an Internet-Draft entitled TLS Server Identity Pinning with Tickets. It proposes to extend TLS with opaque tickets, similar to those being used for TLS session resumption, as a way to pin a server's identity. This feature would allow a client to ensure that it is connecting to the right server, even in the presence of a fraudulently issued certificate, but has a significant advantage over HPKP in that no manual management actions would be required. If this draft comes to fruition, and is subsequently implemented by browsers and servers, this ticket-based approach to pinning could potentially see a greater uptake than HPKP has.

Netcraft offers a range of services that can be used to detect and defeat large-scale pharming attacks, and security testing services that identify man-in-the-middle vulnerabilities in web application and mobile apps. Contact security-sales@netcraft.com for more information.