Most Reliable Hosting Company Sites in April 2016

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Datapipe Linux 0:00:00 0.000 0.160 0.012 0.024 0.031
2 Qube Managed Services Linux 0:00:00 0.000 0.153 0.058 0.117 0.117
3 CWCS Linux 0:00:00 0.000 0.189 0.070 0.142 0.143
4 Pair Networks FreeBSD 0:00:00 0.004 0.246 0.070 0.143 0.143
5 GoDaddy.com Inc Linux 0:00:00 0.009 0.257 0.010 0.024 0.025
6 XILO Communications Ltd. Linux 0:00:00 0.009 0.222 0.063 0.127 0.127
7 Kattare Internet Services Citrix Netscaler 0:00:00 0.009 0.517 0.114 0.228 0.228
8 LeaseWeb Linux 0:00:00 0.013 0.351 0.029 0.055 0.055
9 Hyve Managed Hosting Linux 0:00:00 0.013 0.222 0.060 0.120 0.120
10 Aspserveur Linux 0:00:00 0.013 0.343 0.077 0.326 0.471

See full table

Datapipe had the most reliable hosting company site in April, responding to all of Netcraft's requests. Datapipe's performance has seen it appear in the top ten every month so far in 2016, continuing a streak which has placed the company in the top ten 11 times in the past 12 months. Datapipe provides hosting services out of a number of data centres in Europe, Asia and North America. In April, Datapipe announced that it would be partnering with Singapore's largest electronics retailer to ensure the scalability of the latter's online infrastructure.

Qube had the second most reliable hosting company site. As with Datapipe, Qube's site responded to all of Netcraft's requests, but was fractionally slower to do so. Qube's performance is also consistent: the company has appeared in the top ten 9 times over the last 12 months.

CWCS achieved third place in April, also with a 100% response rate, albeit with an average connect time that was marginally slower than both Datapipe and Qube's. CWCS provides shared and managed hosting solutions, with data centre facilities in England and North America, and counts organisations such as KPMG and the University of York amongst its clients.

Linux is once again the most popular choice of operating system with hosting companies: eight out of the top ten companies in April hosted their websites on Linux machines. Kattare Internet Services and Pair Networks were the only two exceptions, using a Citrix Netscaler device and FreeBSD respectively.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

Hook, like and sinker: Facebook serves up its own phish

Fraudsters are abusing Facebook's app platform to carry out some remarkably convincing phishing attacks against Facebook users.

A phishing site displayed on the real Facebook website.

A phishing site displayed on the real Facebook website.

Masquerading as a Facebook Page Verification form, this phishing attack leverages Facebook's own trusted TLS certificate that is valid for all facebook.com subdomains. This makes the page appear legitimate, even to many seasoned internet users; however, the verification form is actually served via an iframe from an external site hosted by HostGator. The external website also uses HTTPS to serve the fraudulent content, so no warnings are displayed by the browser.

The phishing attack does not require the victim to be already logged in.

The phishing attack does not require the victim to be already logged in.

This phishing attack works regardless of whether the victim is already logged in, so there is little chance of a victim being suspicious of being asked to log in twice in immediate succession.

The source code of the phishing content reveals that it sends the stolen credentials directly to the fraudster's website.

The source code of the phishing content reveals that it sends the stolen credentials directly to the fraudster's website.

To win over anyone who remains slightly suspicious, the phishing site always pretends that the first set of submitted credentials were incorrect. A suspicious user might deliberately submit an incorrect username and password in order to test whether the form is legitimate, and the following error message could make them believe that the credentials really are being checked by Facebook.

The phishing site always pretends the first submitted credentials are incorrect.

The phishing site always pretends the first submitted credentials are incorrect. Note that it now also asks for the victim's date of birth.

Those who were slightly suspicious might then believe it is safe to enter their real username and password. Anyone else who had already entered the correct credentials would probably just think they had made a mistake and try again. After the second attempt, the phishing site will act as if the correct credentials had been submitted:

On the second attempt, the phishing site will ask the victim to wait up to 24 hours.

On the second attempt, the phishing site will ask the victim to wait up to 24 hours.

The final response indicates that the victim will have to wait up to 24 hours for their submission to be approved. Without instant access to the content they were trying to view, the victim will probably carry on doing something else until they receive the promised email notification.

But of course, this email will never arrive. By this point, the fraudster already has the victim's credentials and is just using this tactic to buy himself some time. He can either use the stolen Facebook credentials himself, or sell them to others who might monetize them by posting spam or trying to trick victims' friends into helping them out of trouble by transferring money. If more victims are required, then the compromised accounts could also be used to propagate the attack to thousands of other Facebook users.

Some of Facebook's security settings.

Some of Facebook's security settings.

However, Facebook does provide some features that could make these attacks harder to pull off. For example, if login alerts are enabled, the victim will be notified that their account has been logged into from a different location – this might at least make the victim aware that something untoward is going on. Although not enabled by default, users can completely thwart this particular attack by activating Facebook's login approvals feature, which requires a security code to be entered when logging in from unknown browsers. Only the victim will know this code, and so the fraudster will not be able to log in.

April 2016 Web Server Survey

In the April 2016 survey we received responses from 1,083,252,900 sites and 5,800,222 web-facing computers. This reflects a gain of nearly 80 million sites and 18,100 computers.

This is the largest number of sites the survey has ever seen, beating the previous maximum of 1,028,932,208 in October 2014. The number of web-facing computers is also at its largest, although this total has generally risen much more steadily than the number of sites.

Microsoft was the only major vendor to gain sites this month, and so it was solely responsible for this month's total reaching its highest value ever. Apache lost 33 million sites, while nginx and Google suffered much smaller losses. Many of the 124 million additional sites using Microsoft IIS are aimed at a Chinese audience. Several million are served from just a handful of IP addresses, using either IIS 6.0 or 7.5.

However, this proliferation of new Microsoft-powered websites is largely driven by automated processes. Many are "spam" sites that use link farming techniques to attract traffic. Although Microsoft's website count grew by a remarkable 38.9% in April, it lost 12,100 web-facing computers. High quality websites that attract genuine repeat traffic tend to have a very low number of sites per computer compared with the computers that are involved in link farming, which sometimes host millions of automatically-generated sites each. Corroborating this further, Microsoft suffered a loss of 341,000 active sites this month, taking its total down by 2.0%.

Meanwhile, nginx continued its relentless growth. It gained 19,500 web-facing computers this month (+2.4%), was the only major vendor to increase its active sites count, and increased its share within the top-million websites by 0.49 percentage points.

nginx is particularly prominent at Amazon and DigitalOcean, with the two hosting companies accounting for more than 25% of all nginx computers. In particular, nginx is the most commonly used server at DigitalOcean, being used by just under half of its web-facing droplets. At Amazon, despite its large share of all nginx computers, Apache is more than twice as common, with nginx only used on a quarter of EC2 instances.

Total number of websites

Web server market share

DeveloperMarch 2016PercentApril 2016PercentChange
Microsoft317,761,31831.65%441,470,89440.75%9.10
Apache325,285,18532.40%292,043,54826.96%-5.44
nginx143,464,29314.29%143,349,43913.23%-1.06
Google20,790,7672.07%20,597,6051.90%-0.17
Continue reading

Most Reliable Hosting Company Sites in March 2016

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Qube Managed Services Linux 0:00:00 0.000 0.148 0.058 0.118 0.118
2 Netcetera Linux 0:00:00 0.000 0.078 0.082 0.166 0.166
3 Datapipe Linux 0:00:00 0.008 0.158 0.012 0.024 0.031
4 Kattare Internet Services Citrix Netscaler 0:00:00 0.008 0.518 0.116 0.231 0.231
5 GoDaddy.com Inc Linux 0:00:00 0.013 0.244 0.006 0.018 0.018
6 ServerStack Linux 0:00:00 0.013 0.130 0.066 0.132 0.132
7 www.dinahosting.com Linux 0:00:00 0.013 0.267 0.081 0.163 0.163
8 Hyve Managed Hosting Linux 0:00:00 0.017 0.230 0.059 0.118 0.119
9 Pair Networks FreeBSD 0:00:00 0.017 0.239 0.070 0.142 0.142
10 Anexia Linux 0:00:00 0.017 0.197 0.085 0.181 0.181

See full table

Qube had the most reliable hosting company site in March, responding to all of Netcraft's requests. This continues Qube's strong performance from last year, where it placed in the top ten in eight months of 2015, and both January and February of 2016. Qube is based in London and offers a range of managed services, including private cloud hosting, from data centres in London, New York and Zurich.

In a very close second place, Netcetera also responded to every Netcraft request, but with a slightly longer average connection time. Netcetera owns and operates a carbon neutral data centre on the Isle of Man, and recently celebrated its 20th birthday, having been in the hosting business since 1996.

Datapipe takes third place in March, with two failed requests. Datapipe has an exceptionally consistent record for reliability, with its site maintaining 100% uptime over 10 years, and appearing in the top ten list on 42 occasions over the last 48 months.

Linux remains the most common choice of operating system amongst the most reliable hosting company sites, powering eight out of the top ten. Citrix Netscaler and FreeBSD also make an appearance, being employed by Kattare Internet Services and Pair Networks respectively.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

HTTP Public Key Pinning: You’re doing it wrong!

HTTP Public Key Pinning (HPKP) is a security feature that can prevent fraudulently issued TLS certificates from being used to impersonate existing secure websites.

Our previous article detailed how this technology works, and looked at some of the sites that have dared to use this powerful but risky feature. Notably, very few sites are making use of HPKP: Only 0.09% of the certificates in Netcraft's March 2016 SSL Survey are served with HPKP headers, which equates to fewer than 4,100 certificates in total.

But more surprisingly, around a third of these sites are using the HPKP header incorrectly, which effectively disables HPKP. Consequently, the total number of certificates that are actually using HPKP is effectively less than 3,000.

Firefox's developer console reveals that this site has failed to include a backup pin, and so its HPKP policy is ignored by the browser.Failing to include a backup pin is the most common type of mistake made by sites that try to use HPKP.

Firefox's developer console reveals that this site has failed to include a backup pin, and so its HPKP policy is ignored by the browser.
Failing to include a backup pin is the most common type of mistake made by sites that try to use HPKP.

HPKP is the best way of protecting a site from being impersonated by mis-issued certificates, but it is easy for this protection to backfire with severe consequences. Fortunately, most misconfigurations simply mean that a site's HPKP policy will be ignored by browsers. The site's administrators might not realise it, but this situation is essentially the same as not using HPKP at all.

How can it go wrong?

Our previous article demonstrated a few high-profile websites that were using HPKP to varying degrees. However, plenty of other sites have bungled HPKP to the extent that it simply does not work.

Zero max-age

Every HPKP policy must specify a max-age directive, which suggests how long a browser should regard the website as a "Known Pinned Host". The most commonly used max-age value is 5184000 seconds (60 days). Nearly 1,200 servers use this value, while around 900 use 2592000 seconds (30 days).

But around 70 sites feature pointlessly short max-age values, such as 5 or 10 seconds. These durations are far too short to be effective, as a victim's browser would rapidly forget about these known pinned hosts.

Additionally, a few sites explicitly specify a max-age of zero along with their public key pins. These sites are therefore not protected by HPKP, and are in some cases needlessly sending this header to every client request. It is possible that they are desperately trying to remove a previously set HPKP policy, but this approach obviously cannot be relied upon to remove cached pins from browsers that do not visit the site in the meantime. These sites would therefore have to continue using a certificate chain that conforms to their previous HPKP policy, or run the risk of locking out a few stragglers.

One of the sites that sets a zero max-age is https://vodsmarket.com. Even if this max-age were to be increased, HPKP would still not be enabled because there is only one pinned public key:

Public-Key-Pins: pin-sha256="sbKjNAOqGTDfcyW1mBsy9IOtS2XS4AE+RJsm+LcR+mU="; max-age=0;

Another example can be seen on https://wondershift.biz, which pins two certificates' public keys. Again, even if the max-age were to be increased, this policy would still not take effect because there are no backup pins specified (both of the pinned keys appear in the site's certificate chain):

Public-Key-Pins: pin-sha256="L7mpy8M0VvQcWm7Yyx1LFK/+Ao280UZkz5U38Qk5G5g=";
    pin-sha256="EohwrK1N7rr3bRQphPj4j2cel+B2d0NNbM9PWHNDXpM=";
    includeSubDomains;
    max-age=0;
    report-uri="https://yahvehyireh.com/incoming/hpkp/index.php"

Wrong pin directives

Each pinned public key must be specified via a separate pin-sha256 directive, and each value must be a SHA256 hash; but more than 1% of servers that try to use HPKP fail to specify these pins correctly.

For example, the Department of Technology at Aichi University of Education exhibits the following header on https://www.auetech.aichi-edu.ac.jp:

Public-Key-Pins: YEnyhAxjrMAeVokI+23XQv1lzV3IBb3zs+BA2EUeLFI=";
    max-age=5184000;
    includeSubDomains

This header appears to include a single public key hash, but it omits the pin-sha256 directive entirely. No browser will make any sense of this attempted policy.

In another example, the Fast Forward Imaging Customer Interface at https://endor.ffwimaging.com does something very peculiar. It uses a pin-sha512 directive, which is not supported by the RFC – but in any case, the value it is set to is clearly not a SHA512 hash:

Public-Key-Pins: pin-sha512="base64+info1="; max-age=31536000; includeSubDomains

Some sites try to use SHA1 public key hashes, which are also unsupported:

Public-Key-Pins: pin-sha1='ewWxG0o6PsfOgu9uOCmZ0znd8h4='; max-age=2592000; includeSubdomains

This one uses pin-sha instead of pin-sha256:

Public-Key-Pins: pin-sha="xZ4wUjthUJ0YMBsdGg/bXHUjpEec5s+tHDNnNtdkwq8=";
    max-age=5184000; includeSubDomains

And this one refers to the algorithm "SHA245", which does not exist:

Public-Key-Pins: pin-sha245="pyCA+ftfVu/P+92tEhZWnVJ4BGO78XWwNhyynshV9C4=";
    max-age=31536000; includeSubDomains

The above example was most likely a typo, as is the following example, which specifies a ping-sha256 value:

Public-Key-Pins: ping-sha256="5C8kvU039KouVrl52D0eZSGf4Onjo4Khs8tmyTlV3nU=";
    max-age=2592000; includeSubDomains

These are careless mistakes, but it is notable that these types of mistake alone account for more than 1% of all certificates that set the Public-Key-Pins header. The net effect of these mistakes is that HPKP is not enabled on these sites.

Only one pinned public key

As we emphasised in our previous article, it is essential that a secure site should specify at least two public key pins when deploying HPKP. At least one of these should be a backup pin, so that the website can recover from losing control of its deployed certificate. If the website owner still possesses the private key for one of the backup certificates, the site can revert to using one of the other pinned public keys without any browsers refusing to connect.

But 25% of servers that use HPKP specify only one public key pin. This means that HPKP will not be enabled on the sites that use these certificates.

To prevent sites from inadvertently locking out all of their visitors, and to force the use of backup pins, browsers should only cache a site's pinned public keys if the Public-Key-Pins header contains two or more hashes. At least one of these must correspond to a certificate that is in the site's certificate chain, and at least one must be a backup pin (if a hash cannot be found in the certificate chain, then the browser will assume it is a backup pin without verifying its existence).

https://xcloud.zone is an example of a site that only sets one public key pin:

Public-Key-Pins: pin-sha256="DKvbzsurIZ5t5PvMaiEGfGF8dD2MA7aTUH9dbVtTN28=";
    max-age=2592000; includeSubDomains

This single pin corresponds to the subscriber certificate issued to xcloud.zone. Despite the 30-day max-age value, this lonely public key hash will never be cached by a browser. Consequently, HPKP is not enabled on this site, and the header might as well be missing entirely.

No pins at all

As well as the 1,000+ servers that only have one pinned public key, some HPKP headers neglect to specify any pins at all, and a few try to set values that are not actually hashes (which has the same effect as not setting any pins at all). For example, the Hide My Ass! forum at https://forum.hidemyass.com sets the following:

Public-Key-Pins: pin-sha256="<Subject Public Key Information (SPKI)>";
    max-age=2592000; includeSubDomains

The ProPublica SecureDrop site at https://securedrop.propublica.org also made a subtle mistake last month by forgetting to enclose its pinned public key hashes in double-quotes:

Public-Key-Pins: max-age=86400;
    pin-sha256=rhdxr9/utGWqudj8bNbG3sEcyMYn5wspiI5mZWkHE8A=
    pin-sha256=lT09gPUeQfbYrlxRtpsHrjDblj9Rpz+u7ajfCrg4qDM=

The HPKP RFC mandates that the Base64-encoded public key hashes must be quoted strings, so the above policy would not have worked. ProPublica has since fixed this problem, as well as adding a third pin to the header.

ProPublica is an independent newsroom that produces investigative journalism in the public interest. It provides a SecureDrop site to allow tips or documents to be submitted securely; however, until recently the HPKP policy on this site was ineffectual.

ProPublica is an independent newsroom that produces investigative journalism in the public interest. It provides a SecureDrop site to allow tips or documents to be submitted securely; however, until recently the HPKP policy on this site was ineffectual.

If companies that specialise in online privacy and secure anonymous filesharing are making these kinds of mistake on their own websites, it's not surprising that so many other websites are also getting it wrong.

At least two pins, but no backup pins

A valid HPKP policy must specify at least two pins, and at least one of these must be a backup pin. A browser will assume that a pin corresponds to a backup certificate if none of the certificates in the site's certificate chain correspond to that pin.

The Samba mailing list website fails to include any backup pins. Consequently, its HPKP policy is not enforced.

The Samba mailing list website fails to include any backup pins. Consequently, its HPKP policy is not enforced.

The Samba mailing lists site at https://lists.samba.org specifies two pinned public key hashes, but both of these appear in its certificate chain. Consequently, a browser will not apply this policy because there is no evidence of a backup pin. HPKP is effectively disabled on this site.

Incidentally, the Let's Encrypt Authority X1 cross-signed intermediate certificate has the most commonly pinned public key in our survey. More than 9% feature this in their set of pins, although it should never be pinned exclusively because Let's Encrypt is not guaranteed to always use their X1 certificate. Topically, just a few days ago, Let's Encrypt started to issue all certificates via its new Let's Encrypt Authority X3 intermediate certificate in order to be compatible with older Windows XP clients; but fortunately, the new X3 certificate uses the same keys as the X1 certificate, and so any site that had pinned the public key of the X1 certificate will continue to be accessible when it renews its subscriber certificate, without having to change its current HPKP policy.

The next most common pin belongs to the COMODO RSA Domain Validation Secure Server CA certificate. This pin is used by more than 6% of servers in our survey, all of which – despite the use of HPKP – could be vulnerable to man-in-the-middle attacks if Comodo were to be hacked again.

Pinning only the public keys of subscriber certificates would offer the best security against these kinds of attack, but it is fairly common to also pin the keys of root and intermediate certificates to reduce the risk of "bricking" a website in the event of a key loss. This approach is very common among Let's Encrypt customers, as the default letsencrypt client software generates a new key pair each time a certificate is renewed. If the public key of the subscriber certificate were to be pinned, the pinning would no longer be valid when it is renewed.

Setting HPKP policies over HTTP

Some sites set HPKP headers over unencrypted HTTP connections, which is also ineffectual. For example, the Internet Storm Center website at www.dshield.org sets the following header on its HTTP site:

Public-Key-Pins: pin-sha256="oBPvhtvElQwtqQAFCzmHX7iaOgvmPfYDRPEMP5zVMBQ=";
    pin-sha256="Ofki57ad70COg0ke3x80cbJ62Tt3c/f3skTimJdpnTw=";
    max-age=2592000; report-uri="https://isc.sans.org/badkey.html"

The Public Key Pinning Extension for HTTP RFC states that browsers must ignore HPKP headers that are received over non-secure transport, and so the above header has no effect other than to consume additional bandwidth.

2.2.2.  HTTP Request Type
  Pinned Hosts SHOULD NOT include the PKP header field in HTTP
  responses conveyed over non-secure transport.  UAs MUST ignore any
  PKP header received in an HTTP response conveyed over non-secure
  transport.

One very good reason for ignoring HPKP policies that are set over unencrypted connections is to prevent "hostile pinning" by man-in-the-middle attackers. If an attacker were to inject a set of pins that the site owner does not control—and if the browser were to blindly cache these values—he would be able to create a junk policy on behalf of that website. This would prevent clients from accessing the site for a long period, without the attacker having to maintain his position as a man-in-the-middle.

If a visitor instead browses to https://www.dshield.org (using HTTPS), an HSTS policy is applied which forces future requests to use HTTPS. The HTTPS site also sets an HPKP header which is then accepted and cached by compatible browsers. However, as the HTTP site does not automatically redirect to the HTTPS site, it is likely that many visitors will never benefit from these HSTS or HPKP polices, even though they are correctly implemented on the HTTPS site.

In another bizarre example, HPKP headers are set by the HTTP site at http://www.msvmgroup.com, even though there is no corresponding HTTPS website (it does accept connections on port 443, but does not present a subscriber certificate that is valid for this hostname).

Not quite got round to it yet...

A few sites that use the Public-Key-Pins header have not quite got around to implementing it yet, such as https://justamagic.ru, which sets the following value:

Public-Key-Pins: TODO

Using HPKP headers to broadcast skepticism

One security company's website – https://websec-test.com – uses the Public-Key-Pins header to express its own skepticisms over the usefulness of HPKP:

Public-Key-Pins: This is like the most useless header I have ever seen.
    Preventing MITM, c'mon, whoever can't trust his own network shouldn't
    enter sensitive data anywhere.

Violation reports that will never be received

The Public-Key-Pins header supports an optional report-uri directive. In the event of a pin validation failure, the user's browser should send a report to this address, in addition to blocking access to the site. These reports are obviously valuable, as they will usually be the first indication that something is wrong.

However, if the report-uri address uses HTTPS, and is also known pinned host, the browser must also carry out pinning checks on this address when the report is sent. This makes it foolish to specify a report-uri that uses the same hostname as the site that is using HPKP.

An example of this configuration blunder can be seen on https://yahvehyireh.com, which sets the following Public-Key-Pins header:

Public-Key-Pins: pin-sha256="y+PfuAS+Dx0OspfM9POCW/HRIqMqsa83jeXaOECu1Ns=";
    pin-sha256="klO23nT2ehFDXCfx3eHTDRESMz3asj1muO+4aIdjiuY=";
    pin-sha256="EohwrK1N7rr3bRQphPj4j2cel+B2d0NNbM9PWHNDXpM=";
    includeSubDomains; max-age=0;
     report-uri="https://yahvehyireh.com/incoming/hpkp/index.php"

This header instructs the browser to send pinning validation failure reports to https://yahvehyireh.com/incoming/hpkp/index.php. However, if there were to be a pinning validation failure on yahvehyireh.com, then the browser would be unable to send any reports because the report-uri itself would also fail the pinning checks by virtue of using the same hostname.

Incidentally, Chrome 46 introduced support for a newer header, Public-Key-Pins-Report-Only, which instructs the browser to perform identical pinning checks to those specified by the Public-Key-Pins header, but it will never block a request when no pinned keys are encountered; instead, the browser will send a report to a URL specified by a report-uri parameter, and the user will be allowed to continue browsing the site. This mechanism would make it safe for site administrators to test the deployment of HPKP on their sites, without inadvertently introducing a denial of service.

Summary

The proportion of secure servers that use HPKP headers is woefully low at only 0.09%, but to make matters worse, many of these few HPKP policies have been implemented incorrectly and do not work as intended.

Without delving into developer settings, browsers offer no visible indications that a site has an invalid HPKP policy, and so it is likely that many website administrators have no idea that their attempts at implementing HPKP have failed. Around a third of the sites that attempt to set an HPKP policy have got it wrong, and consequently behave as if there was no HPKP policy at all. Every response from these servers will include the unnecessary overhead of a header containing a policy that will ultimately be ignored by all browsers.

But there is still hope for the masses: A more viable alternative to HPKP might arise from an Internet-Draft entitled TLS Server Identity Pinning with Tickets. It proposes to extend TLS with opaque tickets, similar to those being used for TLS session resumption, as a way to pin a server's identity. This feature would allow a client to ensure that it is connecting to the right server, even in the presence of a fraudulently issued certificate, but has a significant advantage over HPKP in that no manual management actions would be required. If this draft comes to fruition, and is subsequently implemented by browsers and servers, this ticket-based approach to pinning could potentially see a greater uptake than HPKP has.

Netcraft offers a range of services that can be used to detect and defeat large-scale pharming attacks, and security testing services that identify man-in-the-middle vulnerabilities in web application and mobile apps. Contact security-sales@netcraft.com for more information.

Secure websites shun HTTP Public Key Pinning

The HTTP Public Key Pinning header, or HPKP, can prevent fraudsters using mis-issued TLS certificates. While it offers a robust defence against website impersonation, hardly any HTTPS websites are actually making use of this powerful security feature, even though it has been supported by some browsers for more than a year.

Less than 0.1% of certificates found in Netcraft's March 2016 SSL Survey were served with the HPKP header. Where it has been deployed, a third of webmasters have mistakenly set a broken HPKP policy. With so many mistakes being made, the barrier to entry is evidently high.

Even for those webmasters who have set a valid policy, a lot of ongoing care and attention is required: both routine and emergency maintenance poses a significant risk of blocking legitimate visitors, potentially for long periods of time. However, when correctly deployed and carefully maintained, HPKP is a powerful security feature.

What does HPKP defend against?

A website can defend against most man-in-the-middle attacks by deploying HTTPS, HSTS and HSTS preloading. Together, these ensure all communication to and from the website is authenticated and encrypted.

While these provide a fairly robust defence against attacks like pharming and sslstrip, there is still a line of attack open. A knowledgeable and dedicated attacker can still attack an otherwise well-defended HTTPS website if he can convince a certificate authority to fraudulently issue him a certificate for it.

HPKP can prevent a customer from accessing a spoof website even if it uses a fraudulently-issued (but otherwise valid) certificate. Both sites use certificates issued by trusted CAs, but only the legitimate site's certificate chain contains a certificate that is expected by the client browser.

HPKP can prevent a customer from accessing a spoof website when it uses a fraudulently-issued (but otherwise valid) certificate. Both sites use certificates issued by trusted CAs, but only the legitimate site's certificate chain contains a certificate that is expected by the client browser. Hashes that correspond to each pinned certificate will have been noted by the browser during a previous visit to the legitimate site.

Although it is extremely difficult for a fraudster to obtain a certificate for a domain he does not control, it is not impossible. In fact, there is ample precedent. Several certificate authorities have been breached, lax issuance policies have been discovered, and technical flaws have been exploited by attackers.

The HPKP header is motivated by the history of mis-issuance within this ecosystem. To use HPKP, website owners must select a set of public keys that must be used in future connections. After visiting the site, its HPKP policy is then stored by the client to reject future connections to servers that use different, non-whitelisted keys.

However, creating an HPKP policy is not entirely sufficient to defend against impersonation attacks. In particular, HPKP cannot defend against rogue root certificates installed locally on users' computers.

Both Dell and Lenovo have recently been caught deploying local root certificates to their customers, along with accompanying private keys. With this knowledge, an attacker can generate a certificate for any website and use it to impersonate that site. The victim's browser will regard the certificate as valid, regardless of the genuine site's HPKP policy.

How is HPKP used?

There are three types of key that can be pinned using HPKP:

  • The current public key of the certificate issued to a site.
  • Public keys corresponding to certificate authorities and their intermediate certificates.
  • Backup keys.

In order for browsers to accept and store a website's HPKP policy, there must be at least two pins specified. At least one pin must be in the chain of trust formed by the browser when verifying the site's certificate, and there must be at least one pin that is not in the chain (a backup pin).

Here is an example of a valid HPKP header, which sets pins for three distinct public keys (marked in bold). This policy is valid for one year over all subdomains of the current origin:

Public-Key-Pins:
  pin-sha256="Q0ig6URMeMsmXgWNXolEtNhPlmK9Jtslf4k0pEPHAWE=";
  pin-sha256="F5OSegYUVJeJrc4vjzT38LZtDzrjo7hNIewV27pPrcc=";
  pin-sha256="p1Uk2ryJ7QmI5/zIzFmdzme0X+2nvXG5bHwR88A5ZjA=";
  max-age=31536000; includeSubDomains

Webmasters must be cautious when pinning certificate authority keys. CAs may change their issuance practices without notice, and new certificates may not use the same chain of trust as the old ones. If the new certificate chain no longer includes the pinned keys, the website will not be accessible until the HPKP policy expires.

To avoid the problems posed by using certificate authority keys, webmasters can elect to pin their own keys. This is also a risky practice if the backup key cannot be used: it may have been lost, or may no longer qualify for inclusion in certificates (for example, if a backup key is known to be a Debian weak key, CAs will not accept it for use in new certificates).

"Who dares pins"

HPKP is perfectly safe to implement when pins and certificates are well-managed, but it can also be considered rather risky when you think about what could go wrong: A small mistake could effectively wipe out an online business by preventing its own customers from accessing its website for months. Here are some of the most popular sites that are brave enough to be using HPKP today:

GitHub

GitHub is the busiest site to have deployed HPKP. Well-known for taking security seriously, it sets a plethora of well-configured best-practice security headers.

One of the headers that is set when visiting https://www.github.com is the following HPKP header:

Public-Key-Pins: max-age=300;
  pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18=";
  pin-sha256="JbQbUG5JMJUoI6brnx0x3vZF6jilxsapbXGVfjhN8Fg=";
  includeSubDomains

This HPKP policy specifies two pins, and the directive applies to all subdomains.

The first pinned key (identified by the WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18= SHA-256 hash) corresponds to the DigiCert High Assurance EV Root CA. This is the root of the chain of trust currently used by github.com.

The second hash (JbQbUG5JMJUoI6brnx0x3vZF6jilxsapbXGVfjhN8Fg=) is GitHub's backup pin. This corresponds to the VeriSign Class 3 Public Primary Certification Authority - G5 root. As this key does not appear in GitHub's served certificate chain, it is treated as a backup pin.

When GitHub wants to replace its TLS certificate, the new certificate must be signed by either DigiCert or Symantec – otherwise, none of the key hashes in the new certificate chain would match the existing HPKP policy, and its users would be blocked from accessing the site.

Pinning a pair of root certificate keys is arguably less risky than pinning one of GitHub's own backup keys, but there is a rather large trade-off. With GitHub's current HPKP policy, an attacker can still impersonate the site if he can obtain a fraudulent certificate issued by either DigiCert or Symantec. Conversely, if GitHub were to rely on backup keys that only it controlled, then the only way an attacker could impersonate the site is by compromising GitHub's private keys.

Even so, GitHub evidently remains wary — its HPKP header sets a max-age value of 300. This instructs browsers to remember the policy for no longer than 300 seconds, so in the event of a mistake, users will only be denied access for at most five minutes. However, this makes the policy practically toothless.

In the event of an attack, anybody who has not visited the real www.github.com within the past five minutes is a potential victim. Even if a user has visited GitHub within the past five minutes, being denied access might just be put down to a temporary glitch. A savvy attacker may decide to wait until five minutes after the users last access to GitHub to ensure he will not be caught.

Mozilla

Mozilla is using HPKP much more effectively on its support site, as this site sets a much longer max-age attribute:

Public-Key-Pins: max-age=1296000;
  pin-sha256="r/mIkG3eEpVdm+u/ko/cwxzOMo1bk4TyHIlByibiA5E="
  pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18=";

This equates to 15 days, which means it will provide effective protection to anyone who visits the site at least once a fortnight.

Rather than using public key hashes that correspond to more than one certificate authority, Mozilla has chosen to pin to a single CA: both keys are controlled by DigiCert. In some respects, this is a safer policy by ensuring that only a single CA is able to issue new certificates; however, it leaves Mozilla beholden to DigiCert. If DigiCert were ever forced to stop issuance and Mozilla's certificate required replacement, visitors could be locked out of Mozilla's site for up to 15 days.

Pixabay

A much bolder implementation has been deployed by the Pixabay image library on pixabay.com. Its Public-Key-Pins header specifies a max-age of one year.

Public-Key-Pins:
  pin-sha256="Kx1dtEVeqnPn0gfhzqIJfChEYFr5zMe+FjvcJ0AhVgE=";
  pin-sha256="zN9pxsvWtHm05/fKZ6zA1NJOq4j2NJJA3oIecCNc1eU=";
  max-age=31536000;

Rather than pinning CA-controlled keys, Pixabay has pinned its own certificate's key, as well as a backup key held by Pixabay. This option trades complete defence against CA compromise with a significant risk if the backup pin cannot be used.

If Pixabay were to lose the private keys for both of these certificates, it would likely be catastrophic – visitors would be denied access to its site for an entire year. Pixabay has evidently decided that robust prevention of impersonation attacks is worth the risk.

Why are so few sites daring to use HPKP?

Only 0.09% of all certificates in Netcraft's March 2016 SSL Survey are using HPKP – that's fewer than 4,100 certificates in the whole world that are being delivered with the Public-Key-Pins header.

If that amount did not already seem astonishingly low, more than a quarter of these sites are using the HPKP header incorrectly, which effectively disables HPKP. Consequently, the total number of certificates that are really using HPKP is actually less than 3,000.

Still in its infancy

One of the reasons why HPKP is so rarely deployed could be that it is a relatively new standard, and is still not supported by all mainstream browsers. However, this only partly explains its poor uptake on servers. Although the Public Key Pinning Extension for HTTP was not formalised until the publication of RFC 7469 in April 2015, a significant proportion of internet users have already been able to benefit from this feature since October 2014, when HPKP support was introduced to Chrome 38.

By the time HPKP support was also added to Firefox 35 in January 2015, around a quarter of all internet users were in a position to benefit from sites using HPKP. But today, HPKP remains unsupported in Internet Explorer, Edge, Safari and Opera Mini. Nonetheless, there are millions of people using browsers that do support HPKP, and the only reason they are not benefiting from this technology is because so few websites are deploying it.

Lack of awareness

Possibly the largest reason for the lack of HPKP deployment is that many website owners are simply unaware that this security feature exists, or do not realise the benefits it can bring. However, this is not the most significant problem for lots of websites, as most also lack simpler features that are widely supported, such as HSTS and "Secure" cookies. Implementing HPKP is largely redundant if a site does not also implement HSTS, as this would still allow a man-in-the-middle attacker to hijack unencrypted HTTP traffic and prevent the victim's browser being redirected to the HTTPS site.

Lack of understanding

Netcraft's SSL Survey shows that lots of trivial mistakes are being made when website administrators try to deploy HPKP headers, which indicates a widespread lack of understanding. The net result of these mistakes is that HPKP is not enabled on many sites.

Fear of the "HPKP Footgun"

HPKP is the best way of protecting a site from being impersonated by mis-issued certificates, but as we have already discussed, it is very easy for this protection to backfire with severe consequences. A small misconfiguration could result in a website becoming inaccessible to its own customers.

Summary

HPKP offers a very strong defence against man-in-the-middle attacks, providing it is used in conjunction with HTTPS, HSTS and HSTS Preloading – but despite the obvious security benefits, hardly anyone is using it. Currently, only 0.09% of all secure websites are making use of HPKP headers.

The risk of something going wrong when deploying HPKP is hard to overlook, as a small mistake could ultimately destroy a company's business by making its website inaccessible for months. Only a few thousand secure websites have accepted this risk so far, although you could argue that it only makes sense to deploy HPKP on the largest and most visible websites. For smaller websites, the high risk of something going wrong is outweighed by the incredibly low risk of being attacked: Fraudulently issued certificates are a very rare occurrence, and are more likely to be used to impersonate the biggest websites.

An even newer technology known as Expect CT could potentially provide a safer and easier approach to tackling fraudulently issued certificates. Opted-in websites will be able to tell browsers to expect to see their legitimate certificates in a Certificate Transparency log. These logs are open to public scrutiny, allowing mis-issued certificates to be identified by domain owners, CAs and domain users; and fraudulently issued certificates that do not appear in logs would not be trusted under Expect CT. CAs would be responsible for entering correct details into these logs, thus removing the burden from website operators.

Sites that have properly configured HPKP would be extremely hard to attack in practice, although it is still not impossible. Browsers that have never visited a site before could still be vulnerable to man-in-the-middle attacks if an attacker obtains a valid certificate, because unlike with HSTS, there is no common preload list available for HPKP (it is, however, possible to request special treatment in Google Chrome).

Netcraft offers a range of services that can be used to detect and defeat large-scale pharming attacks, and security testing services that identify man-in-the-middle vulnerabilities in web application and mobile apps. Contact security-sales@netcraft.com for more information.