In the aftermath of Heartbleed, it has become clear that revoking potentially compromised certificates is essential. On Thursday, CloudFlare announced it was reissuing and revoking all of its SSL certificates. The effects of CloudFlare's mass revocation are evident in a single Certificate Revocation List (CRL) belonging to GlobalSign, which grew by almost 134,000 certificates.
The vast number of CloudFlare certificates is due, in part, to the way in which it serves content over SSL. In order to work around the lack of support for Server Name Indication (SNI) in some older operating systems and mobile devices CloudFlare uses GlobalSign's Cloud SSL product. CloudFlare's SSL certificates make use of the Subject Alternative Name (SAN) extension, which allows an edge node to use a single certificate for multiple domains. When a new CloudFlare customer enables SSL, CloudFlare reissues an existing certificate with the new customer's domain added to the existing list of other customers' domains.
The number of certificates revoked per hour since 7th April. GlobalSign's OV CRL at http://crl.globalsign.com/gs/gsorganizationvalg2.crl and other CRLs have been separated.
As a result of CloudFlare's revocations, GlobalSign's CRL at http://crl.globalsign.com/gs/gsorganizationvalg2.crl has ballooned in size and now weighs in at 4.5MB. The CRL is hosted at CloudFlare itself but has nonetheless experienced some performance problems. However, the CRL's performance problems will not have had a significant effect on internet users, as most major browsers use OCSP in preference to CRLs and GlobalSign's OCSP responder did not have any performance problems.
Time to connect to http://crl.globalsign.com/gs/gsorganizationvalg2.crl from Pennsylvania
Time to connect to http://ocsp2.globalsign.com/gsorganizationvalg2 from Pennsylvania
However, most Google Chrome users are left in the dark, as Chrome performs neither type of check for non-EV certificates by default. Instead of conventional revocation checks, Google Chrome relies on an aggregated list of revocations, dubbed CRLSets, which are compiled by Google. The revocations from GlobalSign's CRL have not yet appeared in Google's CRLSets and hence Chrome users will not be warned if presented with a potentially compromised, but revoked, CloudFlare certificate.
The CRLSets deliberately do not cover all CRLs in an attempt to reduce the total size of the aggregated list. In effect, Google has traded the completeness of their revocation checking for a speed advantage over rival browsers as downloading CRLs or making OCSP requests imposes a performance penalty.
Google Chrome setting to enable revocation checking.
However, it is possible to configure Google Chrome to check for revocation. There is a checkbox in the Advanced settings menu to "Check for server certificate revocation".
The Netcraft Extension: Heartbleed and phishing protection rolled into one
The Heartbleed bug affected around 17% of all trusted SSL web servers when it was announced a week ago. The critical vulnerability in the OpenSSL cryptographic library has the potential to allow attackers to retrieve private keys and ultimately decrypt a server's encrypted traffic or even impersonate the server. This is not a theoretical problem: practical attacks have actually succeeded in stealing private keys, yet despite the potential dangers, many of the affected sites have yet to take remedial action.
Even if heartbeat support has been disabled, or OpenSSL upgraded to the latest version, a website that was previously vulnerable to Heartbleed is not necessarily secure today. If the vulnerability had been exploited prior to the upgrade, the certificate's private key could have been compromised. If the certificate has not yet been replaced and the old one revoked, an attacker could impersonate the site and carry out man-in-the-middle attacks against the site's visitors.
Netcraft's updated extensions for Chrome, Firefox and Opera now allow you to see whether the sites you visit are still using potentially compromised certificates. The extensions use data from Netcraft's SSL Survey to determine whether a site offered the heartbeat TLS Extension prior to the Heartbleed disclosure. If this is the case, the extension will also check to see if the site's SSL certificate has been replaced; if it has not, then the site is considered to be unsafe, as the certificate's private key could have been compromised. Even if the certificate has been replaced, it does not guarantee that the site cannot still be impersonated with a copy of the old certificate unless the old certificate has been revoked – and even then, the revocation checking done by browsers is not infallible.
Go here to download the Netcraft Extension for Chrome, Firefox or Opera.
Heartbleed indicator in the Netcraft Google Chrome and Opera Extensions
The extension will indicate when a site is potentially unsafe by displaying a bleeding heart icon. Additionally, in the Google Chrome and Opera versions of the Extension, a warning triangle will be displayed on top of the Netcraft icon.
Heartbleed indicator in the Netcraft Firefox Extension
As well as indicating which sites are using a certificate potentially compromised using Heartbleed, the Netcraft Extension also helps protect you from phishing attacks, displays the hosting location and risk rating of every site you visit, and lets you help to defend the internet community against fraudsters.
Netcraft's site report pages can also be used to determine whether a website might still be affected by the fallout from the Heartbleed bug. For example, our site report for https://www.linkedin.com shows that it no longer supports the TLS heartbeat extension and is using a new certificate.
In contrast, the site report for https://www.fedex.com currently shows that the server previously supported TLS heartbeat and the SSL certificate has not been replaced. Even though TLS heartbeat is now disabled, the certificate could still be used to impersonate the site if it had been compromised prior to heartbeat being disabled. Fedex's website is hosted by Akamai, a popular Content Distribution Network, which was potentially vulnerable to Heartbleed. Akamai is in the process of rotating its customers' SSL certificates and stated that "some require extra validation with the certificate authorities and may take longer".
Heartbleed indicator in the Netcraft Site Report
As the results of CloudFlare's challenge have demonstrated, a server's private key can be extracted using the Heartbleed vulnerability. Consequently, the 500,000+ certificates used on web servers supporting TLS heartbeat should be urgently replaced and revoked. Whilst the replacement and revocation process has begun — 80,000 certificates have been revoked since the announcement — it is far from over.
Private key extraction is real
CloudFlare, which uses a modified version of the nginx web server, originally thought it would be extremely hard or impossible to use the Heartbleed bug to steal a certificate's private key from an nginx server. However, this was quickly proved wrong last week after CloudFlare set up a vulnerable website and challenged people to steal its private key. Later on the same day, the private key had been successfully stolen by exploiting the Heartbleed bug.
Fortunately, CloudFlare decided to play it safe and planned to reissue and revoke potentially affected certificates anyway. CloudFlare also acknowledges that the revocation process is far from perfect and not suitable at mass scale: "If every site revoked its certificates, it would impose a significant burden and performance penalty on the Internet". CloudFlare's own website at cloudflare.com started using a new SSL certificate yesterday, despite the new certificate being marked as valid from 10 April 2014.
Akamai is also planning to rotate all of its customers' SSL certificates after realising a flaw in its recent patch which it originally believed would protect users against the Heartbleed bug. Akamai is notable for its content delivery network of more than 61,000 servers, which they claim delivers 15-20% of all web traffic.
There are already reports that Heartbleed has been used to compromise secure web sites including Canada's tax agency and popular UK web forum Mumsnet.
Revocation is critical (even if it doesn't always work)
As of this morning (Tuesday 15th April), more than 80,000 certificates have been revoked since the public announcement of the vulnerability on 7th April.
The Heartbleed bug has caused a rise in certificate revocations, but the rate predictably fell over the weekend.
Based on list prices, the cost of replacing all of the potentially-compromised certificates with completely new certificates is more than $100 million, but, helpfully, most (but not all) certificate authorities are allowing their customers to reissue and revoke certificates for free. Nonetheless, plenty of the affected websites (e.g. Etsy, Yahoo, GitHub, Steam) appear to have bought new certificates instead of going through the reissuance process, as the new expiry dates are significantly later than the expiry dates in the previous certificates. Perhaps in the haste of resolving the problem, this seemed the easiest approach, making Heartbleed a bonanza for certificate authorities.
While some companies quickly recognised the need to issue new certificates in response to the Heartbleed bug, the number of revocations has not kept up. This is a mistake, as there is little point issuing a new certificate if an attacker is still able to impersonate a website with the old one.
Yahoo was one of the first companies to deploy new SSL certificates after the Heartbleed bug became public knowledge, but the certificate that was previously used by mlogin.yahoo.com has not yet been revoked — it has not been placed on a CRL, and the certificate's OCSP responder says the certificate is "good".
Yahoo is not the only company to have issued a new certificate without ensuring that the previously vulnerable certificate has been revoked. Other sites which fall into this category include banking websites (such as entry7.credit-suisse.ch), the United States Senate large file transfer system at lfts.senate.gov, and GeoTrust's SSL Toolbox at https://ssltools.geotrust.com/checker/ (GeoTrust is a brand owned by Symantec, the largest certificate authority).
Thousands of certificates could still be misused after being revoked
Critically, some of the certificates affected by the Heartbleed bug will remain usable even if revoked: Nearly 4% of the certificates do not specify a URL for an OCSP responder, which means that they can only be revoked via a CRL. This makes the certificates effectively irrevocable in some browsers — for example, the latest version of Mozilla Firefox no longer uses CRLs at all (previously it would fall back to checking a CRL if an OCSP request failed, but only for Extended Validation certificates).
Worse still, a small number of the certificates that could have been compromised through exploitation of the Heartbleed bug fail to specify either an OCSP or a CRL address. These certificates are therefore completely irrevocable in all browsers and could be impersonated until their natural expiry dates if an attacker has already compromised the private keys.
For example, Telecom Italia (a sub-CA of Verizon Business) is still using an irrevocable certificate on www.cloudpeople.it, which supported the TLS heartbeat extension prior to the disclosure of the Heartbleed bug. The 3-year certificate was issued by I.T. Telecom Global CA at the end of 2011 and will remain valid until the end of 2014 because it does not permit either form of revocation.
CRLs will balloon as a result of the surge of revocations
To obtain the certificate revocation lists (CRLs) used by each publicly trusted certificate authority, a web browser would need to download more than 100MB of data. These CRLs will grow by about 35% if all of the certificates affected by the Heartbleed bug were revoked. Downloading this much data is clearly impractical for many mobile devices, and several CRLs either time-out or take more than a minute to download, even from a desktop machine with a fast internet connection. This goes against the CA/Browser Forum's Baseline Requirements, which expect CAs to provide response times of less than 10 seconds.
The largest CRL (11MB) is operated by the US Department of the Treasury, and despite containing more than 200,000 revocation entries, it is only used by one publicly accessible certificate. Nonetheless, any browser wishing to perform a CRL check for that one site will have to download the whole list. Governments also feature amongst the worst-performing CRLs: For example, the Taiwanese government offers a CRL at http://hcaocsp.nat.gov.tw/repository/HCA/CRL/complete.crl, which would not respond when tested earlier today, and the Brazilian government offers several CRLs from its site at repositorio.icpbrasil.gov.br, but each took 2-3 minutes to download, despite being of relatively modest sizes.
There has been a noticeable rise in certificate re-issuance since 7 April 2014
Some of the first sites to deploy newly issued certificates in response to the OpenSSL vulnerability included Yahoo, Adobe, CloudFlare, DuckDuckGo, GitHub, Reddit , Launchpad, PayPal, Netflix and Amazon's CloudFront content delivery network.
Such is the haste to fix the fallout of the Heartbleed bug, some certificate authorities and website administrators have been making careless mistakes. PayPal's Hosted Message Applications, such as the one at https://view.paypal-communication.com, are now using Extended Validation certificates issued by VeriSign on 10 April 2014. The CAB Forum requires certificate authorities to adhere to a stringent set of guidelines [pdf] when issuing EV certificates, and it is the CA's responsibility to verify the accuracy of the information in the certificate. In particular, they must verify that the legal name of the subject in an EV certificate matches the name which appears on official government records.
However, this verification does not appear to have been performed correctly in the case of these certificates, as they have been erroneously issued to an organisation named "PayPal, Inc.\0a" instead of "PayPal, Inc."
Extended Validation certificates cause browsers to display a green dialog, indicating the additional — and supposedly more accurate — identify verification criteria.
If you don't revoke your certificate, you may still be vulnerable to impersonation
If your private key has been stolen, just reissuing the certificate is not enough to mitigate the risks posed by the Heartbleed bug. Websites which were affected by the bug could still be vulnerable to impersonation attacks in the future if they fail to revoke their certificates, even if they have upgraded to the latest version of OpenSSL and replaced their SSL certificates.
If a remote attacker successfully retrieved private keys from a server while it was still vulnerable to the Heartbleed bug, then he would be able to impersonate the server by creating his own valid SSL certificate. The crucial issue is that an attacker can still do this after the affected website has upgraded to the latest version of OpenSSL, and it does not matter whether the real website has since deployed a new SSL certificate with different keys: Unless the previous certificate is revoked, the site will still be vulnerable to man-in-the-middle attacks.
Certificate revocations have gone up, but most of the vulnerable certificates are yet to be revoked.
Despite the importance of revoking certificates which could have been stolen using the Heartbleed bug, many website administrators and certificate authorities have yet to do this. Activity on certificate revocation lists peaked at a rate of 3,900 revocations per hour on the day the Heartbleed bug was announced (Monday April 7, 2014). On a typical Monday, we would expect to see a total of around 22,000-30,000 SSL certificates being revoked over the course of the day. On the Monday that the Heartbleed bug was announced to the public, there were 29,000 revocations. On the next day (Tuesday), 33,000 certificates were revoked, followed by 32,000 on Wednesday. These were both above average, suggesting that around 5,000 certificates were revoked in direct response to the Heartbleed bug each day. Note that fewer revocations usually take place over weekends.
Certificate authorities must revoke certificates within 24 hours if there is evidence of a key compromise. A private key is said to be compromised if its value has been disclosed, or if there exists a practical technique by which an unauthorised person may discover its value. Arguably, all certificates on sites vulnerable to the Heartbleed bug should be revoked by now, as such a technique was successfully carried out by the researchers behind heartbleed.com.
Even if you revoke your certificate, you may still be vulnerable to impersonation
However, even if all of the affected certificates were to be revoked, contemporary web browser software handles certificate revocation poorly. The most frequent users of a site — often its administrators — can continue using a revoked certificate for weeks or months without the browser notifying them that anything is amiss. In this situation, an attacker can perform a man-in-the-middle (MITM) attack by presenting the certificate to unsuspecting users whose browsers will behave as if they were connecting to the legitimate site. For example, some browsers only perform OCSP revocation checks for Extended Validation certificates, while others ignore certificate revocation lists completely.
You are encouraged to read our previous article on certificate revocation.
Since this article was first published, the revocation data has been updated to include more events.
A serious overrun vulnerability in the OpenSSL cryptographic library affects around 17% of SSL web servers which use certificates issued by trusted certificate authorities. Already commonly known as the Heartbleed bug, a missing bounds check in the handling of the TLS heartbeat extension can allow remote attackers to view up to 64 kilobytes of memory on an affected server. This could allow attackers to retrieve private keys and ultimately decrypt the server's encrypted traffic or even impersonate the server.
The Heartbleed bug write-up mentions Apache and nginx as being the most notable software using OpenSSL, and also points out that these have a combined active site market share of over 66% according to our April 2014 Web Server Survey. However, not all of these servers are running an HTTPS service, nor are they all running vulnerable versions of OpenSSL with heartbeats enabled.
Our most recent SSL Survey found that the heartbeat extension was enabled on 17.5% of SSL sites, accounting for around half a million certificates issued by trusted certificate authorities. These certificates are consequently vulnerable to being spoofed (through private key disclosure), allowing an attacker to impersonate the affected websites without raising any browser warnings.
Most vulnerable servers are using Apache.
Note that a small percentage of Microsoft web servers also appear to support the TLS heartbeat extension; these are actually likely to be vulnerable Linux machines acting as reverse proxy frontends to Windows servers.
Support for heartbeats was added to OpenSSL 1.0.1 (released in 2012) by Robin Seggelmann, who also coauthored the Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS) Heartbeat Extension RFC. The new code was committed to OpenSSL's git repository just before midnight on new year's eve 2011.
OpenSSL's security advisory states that only versions 1.0.1 and 1.0.2-beta are affected, including 1.0.1f and 1.0.2-beta1. The vulnerability has been fixed in OpenSSL 1.0.1g, and users who are unable to upgrade immediately can disable heartbeat support by recompiling OpenSSL with the
Popular sites which exhibit support for the TLS heartbeat extension include Twitter, GitHub, Yahoo, Tumblr, Steam, DropBox, HypoVereinsbank, PostFinance, Regents Bank, Commonwealth Bank of Australia, and the anonymous search engine DuckDuckGo.
Certificates and keys at risk of compromise should be revoked and replaced, particularly if they are used to protect sensitive data. Certificate Authorities, hosting companies and other interested parties can contact us for assistance in identifying affected certificates.
You can check whether your own HTTPS website might be vulnerable using the form below, and looking for the RFC6520 heartbeat TLS extension.
Netcraft site report
Thousands of websites are still hosted on Windows XP computers, despite the operating system reaching the end of its extended support period today. After today, Microsoft will stop providing automatic security updates for Windows XP, and Microsoft Security Essentials will also no longer be available for Windows XP.
Originally released in 2001, Windows XP is currently used by more than 6,000 websites in Netcraft's April 2014 Web Server Survey. Although China is often regarded as one of the most prolific users of Windows XP, only 3% of these sites are hosted there, suggesting that Windows XP has a predominantly desktop role in China. The largest share (nearly a third) of all Windows XP-powered websites are actually hosted in the United States.
Distribution of Windows XP-powered websites (logarithmic scale)
Notably, there are 14 US government websites still running on Windows XP, including a webmail system used by the State of Utah. Unsupported web-facing Windows XP servers are likely to become prime targets for hackers, particularly if any new Windows XP vulnerabilities are discovered, as no security updates will be available to fix them. To afford some breathing space, the UK Government recently struck a £5.5m deal for Microsoft to provide it with an extra year of support for Windows XP, although there are currently no Windows XP-powered websites under the gov.uk top-level domain.
One of the busiest sites still using Windows XP is TransFerry.com. This site was previously using Windows 2000, and perhaps more worrying is the significantly larger number of websites which still use Windows 2000. This version of Windows reached its extended support end date in July 2010, yet nearly half a million of today's websites are hosted on Windows 2000 servers, most of which are using the Microsoft IIS 5.0 web server software they were shipped with. This version of IIS is practically identical to that used by Windows XP (IIS 5.1).
Netcraft's April 2014 survey also found 50,000 websites which are hosted on even older Windows NT4 servers running Microsoft IIS 4.0, although three quarters of these sites are served from the same computer in Norway. One of the busiest sites still running on Windows NT4 is the Australian Postal Corporation's post.com.au, which has been using the same operating system for at least 13 years. Window NT4 and IIS 4.0 are also still used by Australia Post's Postbillpay bill payment service, airindia.co.in and by the French government's Ministère de l'Économie, des Finances et de l'Industrie.