As the results of CloudFlare's challenge have demonstrated, a server's private key can be extracted using the Heartbleed vulnerability. Consequently, the 500,000+ certificates used on web servers supporting TLS heartbeat should be urgently replaced and revoked. Whilst the replacement and revocation process has begun — 80,000 certificates have been revoked since the announcement — it is far from over.
Private key extraction is real
CloudFlare, which uses a modified version of the nginx web server, originally thought it would be extremely hard or impossible to use the Heartbleed bug to steal a certificate's private key from an nginx server. However, this was quickly proved wrong last week after CloudFlare set up a vulnerable website and challenged people to steal its private key. Later on the same day, the private key had been successfully stolen by exploiting the Heartbleed bug.
Fortunately, CloudFlare decided to play it safe and planned to reissue and revoke potentially affected certificates anyway. CloudFlare also acknowledges that the revocation process is far from perfect and not suitable at mass scale: "If every site revoked its certificates, it would impose a significant burden and performance penalty on the Internet". CloudFlare's own website at cloudflare.com started using a new SSL certificate yesterday, despite the new certificate being marked as valid from 10 April 2014.
Akamai is also planning to rotate all of its customers' SSL certificates after realising a flaw in its recent patch which it originally believed would protect users against the Heartbleed bug. Akamai is notable for its content delivery network of more than 61,000 servers, which they claim delivers 15-20% of all web traffic.
There are already reports that Heartbleed has been used to compromise secure web sites including Canada's tax agency and popular UK web forum Mumsnet.
Revocation is critical (even if it doesn't always work)
As of this morning (Tuesday 15th April), more than 80,000 certificates have been revoked since the public announcement of the vulnerability on 7th April.
The Heartbleed bug has caused a rise in certificate revocations, but the rate predictably fell over the weekend.
Based on list prices, the cost of replacing all of the potentially-compromised certificates with completely new certificates is more than $100 million, but, helpfully, most (but not all) certificate authorities are allowing their customers to reissue and revoke certificates for free. Nonetheless, plenty of the affected websites (e.g. Etsy, Yahoo, GitHub, Steam) appear to have bought new certificates instead of going through the reissuance process, as the new expiry dates are significantly later than the expiry dates in the previous certificates. Perhaps in the haste of resolving the problem, this seemed the easiest approach, making Heartbleed a bonanza for certificate authorities.
While some companies quickly recognised the need to issue new certificates in response to the Heartbleed bug, the number of revocations has not kept up. This is a mistake, as there is little point issuing a new certificate if an attacker is still able to impersonate a website with the old one.
Yahoo was one of the first companies to deploy new SSL certificates after the Heartbleed bug became public knowledge, but the certificate that was previously used by mlogin.yahoo.com has not yet been revoked — it has not been placed on a CRL, and the certificate's OCSP responder says the certificate is "good".
Yahoo is not the only company to have issued a new certificate without ensuring that the previously vulnerable certificate has been revoked. Other sites which fall into this category include banking websites (such as entry7.credit-suisse.ch), the United States Senate large file transfer system at lfts.senate.gov, and GeoTrust's SSL Toolbox at https://ssltools.geotrust.com/checker/ (GeoTrust is a brand owned by Symantec, the largest certificate authority).
Thousands of certificates could still be misused after being revoked
Critically, some of the certificates affected by the Heartbleed bug will remain usable even if revoked: Nearly 4% of the certificates do not specify a URL for an OCSP responder, which means that they can only be revoked via a CRL. This makes the certificates effectively irrevocable in some browsers — for example, the latest version of Mozilla Firefox no longer uses CRLs at all (previously it would fall back to checking a CRL if an OCSP request failed, but only for Extended Validation certificates).
Worse still, a small number of the certificates that could have been compromised through exploitation of the Heartbleed bug fail to specify either an OCSP or a CRL address. These certificates are therefore completely irrevocable in all browsers and could be impersonated until their natural expiry dates if an attacker has already compromised the private keys.
For example, Telecom Italia (a sub-CA of Verizon Business) is still using an irrevocable certificate on www.cloudpeople.it, which supported the TLS heartbeat extension prior to the disclosure of the Heartbleed bug. The 3-year certificate was issued by I.T. Telecom Global CA at the end of 2011 and will remain valid until the end of 2014 because it does not permit either form of revocation.
CRLs will balloon as a result of the surge of revocations
To obtain the certificate revocation lists (CRLs) used by each publicly trusted certificate authority, a web browser would need to download more than 100MB of data. These CRLs will grow by about 35% if all of the certificates affected by the Heartbleed bug were revoked. Downloading this much data is clearly impractical for many mobile devices, and several CRLs either time-out or take more than a minute to download, even from a desktop machine with a fast internet connection. This goes against the CA/Browser Forum's Baseline Requirements, which expect CAs to provide response times of less than 10 seconds.
The largest CRL (11MB) is operated by the US Department of the Treasury, and despite containing more than 200,000 revocation entries, it is only used by one publicly accessible certificate. Nonetheless, any browser wishing to perform a CRL check for that one site will have to download the whole list. Governments also feature amongst the worst-performing CRLs: For example, the Taiwanese government offers a CRL at http://hcaocsp.nat.gov.tw/repository/HCA/CRL/complete.crl, which would not respond when tested earlier today, and the Brazilian government offers several CRLs from its site at repositorio.icpbrasil.gov.br, but each took 2-3 minutes to download, despite being of relatively modest sizes.
There has been a noticeable rise in certificate re-issuance since 7 April 2014
Some of the first sites to deploy newly issued certificates in response to the OpenSSL vulnerability included Yahoo, Adobe, CloudFlare, DuckDuckGo, GitHub, Reddit , Launchpad, PayPal, Netflix and Amazon's CloudFront content delivery network.
Such is the haste to fix the fallout of the Heartbleed bug, some certificate authorities and website administrators have been making careless mistakes. PayPal's Hosted Message Applications, such as the one at https://view.paypal-communication.com, are now using Extended Validation certificates issued by VeriSign on 10 April 2014. The CAB Forum requires certificate authorities to adhere to a stringent set of guidelines [pdf] when issuing EV certificates, and it is the CA's responsibility to verify the accuracy of the information in the certificate. In particular, they must verify that the legal name of the subject in an EV certificate matches the name which appears on official government records.
However, this verification does not appear to have been performed correctly in the case of these certificates, as they have been erroneously issued to an organisation named "PayPal, Inc.\0a" instead of "PayPal, Inc."
Extended Validation certificates cause browsers to display a green dialog, indicating the additional — and supposedly more accurate — identify verification criteria.
If you don't revoke your certificate, you may still be vulnerable to impersonation
If your private key has been stolen, just reissuing the certificate is not enough to mitigate the risks posed by the Heartbleed bug. Websites which were affected by the bug could still be vulnerable to impersonation attacks in the future if they fail to revoke their certificates, even if they have upgraded to the latest version of OpenSSL and replaced their SSL certificates.
If a remote attacker successfully retrieved private keys from a server while it was still vulnerable to the Heartbleed bug, then he would be able to impersonate the server by creating his own valid SSL certificate. The crucial issue is that an attacker can still do this after the affected website has upgraded to the latest version of OpenSSL, and it does not matter whether the real website has since deployed a new SSL certificate with different keys: Unless the previous certificate is revoked, the site will still be vulnerable to man-in-the-middle attacks.
Certificate revocations have gone up, but most of the vulnerable certificates are yet to be revoked.
Despite the importance of revoking certificates which could have been stolen using the Heartbleed bug, many website administrators and certificate authorities have yet to do this. Activity on certificate revocation lists peaked at a rate of 3,900 revocations per hour on the day the Heartbleed bug was announced (Monday April 7, 2014). On a typical Monday, we would expect to see a total of around 22,000-30,000 SSL certificates being revoked over the course of the day. On the Monday that the Heartbleed bug was announced to the public, there were 29,000 revocations. On the next day (Tuesday), 33,000 certificates were revoked, followed by 32,000 on Wednesday. These were both above average, suggesting that around 5,000 certificates were revoked in direct response to the Heartbleed bug each day. Note that fewer revocations usually take place over weekends.
Certificate authorities must revoke certificates within 24 hours if there is evidence of a key compromise. A private key is said to be compromised if its value has been disclosed, or if there exists a practical technique by which an unauthorised person may discover its value. Arguably, all certificates on sites vulnerable to the Heartbleed bug should be revoked by now, as such a technique was successfully carried out by the researchers behind heartbleed.com.
Even if you revoke your certificate, you may still be vulnerable to impersonation
However, even if all of the affected certificates were to be revoked, contemporary web browser software handles certificate revocation poorly. The most frequent users of a site — often its administrators — can continue using a revoked certificate for weeks or months without the browser notifying them that anything is amiss. In this situation, an attacker can perform a man-in-the-middle (MITM) attack by presenting the certificate to unsuspecting users whose browsers will behave as if they were connecting to the legitimate site. For example, some browsers only perform OCSP revocation checks for Extended Validation certificates, while others ignore certificate revocation lists completely.
You are encouraged to read our previous article on certificate revocation.
Since this article was first published, the revocation data has been updated to include more events.
A serious overrun vulnerability in the OpenSSL cryptographic library affects around 17% of SSL web servers which use certificates issued by trusted certificate authorities. Already commonly known as the Heartbleed bug, a missing bounds check in the handling of the TLS heartbeat extension can allow remote attackers to view up to 64 kilobytes of memory on an affected server. This could allow attackers to retrieve private keys and ultimately decrypt the server's encrypted traffic or even impersonate the server.
The Heartbleed bug write-up mentions Apache and nginx as being the most notable software using OpenSSL, and also points out that these have a combined active site market share of over 66% according to our April 2014 Web Server Survey. However, not all of these servers are running an HTTPS service, nor are they all running vulnerable versions of OpenSSL with heartbeats enabled.
Our most recent SSL Survey found that the heartbeat extension was enabled on 17.5% of SSL sites, accounting for around half a million certificates issued by trusted certificate authorities. These certificates are consequently vulnerable to being spoofed (through private key disclosure), allowing an attacker to impersonate the affected websites without raising any browser warnings.
Most vulnerable servers are using Apache.
Note that a small percentage of Microsoft web servers also appear to support the TLS heartbeat extension; these are actually likely to be vulnerable Linux machines acting as reverse proxy frontends to Windows servers.
Support for heartbeats was added to OpenSSL 1.0.1 (released in 2012) by Robin Seggelmann, who also coauthored the Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS) Heartbeat Extension RFC. The new code was committed to OpenSSL's git repository just before midnight on new year's eve 2011.
OpenSSL's security advisory states that only versions 1.0.1 and 1.0.2-beta are affected, including 1.0.1f and 1.0.2-beta1. The vulnerability has been fixed in OpenSSL 1.0.1g, and users who are unable to upgrade immediately can disable heartbeat support by recompiling OpenSSL with the
Popular sites which exhibit support for the TLS heartbeat extension include Twitter, GitHub, Yahoo, Tumblr, Steam, DropBox, HypoVereinsbank, PostFinance, Regents Bank, Commonwealth Bank of Australia, and the anonymous search engine DuckDuckGo.
Certificates and keys at risk of compromise should be revoked and replaced, particularly if they are used to protect sensitive data. Certificate Authorities, hosting companies and other interested parties can contact us for assistance in identifying affected certificates.
You can check whether your own HTTPS website might be vulnerable using the form below, and looking for the RFC6520 heartbeat TLS extension.
Netcraft site report
Thousands of websites are still hosted on Windows XP computers, despite the operating system reaching the end of its extended support period today. After today, Microsoft will stop providing automatic security updates for Windows XP, and Microsoft Security Essentials will also no longer be available for Windows XP.
Originally released in 2001, Windows XP is currently used by more than 6,000 websites in Netcraft's April 2014 Web Server Survey. Although China is often regarded as one of the most prolific users of Windows XP, only 3% of these sites are hosted there, suggesting that Windows XP has a predominantly desktop role in China. The largest share (nearly a third) of all Windows XP-powered websites are actually hosted in the United States.
Distribution of Windows XP-powered websites (logarithmic scale)
Notably, there are 14 US government websites still running on Windows XP, including a webmail system used by the State of Utah. Unsupported web-facing Windows XP servers are likely to become prime targets for hackers, particularly if any new Windows XP vulnerabilities are discovered, as no security updates will be available to fix them. To afford some breathing space, the UK Government recently struck a £5.5m deal for Microsoft to provide it with an extra year of support for Windows XP, although there are currently no Windows XP-powered websites under the gov.uk top-level domain.
One of the busiest sites still using Windows XP is TransFerry.com. This site was previously using Windows 2000, and perhaps more worrying is the significantly larger number of websites which still use Windows 2000. This version of Windows reached its extended support end date in July 2010, yet nearly half a million of today's websites are hosted on Windows 2000 servers, most of which are using the Microsoft IIS 5.0 web server software they were shipped with. This version of IIS is practically identical to that used by Windows XP (IIS 5.1).
Netcraft's April 2014 survey also found 50,000 websites which are hosted on even older Windows NT4 servers running Microsoft IIS 4.0, although three quarters of these sites are served from the same computer in Norway. One of the busiest sites still running on Windows NT4 is the Australian Postal Corporation's post.com.au, which has been using the same operating system for at least 13 years. Window NT4 and IIS 4.0 are also still used by Australia Post's Postbillpay bill payment service, airindia.co.in and by the French government's Ministère de l'Économie, des Finances et de l'Industrie.
The website of the Agency for the Safety of Aerial Navigation in Africa and Madagascar (ASECNA) has been hijacked by hackers. Browsing to the site's homepage currently presents visitors with a PayPal phishing site, where visitors are asked to submit PayPal account details, including their password, address and credit card details. After entering these details, victims are redirected to the real PayPal website.
Visitors to the ASECNA homepage are automatically redirected to this phishy PHP script in the root directory.
ASECNA is responsible for managing 16 million square kilometers of airspace (1.5x the size of Europe), covering six flight information regions, but has yet to remove the phishing site from its own homepage. Netcraft detected and blocked the above PayPal phishing site on Tuesday, yet visitors to www.asecna.aero who ignore their browser's warnings are still being presented with the phishing content today (Friday). Comments within the source code suggest that the phishing site was designed by a man living in Salé, Morocco.
A second PayPal phishing site was also found in a subdirectory on the same server, but it has since been deleted. It is possible that it was deleted by the fraudster behind the current attack, as it would be peculiar for ASECNA to have deleted phishing content from a subdirectory while leaving the more obvious phishing content on its homepage. The deleted phishing site used a phishing kit which hid its author's hotmail.fr email address in a Base64 encoded string. This made it less obvious to anyone deploying the kit that a duplicate copy of any stolen credentials would also be surreptitiously emailed directly to the kit's author. The phishing kit author's email address links him to a Facebook account which places him in Rabat, a Moroccan city which attracts many commuters from Salé. The same email address has been found in several other phishing kits, including some which target Visa customers.
It is rather unusual to see phishing sites hosted on .aero domains because they can only be registered by eligible members of the aviation community. SITA (an air transport IT and communications specialist) is responsible for verifying eligibility, and may ask applicants to provide company documents and pilot licenses, which reduces the likelihood of a fraudster registering a .aero domain specifically for the purpose of phishing. Many other top-level domains are easier to register, and some are even free.
.aero is a sponsored top-level domain (sTLD). The original agreement for the domain was signed in 2001, and domains became available for registration in March 2002. In 2009, SITA signed a new 10-year sponsorship agreement for the .aero sTLD with ICANN.
How the ASECNA site looked prior to the compromise.
Netcraft's April 2014 survey found more than 9,000 sites using the .aero sTLD, and in the past 6 months they have hosted a total of 9 phishing sites. Each attack used an established .aero website which was compromised to host phishing content, rather than using a .aero domain registered specifically for fraud.
It is not apparent how the ASECNA website was compromised, although it appears to be running Apache 2.2.14, which could be vulnerable to a plethora of security issues which can be exploited remotely. The server also uses PHP 5.2.5, which was released in 2007, and the entire 5.2 branch of releases reached end of life status at the beginning of 2011. Unless the server is using a backporting approach to software maintenance, this old version of PHP could also expose a large number of vulnerabilities to remote attackers.
Netcraft's continuously updated, professionally validated phishing feed is used throughout the internet infrastructure industry. In addition to internet registries, all of the main web browsers, along with major anti-virus companies, firewall vendors, SSL Certificate authorities, large hosting companies and domain registrars use Netcraft's feed to protect their user communities. Please contact us for more information about these services, or about Netcraft's phishing site takedown service.
Rank Performance Graph OS Outage
DNS Connect First
Total 1 Datapipe FreeBSD 0:00:00 0.011 0.081 0.018 0.037 0.055 2 www.choopa.com Linux 0:00:00 0.011 0.163 0.074 0.157 0.204 3 ReliableServers.com Linux 0:00:00 0.015 0.177 0.075 0.154 0.199 4 Qube Managed Services Linux 0:00:00 0.019 0.095 0.036 0.076 0.076 5 Hyve Managed Hosting Linux 0:00:00 0.019 0.230 0.064 0.127 0.131 6 Anexia Linux 0:00:00 0.019 0.232 0.089 0.411 0.685 7 Bigstep Linux 0:00:00 0.022 0.244 0.065 0.177 0.209 8 Webzilla unknown 0:00:00 0.026 0.124 0.070 0.138 0.393 9 Netcetera Windows Server 2012 0:00:00 0.030 0.059 0.072 0.158 0.291 10 ServerStack Linux 0:00:00 0.030 0.081 0.075 0.148 0.148
Managed services provider Datapipe had the most reliable hosting company site in March, closely followed by Choopa in second place. Both of the top two hosting company sites experienced three failed requests, and therefore the tie for first place was broken by analysing average connection times. Datapipe had the lowest average connection time within the top ten of 18ms and therefore ranked in first place.
Datapipe has a 100% uptime record which now stretches back over eight years; its last outage occurred back in March 2006. Over this time Datapipe's infrastructure has proved it can withstand the brutal forces of nature, surviving several hurricanes, typhoons and a snow storm. Along with 100% uptime, Datapipe has a low proportion of failed requests which has led to them ranking in first place many times over the years.
Second-place Choopa is based in a data centre in Piscataway, New Jersey and additionally has infrastructure in Los Angeles, Amsterdam, and Tokyo. Choopa describes its infrastructure's architecture as redundant with no single point of failure, and has backed this up with a 100% Uptime SLA plus a 0% Packet Loss Guarantee within its network. Choopa offers IPv6 throughout its entire network using a dual stack approach — avoiding the need to tunnel over IPv4. Recently Choopa has launched its own SSD VPS service via a new brand Vultr.
In third place with four failed requests is ReliableServers which lists reliability as its number 1 policy. ReliableServers is based in New Jersey and purchases server racks and network bandwidth from Choopa in Piscataway which hooks its servers directly into Choopa's network. ReliableServers offers Dedicated hosting with a 100% uptime guarantee.
Elsewhere in the table Webzilla made its first appearance in the top ten, which may be a result of its recent infrastructure upgrades. Webzilla launched in 2005 and offers a range of hosting services including dedicated, cloud, colocation and CDN.
Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.
From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.
Information on the measurement process and current measurements is available.