Most Reliable Hosting Company Sites in April 2014

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Qube Managed Services Linux 0:00:00 0.004 0.105 0.036 0.074 0.074
2 Host Europe Linux 0:00:00 0.012 0.110 0.072 0.149 0.150
3 XILO Communications Ltd. Linux 0:00:00 0.015 0.202 0.066 0.131 0.226
4 Netcetera Windows Server 2012 0:00:00 0.015 0.065 0.071 0.157 0.295
5 New York Internet FreeBSD 0:00:00 0.015 0.145 0.071 0.148 0.555
6 ServerStack Linux 0:00:00 0.015 0.079 0.074 0.146 0.146
7 Datapipe FreeBSD 0:00:00 0.019 0.102 0.018 0.037 0.054
8 Swishmail FreeBSD 0:00:00 0.019 0.127 0.070 0.141 0.186
9 Logicworks Linux 0:00:00 0.019 0.149 0.071 0.148 0.299
10 Webzilla FreeBSD 0:00:00 0.023 0.127 0.071 0.140 0.355

See full table

Qube had the most reliable hosting company site in April, with only one failed request. The London-based managed hosting provider uses data centres in the UK, USA and Switzerland, all of which are ISO 27001 certified. This information security management system standard uses a three-stage external audit process to ensure the company has suitable security policies and risk treatment plans.

Qube (which stands for "Qualified By Experience") has performed rather well so far this year — it also had the most reliable hosting company site in February (with no failed requests), narrowly missed out on first place in January, and made 4th place in March. Qube provides a Virtual Data Centre cloud computing service powered by VMware vCloud, managed dedicated servers, and managed colocation in its Tier 3 central London data centre.

Host Europe had the second most reliable hosting company site in April, with three isolated failed requests from Netcraft's globally distributed performance collectors. The company has attained 100% uptime over the past six months, and 99.98% uptime over 12 months. The Host Europe Group also owns the largest domain name provider in the UK, 123-reg, which recently teamed up with Knowhow to offer domain and website building packages from Currys and PC World stores.

XILO Communications came third in April, with four failed requests. Its uptime over the past 12 months is an impressive 99.993%. Netcetera, New York Internet, and ServerStack also had only four failed requests, but with longer average connection times than XILO.

Half of April's top ten hosting company websites were served from Linux machines, including all of the top three. Four of the other sites used FreeBSD, and one used Windows Server 2012.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

Phishers find Microsoft Azure 30-day trial irresistible!

Fraudsters have taken to Microsoft Azure to deploy phishing sites, taking advantage of Microsoft's free 30-day trial.

Free hosting!

In order to get a phishing site hosted at Azure, the fraudster has several options: steal the credentials for a Microsoft account, compromise a virtual machine running at Azure, or use Microsoft’s free trial which provides $200 of credit. Given the number of subdomains registered explicitly for phishing, it is unlikely that many fraudsters are exploiting legitimate customers’ virtual machines.

Free subdomains!

Microsoft Azure offers free subdomains to users: azurewebsites.net for its Azure Web Sites service and cloudapp.net for Cloud Apps and virtual machines. Almost twice as many phishing sites used azurewebsites.net rather than cloudapp.net, perhaps reflecting the ease-of-use of Azure Web Sites. The remainder of the phishing sites are accessed using their IP addresses or custom domains.

An Apple phishing site on itune-billing2update-ssl-apple.azurewebsites.net (Site Report).

Many of the subdomains are clearly registered with the intention of phishing; the table below includes some of the most egregious examples targeting well-known institutions.

SubdomainTarget
itune-billing2update-ssl-apple.azurewebsites.netApple
paypalscurity.azurewebsites.netPayPal
www22online-americanexpress.azurewebsites.netAmerican Express
3seb-verifiedbyvisa.azurewebsites.netVisa
login-comcastforceauthn.azurewebsites.netComcast
cielo-2014.cloudapp.netCielo

Free SSL certificate!

Microsoft Azure Web Sites also offers fraudsters the ability to use an SSL certificate. All subdomains of azurewebsites.net are automatically accessible via HTTPS using a *.azurewebsites.net SSL certificate. The Apple phishing site featured below includes mixed content, indicating it was probably not designed with SSL in mind despite its subdomain: itune-billing2update-ssl-apple. Phishing sites that make proper use of the wildcard SSL certificate may be able to instil more trust than those that do not.

An SSL certificate on itune-billing2update-ssl-apple.azurewebsites.net (Site Report).

SSL certificate is irrevocable!

The Baseline Requirements that forms part of Mozilla's CA policy suggests that the SSL certificate must be revoked within 24 hours: "The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: [..] [t]he CA is made aware that a Wildcard Certificate has been used to authenticate a fraudulently misleading subordinate Fully-Qualified Domain Name". However, Microsoft itself issued the SSL certificate from its sub-CA of Verizon Business and has chosen not to revoke it. Moreover, the SSL certificate does not include an OCSP responder URL and is not served with a stapled response (which is also in violation of the Baseline Requirements) and consequently the SSL certificate is irrevocable in some major browsers, particularly Firefox.

Free email addresses!

Fraudsters are also using Microsoft-provided free email addresses (at live.com, hotmail.com, and outlook.com) to receive and store stolen phishing credentials. Fraudsters commonly use phishing kits to quickly deploy phishing sites — before deployment, the fraudster configures the phishing kit with his email address. If a victim is tricked by the phishing site into providing his credentials, they are sent back to the fraudster's email address.

Free anonymising proxy!

One fraudster used Azure to proxy his internet traffic when accessing the phishing site, but was exposed when he used the same email address in the phishing kit as he used on his Facebook profile. The fraudster left the log file that records visits to the phishing site accessible to the public. The first two entries in the log, which preceded all other accesses by several hours, were from Microsoft Azure IP addresses. It is likely these correspond to the fraudster checking his phishing site was ready to be sent out to would-be victims.

1  137.117.104.222  -  2014-3-27 @ 02:56:03
2  137.117.104.222  -  2014-3-27 @ 02:57:16
3  109.XXXXXXXXX  -  2014-3-27 @ 11:22:26
4  212.XXXXXXXXX  -  2014-3-27 @ 11:39:47
5  62.XXXXXXXXXXX  -  2014-3-27 @ 11:39:57
6  72.XXXXXXXX  -  2014-3-27 @ 11:40:02
7  64.XXXXXXXXXX  -  2014-3-27 @ 11:40:04
8  37.XXXXXXXXXX  -  2014-3-27 @ 11:40:20
9  194.XXXXXXXXXX  -  2014-3-27 @ 11:47:18
10 194.XXXXXXXXXX  -  2014-3-27 @ 11:47:20
11 89.XXXXXXXXX  -  2014-3-27 @ 11:49:50
12 65.XXXXXXXXXX  -  2014-3-27 @ 11:49:54
13 92.XXXXXXXXX  -  2014-3-27 @ 11:49:56
14 37.XXXXXXXXXX  -  2014-3-27 @ 11:51:20
15 94.XXXXXXXXXX  -  2014-3-27 @ 11:51:24
16 62.XXXXXXXXXXX  -  2014-3-27 @ 11:51:26

The sting

However, Microsoft may yet have a trick up its sleeve: customers must provide a phone number and credit card details in order to register for the trial. Whilst the credit card details could have been stolen in a previous phishing attack, physical access to a phone is required in order to register an account. This may prove to be the fraudsters' downfall — in serious cases, information gathered from the fraudsters mobile phone could be used as evidence subject to the phone company's cooperation and local police involvement.

Netcraft's Domain Registration Risk service can be used to pre-empt fraud by highlighting domains or subdomains that are deceptively similar to legitimate websites run by banks and other institutions that are commonly targeted by fraudsters.

Fraudsters modify eBay listings with JavaScript redirects and proxies

Fraudulent classified ads posted on eBay have been exploiting an opportunity to establish convincing attacks against potential car buyers. Simply viewing one of the sneaky eBay ads causes the victim's browser to instead request the same listing via an intermediate server, which subtly modifies the content of the page to the fraudster's advantage.

Similar to a man-in-the-middle attack, the modifications are performed on-the-fly by a web server located in the US.

1. Victim browses to one of the fraudulent listings on eBay.co.uk; 2. eBay returns the listing to the victim's browser; 3. The fraudulent listing automatically redirects the browser to the attacker's website, passing the eBay item number to a PHP script; 4. The attacker's website uses the item number to fetch the same listing directly from ebay.co.uk; 5. eBay returns the listing to the attacker's website; 6. The attacker modifies the real eBay page before returning it to the victim's browser.

When a customer views one of the fraudulent ads on eBay, specially crafted JavaScript embedded within the item's description will automatically redirect the victim's browser to the attacker's website. The eBay item number is passed to a PHP script on the attacker's site, which allows it to fetch the same listing from ebay.co.uk before delivering a slightly altered version to the victim.

Most customers would not expect their browser to end up on a different website by merely viewing a listing on the real eBay website, which makes this attack dangerously effective. Additionally, because the modified listing looks extremely similar to the real thing (and displays the item they were expecting to see), it is likely that many victims would have no cause to suspect that the bogus content is being served from a completely different website. Although there are still a few small clues for the wary, this apparent weakness in the eBay platform is certainly much easier to exploit than a completely undetectable man-in-the-middle attack.


The fraudulent sites can also display legitimate eBay listings, changing the seller's contact details on-the-fly.
Images are sourced directly from eBay's own web servers.

Interestingly, the only significant differences on the modified page are that the Email the seller and the Ask a question links have been replaced with different links which send an email directly to EBAY@REGOWNER.CO.UK. On the real eBay website, these parts of the page cannot be altered because the item description is displayed within an iframe, making any JavaScript within the description unable to directly alter the contents of the parent window. By encouraging victims to immediately establish an email dialogue outside of the eBay website, the fraudster can attempt to secure money through non-reversible payment methods without eBay being able to monitor even the initial communication.

Victims are unlikely to be spooked by having to deal directly with the seller. While eBay's terms and conditions forbid anyone to buy or sell outside eBay, this applies only to its auction-style and Buy-It-Now listing formats. This scam makes use of eBay's newer classified ad listing format, where a purchase can only be carried out by dealing directly with the seller. In these cases, the victim would not be covered under eBay's buyer protection policy, nor would they be able to leave negative feedback which might alert other potential victims.

The fraudulent listings used in these attacks are posted from compromised eBay accounts, which allows the fraudster to piggyback on the trustworthiness and reputation of established sellers. If these compromised accounts have accrued lots of positive feedback from previous auctions, then this will also serve to leverage the trust of potential victims much more than a brand new account possibly could.

This type of attack is rather subtle considering the other opportunities that could have been exploited by the fraudster. Most obviously, the fraudster could have attempted to steal login credentials by presenting a spoof login form, but clicking on the Buy it now or Make offer buttons, or the My eBay menu item, actually directs the victim to the real eBay login page instead. However, the subtle changes that are made are the only ones necessary for these types of listings — when it is possible to score thousands of pounds with a single fraudulent sale via email, perhaps it is not worth attracting undue attention by also phishing for account details.


A fragment of JavaScript used by one of the fraudulent eBay listings.
This automatically causes a browser to display the modified content from the fraudster's server, without any user interaction.

The man-in-the-middle scenario is made possible by the inclusion of arbitrary JavaScript in the fraudulent listings. eBay's HTML and JavaScript policy explicitly prohibits the use of JavaScript to redirect a user from eBay to another webpage, but this rule is clearly being flouted. Accounts may be suspended for breaching the guidelines in this policy, which is another reason why it is common to see fraudulent listings being posted from compromised eBay accounts – whether or not these accounts get permanently suspended is largely inconsequential to the attacker.

Banning nefarious JavaScript through policy alone is rather ineffective, as fraudsters aren't going to mind breaking the rules. Given the potential for misuse, the lack of sufficient technical measures to prevent malicious scripts being embedded within an eBay listing poses a security risk, and the fraudulent listings posted on eBay over the past week demonstrate that this issue can be exploited rather effectively.

Because the description of an eBay listing is displayed within an iframe, the attack relies on being able to use a hyperlink to change the location of the parent window. This could be prevented by using HTML5's sandboxing features, which would cause a hyperlink with a target="_top" attribute to do nothing. The framed content would only be able to navigate within itself and not change the contact details in the surrounding top-level parent.

Although the fraudulent listings are eventually deleted by eBay, the same fraudster keeps coming back for more. Buster Jack — who regularly reports such scams to eBay — noted a similar attack by the same fraudster more than a week ago, which presented the modified content via the yugoslavic.info domain. In terms of value, Jack told Netcraft that the used car market is the most serious area of fraud on eBay.

Within the past week, Netcraft has blocked more than 20 other websites that the same fraudster had been using to modify the content of eBay listings. All of these sites used the .info top-level domain, shared the same IP address, and were hosted by HostGator in the United States.

The Scamwarners forum has documented similar cases of suspected fraudulent activity on the car trading website Autotrader. Here, the same fraudster has attempted to get potential buyers to make contact via various email addresses under his regowner.co.uk domain, rather than by phone or via the Autotrader website. The affected listings have since been removed from the Autotrader website, but the regowner.co.uk domain is still operational and able to receive email. The domain name itself lends authority to the scam by pretending it has something to do with the registered owner of a vehicle, and the local part of the email address (the part before the @ symbol) was the same as the car's number plate, such as KX60YSJ@REGOWNER.CO.UK.

The regowner.co.uk domain was registered with eNom on 27 March and currently points to a holding page hosted by Arvixe in the UK. Despite the domain's WHOIS registration type being set to "UK Individual", the registrant's address is purportedly in the United States. The .info domains used by the man-in-the-middle scripts were also registered last month, using an address in London.

Heartbleed: Why aren’t certificates being revoked?

Netcraft's site reports now make it easy to see which websites have or have not revoked their SSL certificates in response to the Heartbleed bug.

Around 17% of all trusted SSL web servers were vulnerable to the Heartbleed bug when it was publicly disclosed earlier this month. The bug made it possible to steal a server's private keys, thus allowing unauthorised parties to impersonate an affected website using its own SSL certificate. Consequently, around a quarter of the 500,000+ potentially-compromised certificates have already been reissued to date, but despite the importance of doing so, relatively few of these have also been revoked.

Some website administrators quickly responded to the Heartbleed bug by upgrading OpenSSL and issuing new SSL certificates, but issuing new certificates alone is not enough. Despite the difficulties involved in online revocation checking during a man-in-the-middle attack, the previous, possibly-compromised certificates must be revoked. Revocation checking can still be effective in some cases, especially when the revocation is included in Google's CRLSets.

For example, Yahoo had several high-profile websites which were vulnerable to the Heartbleed bug, and if the SSL certificates' private keys were compromised, they still are. Although the underlying OpenSSL vulnerability was quickly fixed on Yahoo's servers, it was not quick enough to prevent the vulnerability being exploited to reveal some of the email addresses and passwords used by Yahoo users. Yahoo has since reissued the affected certificates, and with the possibility of a key compromise, it would also have been sensible for Yahoo to revoke the old ones — but they have yet to do so.

Netcraft's site report for https://login.yahoo.com shows that the site offered the Heartbeat TLS extension prior to the Heartbleed disclosure, but is now using a new certificate. However, the new Heartbleed revocation section shows that the certificate previously used on login.yahoo.com has not yet been revoked. This means that anyone who uses Yahoo Mail, Yahoo Messenger, Flickr – and anything else which uses Yahoo's single sign-on mechanism – could still be vulnerable to man-in-the-middle attacks until it is revoked, or if not revoked, until February 2015.

Unfortunately, even when a certificate has been revoked, there is no guarantee that it cannot still be used to carry out man-in-the-middle attacks. If the attacker is also able to hijack OCSP requests, then he can exploit a browser's "soft-fail" approach to revocation checking, where a failed request will cause the browser to assume that the certificate is still good. CRLs (and Chrome's CRLSets) potentially offer slightly better protection under these circumstances, as the revocation lists may have already been downloaded while the browser was connected to a trusted network.

Yahoo is not alone in failing to revoke all of the certificates it reissued in response to the Heartbleed bug. At the time of writing, other companies in the same boat include Twitter, LinkedIn, Facebook, Apple, FedEx, PayPal and American Express, as well as the Schneier on Security blog. Many of these websites use Akamai's content distribution network, which was previously vulnerable to Heartbleed.

But why haven't all sites revoked their potentially compromised certificates? Some believe that online revocation checking is useless, some may not want to incur the cost of revoking a certificate, and many others may simply not realise (or believe) it necessary. Nevertheless, anybody who reissued a certificate in response to the Heartbleed bug presumably accepted there being some risk of the previous certificate being misused, in which case there is little justification for not revoking the old certificate. Administrators may want to delay revoking certificates to ensure that the new certificate has been fully deployed, but arguably, certificate authorities should not allow the delay between reissuance and revocation to stretch to several weeks.

Certificate revocation: Why browsers remain affected by Heartbleed

More than 80,000 SSL certificates were revoked in the week following the publication of the Heartbleed bug, but the certificate revocation mechanisms used by major browsers could still leave Internet users vulnerable to impersonation attacks. Little has changed since Netcraft last reported on certificate revocation behaviour.

Why is revocation necessary?

The Heartbleed bug made it possible for remote attackers to steal private keys from vulnerable servers. Most web server access logs are unlikely to show any evidence of such a compromise, and so certificates used on previously-vulnerable web servers should be replaced without delay.

However, even if the certificate is replaced, the secure site could still be vulnerable. If the pre-Heartbleed certificate had been compromised, it will remain usable by an attacker until its natural expiry date, which could be years away. A correctly positioned attacker, with knowledge of the old certificate's private key and the ability to intercept a victim's internet traffic, can use the old certificate to impersonate the target site.

Certificate authorities can curtail the lifetime of the compromised certificate by revoking the certificate. In principle, a revoked certificate should not be trusted by browsers, which would protect users from misuse of the certificate. The realities of revocation behaviour in browsers, however, could leave some internet users vulnerable to attack with compromised certificates.

The Heartbleed bug is currently the largest cause of certificate revocations, but other reasons for revoking certificates can include the use of weak signature algorithms, fraudulent issuance, or otherwise breaching the requirements laid out by the CA/Browser Forum.

How does revocation checking work?

There are two main technologies for browsers to check the revocation status of a particular certificate: the Online Certificate Status Protocol (OCSP) and Certificate Revocation Lists (CRLs). OCSP provides real-time revocation information about an individual certificate from an issuing certificate authority, whereas CRLs provide a list of revoked certificates which is typically retrieved by clients less frequently.

Of the major browsers, only Internet Explorer and Opera behave correctly in a wide variety of revocation scenarios, including where end-entity and intermediate certificates had been revoked only via a CRL or only via OCSP. The remaining browsers — Google Chrome, Safari, and Firefox — all have less consistent behaviour when checking the revocation status of SSL certificates.


Firefox blocks access to certificates which have been revoked via OCSP.

OCSP, the more recent standard, is effectively the revocation method of choice on the internet: providing the URL to a CRL in individual certificates is optional in the Baseline Requirements, and only Opera and Internet Explorer consistently check them when OCSP is not available. The latest version of Firefox removed the last vestiges of CRL checking: previously CRLs were checked only for EV certificates when OCSP failed.

Although CRLs have some disadvantages — their size for one — they do offer some key advantages over OCSP: CRLs can be downloaded ahead of time on a trusted network and, unlike OCSP, CRLs do not reveal which sites you are visiting to the certificate authority. Google's decision to disable OCSP checking by default was also partly due to these privacy concerns.

OCSP stapling is an alternative approach to distributing OCSP responses. By including a recent OCSP response in its own TLS/SSL handshake, a website can remove the need for each visitor to make a separate connection to the certificate authority. As well as improving performance, stapled responses remove the privacy concerns surrounding standard OCSP leaking user IPs to certificate authorities. However, only 24% of all SSL certificates found in the most recent Netcraft SSL survey were used on websites that stapled an OCSP response.

Google has shunned the traditional methods of revocation: whilst Chrome does check the status of EV certificates, revocation checking is not enabled by default for any other type of certificate. Instead, Chrome uses its own updating mechanism to maintain an aggregated list of revoked certificates gathered by crawling CRLs. This is a subset of all revocations and is intended to cover only the most important.

Is revocation checking useful for certificates potentially compromised by Heartbleed?

As explained by Adam Langley, online revocation checking can easily be blocked if the compromised certificate is being used in a man-in-the-middle attack. An attacker able to intercept traffic to the targeted website will likely also be able to block OCSP requests. If the victim is using a browser which does not hard-fail (which is the default setting of all major browsers) when an OCSP response isn't received, the attacker will be able to use a revoked certificate as normal.

However, the same logic does not apply to CRLs: if the CRL was downloaded earlier when on a trusted network, a revoked certificate used in a man-in-the-middle attack will not be trusted. This requires the certificate to have been revoked before the CRL was downloaded; however, many CRLs can be cached for a significant length of time (up to 10 days in the Baseline Requirements). Although, if a new CRL is needed, its download can be blocked just as effectively as OCSP's can be. When CRLs are used, an attacker cannot rely on the certificate passing validation: a subset of users, those with cached CRLs, will be prevented from continuing on the attacker's site. The same logic also applies to Google's CRLSets, including the ability to block updates.

As such, despite the difficulties of revocation checking in the MITM scenario, it is still critical for site owners to revoke certificates. If the certificate is revoked, an attackers job is made that much more difficult: he must chose sites with certificates issued without a CRL distribution point (which is permissible under the Baseline Requirements) or that are not covered by Google's CRLSets, and his victims must be using a browser that checks neither. Certificates that are not revoked are unlikely to ever be included in more effective revocation methods such as CRLSets.

Should I enable revocation checking in Chrome?

Whilst OCSP is easily blocked in man-in-the-middle attacks, if revocation checking is enabled, Chrome (on both Windows and Linux) will check CRLs for certificates that do not support OCSP. It is likely that you will have cached CRLs for websites you have visited recently — if you move onto an untrusted network, you will be protected by the CRLs that were downloaded earlier. Over 4% of currently valid certificates are only revocable by CRL, including login.skype.com. Unfortunately, for the majority of sites where OCSP is available CRLs will not be downloaded, any OCSP requests made can be blocked, and the attacker can continue as if the certificate is not revoked.

Perfect OCSP checks: A chicken and egg problem

By default, all browsers take the "soft-fail" approach to OCSP checks. A revoked certificate will be regarded as valid if the OCSP request fails. While this sounds like unsafe behaviour, browser vendors are reluctant to force a hard-fail approach because of the problems it can cause. For example, paid-for internet connections, such as WiFi hotspots or hotel room connections, that use captive portals are one of the major chicken-and-egg scenarios. Before a user can access the internet, he must visit a secure payment page, but this would fail because the OCSP responder used by the site's certificate cannot be reached until after he has paid. There are methods to resolve this problem, including OCSP stapling and less restrictive blocking; however, such solutions are unlikely to adopted quickly.


Firefox can be forced to use a hard-fail approach to OCSP checking, but this setting is not enabled by default.

It is critical that OCSP responders have 100% uptime, as any outage whatsoever could provide a window of opportunity to misuse compromised revoked certificates. Netcraft publishes a list of OCSP responder sites ordered by failures over the past day. Partly due to the reliability concerns, the Mozilla Foundation suggests that there is some way to go before a hard-fail approach can be enabled by default.

Despite the drawbacks of soft-fail OCSP checking, there are circumstances in which a soft-fail approach can still be useful. For example, it might be desirable to revoke a domain-validated certificate which had been issued to a deceptive domain name (e.g. paypol.com), or when a domain changes hands. In the absence of any man-in-the-middle attackers, soft-fail OCSP is likely to be effective.

Irrevocable certificates

Browsers that do not support CRLs, such as Firefox, are not able to determine whether or not the 4% of certificates without OCSP responder URLs have been revoked. Only if an OCSP response has been stapled to the TLS connection can such browsers check the revocation status. Given the majority of certificates (76%) are served without a stapled OCSP response, such certificates are effectively irrevocable for a large proportion of internet users. As a result, the compromised certificates can be misused for fraud up until their natural expiry dates. A smaller number of certificates fail to specify URLs for either method of revocation, which makes them completely irrevocable in all browsers which rely on these technologies.

It is likely that browser vendors will be forced to take additional steps to ensure that irrevocable certificates are correctly regarded as invalid. Such measures were taken in 2011, when Mozilla released new versions of Firefox which explicitly blacklisted some of the fraudulent certificates generated by the Comodo Hacker, even though the affected certificates had already been revoked by the issuer. One of the fraudulent certificates released to the public impersonated Firefox's addons site at addons.mozilla.org. Google's CRLSet gives it the ability to distribute such revocations without relying on any certificate authority to revoke the certificate.

Accenture was using a CRL-only Extended Validation certificate on its website at https://apps.accenture.com using a vulnerable version of OpenSSL (1.0.1e). The potentially compromised certificate was subsequently replaced with a new certificate issued on 14 April, and the previous certificate (serial number 0x0100000000013b03d6adfeff5c37) was revoked. The serial number was added to the CRL at http://crl.omniroot.com/PublicSureServerEV.crl. If an attacker had managed to compromise the private key used by the old certificate, he can continue impersonating apps.accenture.com with a seemingly valid SSL certificate until its natural expiry date in November 2014 for victims using browsers which do not check CRLs, which includes Firefox 28. The only indication that revocation checking has not been completed is the lack of the EV browser cues. This certificate is present in Google's CRLSet, and so Google Chrome users are protected against its misuse.

A currently deployed EV certificate without OCSP in Firefox 28 (left). The EV browser cues are not displayed in Firefox as the revocation status has not been checked. Internet Explorer (right), which has checked the revocation status on the CRL, does display the additional green bar with the company's name.

Apple's Safari web browser also does not perform any CRL revocation checks for Extended Validation certificates despite doing so for non-EV certificates. This behaviour may be based on the Baseline Requirements and the EV guidelines, which have mandated that EV certificates contain an OCSP responder URL for some time. As a consequence, the certificate previously used on apps.accenture.com is also irrevocable in Safari. In addition, despite making no revocation checks, Safari retains the EV browser cues rather than downgrading to standard SSL.

Problems revoking intermediate certificates

Digital certificates are verified using a chain of trust. At the top of the chain is the root CA's public key, which is built into the browser. The corresponding private keys can be used by the root CA to sign an intermediate certificate one step down the chain. At the very bottom of the chain is the certificate for the website itself, which is signed by the sub-CA whose intermediate certificate is immediately above the site's certificate. A single chain of trust can have multiple intermediate certificates chained together in order to form a path from the website's certificate to a trusted root.

An example of an SSL certificate's chain. This one is used by www.mcafeecustomerrewards.com.

Browsers must trust each level of the chain: all intermediate certificates in the chain must ultimately be signed by a root CA in order for the website's certificate to be trusted. Most root certificate authorities are understandably paranoid about the security of their private keys, and so root certificates are rarely compromised directly. Smaller certificate authorities, however, may not have as much funding or expertise, and may be more likely to suffer from security breaches which could result in the disclosure of an intermediate certificate's private key.

If the private key of a sub-CA's intermediate certificate is leaked, it has serious implications for the whole internet. A fraudster could use the certificate's private key to issue arbitrary publicly trusted certificates, essentially allowing him to impersonate any website on the planet. It is imperative that compromised intermediate certificates are immediately revoked, but it difficult to achieve this in practice.

For example, when a Firefox user visits www.mcafeecustomerrewards.com, a website which has a non-EV certificate, Firefox will only make an OCSP request for the website's certificate. This means that the revoked intermediate certificate (McAfee Public CA v1) will continue to be trusted by Firefox, and the only way to resolve this would be for Mozilla to release a new version of Firefox. The same behaviour is seen in Google Chrome unless revocation checking is enabled, as the intermediate certificate is not in Google's CRLSet. When Chrome has revocation checking turned on, the certificate is correctly marked as revoked.

    Serial Number: 55A1BA093A529CB41F12EB6A1FF71EF6
        Revocation Date: Oct  7 14:03:19 2013 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code:
                Cessation Of Operation
            Invalidity Date:
                Oct  7 14:03:09 2013 GMT

The entry for McAfee Public CA v1 in http://www.rsasecurity.com/products/keon/repository/certificate_status/RSA_Security_2048_v3.CRL.

www.mcafeecustomerrewards.com uses a certificate which has been signed by a revoked intermediate certificate (McAfee Public CA v1). Firefox displays the site without showing any warnings.

Google Chrome revocation bug

Although Google Chrome does not perform OCSP checks by default, it does perform them in the case of Extended Validation certificates (unless the certificate is already covered by the CRLSet). However, the Linux version of Google Chrome does not prevent access to sites using a revoked EV certificate when not covered by the CRLSet. Despite the browser sending an OCSP request and receiving a 'revoked' response, it mishandles the results and fails to block access. Instead, the EV browser cues (the green bar) is removed. Netcraft reported this apparent bug to Google in August 2013, but it was classed as low severity and has yet to be fixed on Linux.

The Windows version of Chrome (on left) behaves correctly and blocks access to a site with a revoked EV certificate. However, Chrome on Linux (on right) does not display any errors when a site uses a revoked EV certificate; it merely downgrades the UI from EV to standard SSL.

Where can we go from here?

Each of the currently available revocation methods has significant disadvantages: CRLs are potentially very large; OCSP can be blocked easily; and CRLSets are not intended to provide complete coverage. To those looking to move towards hard-fail, despite being far from pervasive, OCSP stapling could offer the answer. When combined with must-staple, currently an Internet draft, it would enable per-site, opt-in hard-fail behaviour. However, this solution is limited by the length of time (the Baseline Requirements limit the validity to 10 days) an attacker can use a cached 'good' OCSP response saved just before the certificate was revoked.

In the meantime, CRLSets, if they provided wider coverage, would be a more robust alternative to soft-fail OCSP checking. Mozilla is also looking to join Google by move towards a CRLSet-like mechanism for some of the revocation checking in Firefox.

Even soft-fail OCSP checking can be made more robust by removing any secure indicators (such as padlocks) when visiting a site without up-to-date revocation information.

Chrome users oblivious to Heartbleed revocation tsunami

In the aftermath of Heartbleed, it has become clear that revoking potentially compromised certificates is essential. On Thursday, CloudFlare announced it was reissuing and revoking all of its SSL certificates. The effects of CloudFlare's mass revocation are evident in a single Certificate Revocation List (CRL) belonging to GlobalSign, which grew by almost 134,000 certificates.

The vast number of CloudFlare certificates is due, in part, to the way in which it serves content over SSL. In order to work around the lack of support for Server Name Indication (SNI) in some older operating systems and mobile devices CloudFlare uses GlobalSign's Cloud SSL product. CloudFlare's SSL certificates make use of the Subject Alternative Name (SAN) extension, which allows an edge node to use a single certificate for multiple domains. When a new CloudFlare customer enables SSL, CloudFlare reissues an existing certificate with the new customer's domain added to the existing list of other customers' domains.

The number of certificates revoked per hour since 7th April. GlobalSign's OV CRL at http://crl.globalsign.com/gs/gsorganizationvalg2.crl and other CRLs have been separated.

As a result of CloudFlare's revocations, GlobalSign's CRL at http://crl.globalsign.com/gs/gsorganizationvalg2.crl has ballooned in size and now weighs in at 4.5MB. The CRL is hosted at CloudFlare itself but has nonetheless experienced some performance problems. However, the CRL's performance problems will not have had a significant effect on internet users, as most major browsers use OCSP in preference to CRLs and GlobalSign's OCSP responder did not have any performance problems.

Time to connect to http://crl.globalsign.com/gs/gsorganizationvalg2.crl from Pennsylvania

Time to connect to http://ocsp2.globalsign.com/gsorganizationvalg2 from Pennsylvania

However, most Google Chrome users are left in the dark, as Chrome performs neither type of check for non-EV certificates by default. Instead of conventional revocation checks, Google Chrome relies on an aggregated list of revocations, dubbed CRLSets, which are compiled by Google. The revocations from GlobalSign's CRL have not yet appeared in Google's CRLSets and hence Chrome users will not be warned if presented with a potentially compromised, but revoked, CloudFlare certificate.

The CRLSets deliberately do not cover all CRLs in an attempt to reduce the total size of the aggregated list. In effect, Google has traded the completeness of their revocation checking for a speed advantage over rival browsers as downloading CRLs or making OCSP requests imposes a performance penalty.

Google Chrome setting to enable revocation checking.

However, it is possible to configure Google Chrome to check for revocation. There is a checkbox in the Advanced settings menu to "Check for server certificate revocation".