DigitalOcean becomes the second largest hosting company in the world

DigitalOcean has grown to become the second-largest hosting company in the world in terms of web-facing computers, and shows no signs of slowing down.

The virtual private server provider has shown phenomenal growth over the past two-and-a-half years. First seen in our December 2012 survey, DigitalOcean today hosts more than 163,000 web-facing computers, according to Netcraft's May 2015 Hosting Provider Server Count. This gives it a small lead over French company OVH, which has been pushed down into third place.

Amazing growth at DigitalOcean

Amazing growth at DigitalOcean

DigitalOcean's only remaining challenge will be to usurp Amazon Web Services, which has been the largest hosting company since September 2012. However, it could be quite some time until we see DigitalOcean threatening to gain this ultimate victory: Although DigitalOcean started growing at a faster rate than Amazon towards the end of 2013, Amazon still has more than twice as many web-facing computers than DigitalOcean today.

Nonetheless, DigitalOcean seems committed to growing as fast as it can. Since October 2014, when we reported that DigitalOcean had become the fourth largest hosting company, DigitalOcean has introduced several new features to attract developers to its platform. Its metadata service enables Droplets (virtual private servers) to query information about themselves and bootstrap new servers, and a new DigitalOcean DNS service brought more scalability and reliability to creating and resolving DNS entries, allowing near-instantaneous propagation of domain names.

Other companies are also helping to fuel growth at DigitalOcean. Mesosphere created an automated provisioning tool which lets customers use DigitalOcean's resources to create self-healing environments that offer fault tolerance and scalability with minimal configuration. Mesosphere's API makes it possible to manage thousands of Droplets as if they were a single computer, and with DigitalOcean's low pricing models and SSD-only storage, it's understandable how this arrangement can appeal to particularly power-hungry developers.

In January, DigitalOcean introduced its first non-Linux operating system, FreeBSD. Although less commonly used these days, FreeBSD has garnered a reputation for reliability and it was not unusual to see web-facing FreeBSD servers with literally years of uptime in the past. In April, DigitalOcean launched the second version of its API, which lets developers programmatically control their Droplets and resources within the DigitalOcean cloud by sending simple HTTP requests.

DigitalOcean added a new Frankfurt region in April 2015.

DigitalOcean added a new Frankfurt region in April 2015.

More recently, DigitalOcean introduced a new European hosting region in Frankfurt, Germany. This is placed on the German Commercial Internet Exchange (DE-CIX), which is the largest internet exchange point worldwide by peak traffic, allowing Droplets hosted in this region to offer good connectivity to neighbouring countries. (An earlier announcement of an underwater Atlantis datacenter sadly turned out to be an April Fool's joke, despite the obvious benefits of free cooling).

Even so, Amazon still clearly dwarfs DigitalOcean in terms of variety of features and value-added services. Notably, Amazon offers a larger variety of operating systems on its EC2 cloud instances (including Microsoft Windows), and its global infrastructure is spread much wider. For example, EC2 instances can be hosted in America, Ireland, Germany, Singapore, Japan, Australia, Brazil, China or even within an isolated GloudGov US region, which allows US government agencies to move sensitive workloads into the cloud whilst fulfilling specific regulatory and compliance requirements. As well as these EC2 regions, Amazon also offers additional AWS Edge Locations to be used by its CloudFront content delivery network and its Route 53 DNS service.

Yet, as well as its low pricing, part of the appeal of using DigitalOcean could lie within its relative simplicity compared with Amazon's bewilderingly vast array of AWS services (AppStream, CloudFormation, ElastiCache, Glacier, Kinesis, Cognito, Simple Workflow Service, SimpleDB, SQS and Data Pipeline to name but a few). Signing up and provisioning a new Droplet on DigitalOcean is remarkably quick and easy, and likely fulfils the needs of many users. DigitalOcean's consistent and strong growth serves as testament to this, and will make the next year very interesting for the two at the top.

November 2015 Web Server Survey

In the November 2015 survey we received responses from 902,997,800 sites and 5,539,129 web-facing computers. This reflects a monthly gain of 24.7 million sites, and 47,200 computers.

This month's website growth was dominated by Apache, which gained nearly 31 million sites – more than eight times as many as nginx, which had the second largest growth amongst the top three. Helped by a loss of 22 million Microsoft-powered websites, Apache's market share has increased to 37%, with its lead over Microsoft more than doubling to 9.9 percentage points.

This sizeable shift in market shares can be mostly attributed to 17 million websites whose domain names became due for renewal. This caused them to be moved from IIS servers to a set of domain holding pages hosted on Apache servers.

Despite Apache also having the greatest growth in web-facing computers this month, with an increase of 23,405 computers, its market share grew by just 0.03 percentage points. In contrast, nginx's similar growth of 21,004 computers increased its market share by 0.27 percentage points.

The number of web-facing computers using each vendor's software serves as a more stable metric, due in part to the cost of provisioning machines. Conversely, website counts are more prone to large fluctuations, as a single computer can serve countless websites at little incremental cost.

Demonstrating this disconnect, Tengine – an nginx fork developed by Alibaba – made a significant contribution to the overall growth in hostnames despite being used on only 5,100 web-facing computers. While the number of sites using this server grew by nearly 30%, rising to 42 million, the number of active sites using Tengine actually fell by 5%.

nginx continues to increase its presence amongst the top million sites. It now powers an additional 2,708 of the top sites, with Apache, Microsoft and Google each losing out to make room. nginx also showed the largest active sites growth in November, growing by 1.6 million (+6.2%) to reach a total of 27.9 million.

Since the launch of Yunjiasu ("fast cloud") in December 2014, more than 2.5 million sites (and 108,000 active sites) are now being served by a modified version of nginx called yunjiasu-nginx, making it the 10th most commonly used web server software by hostnames. Most of this growth has taken place in the last few months, with the total number of sites using this server growing by more than 5x since August.

Yunjiasu is operated by Chinese search engine giant Baidu, in collaboration with CloudFlare, who are responsible for the similar cloudflare-nginx server that is currently used by more than 5 million sites. Baidu's Yunjiasu offers the same features and functionality as CloudFlare (CDN, DNS, DDoS protection, etc.), but it is optimised for performance and regulatory controls within China.

By combining Baidu's network of 17 mainland China data centers with CloudFlare's 47 data centers outside of China, it is possible to start addressing some of the performance issues that have been dampening the appeal of Chinese hosting companies. For example, the largest hosting company in China, Aliyun, only allows its customers to host websites within China, and although it provides its own CDN service, all of the nodes are also within China. Websites that are hosted in China, and available across the combined CloudFlare/Baidu network, will benefit from much greater availability and faster load times from outside of China. Symmetrically, websites that are hosted outside of China will load faster and become much more available within China.

One of the first customers to be served across Baidu's network was TechCrunch, whose local Chinese edition ( was previously only available about 50% of the time within mainland China. CloudFlare claims that it now achieves nearly 100% availability, with an average page load time of 2.5 seconds rather than 17. CloudFlare customers must explicitly opt in to enjoy the performance benefits of the China network: To overcome technical, economic and regulatory issues, Baidu operates all services within China, while CloudFlare operates all of those outside, and by default, no CloudFlare customer traffic will pass through the China network.

Total number of websites

Web server market share

DeveloperOctober 2015PercentNovember 2015PercentChange
Continue reading

January 2015 Web Server Survey

In the January 2015 survey we received responses from 876,812,666 sites and 5,061,365 web-facing computers.

This is the lowest website count since last January, and the third month in a row which has seen a significant drop in the total number of websites. As was the case in the last two months, the loss was heavily concentrated at just a few hosting companies, and a single IP address that was previously hosting parked websites was responsible for over 50% of the drop.

Microsoft continues to be impacted most by the decline. Having overtaken Apache in the July 2014 survey their market share now stands at just 27.5%, giving Apache a lead of more than 12 percentage points.

Microsoft's decline seems far less dramatic when looking at the number of web-facing computers that use its server software. A net loss of 6,200 computers this month resulted in its computer share falling by only 0.28 percentage points, while Apache's went up by 0.18 to 47.5%.

These losses included many sites running on Microsoft IIS 6.0, which along with Windows Server 2003, will reach the end of its Extended Support period in July. Further abandonment of these platforms is therefore expected in the first half of this year, although Microsoft does offer custom support relationships which go beyond the Extended Support period.

Apache made an impressive gain of 22,000 web-facing computers this month. Half of this net growth can be attributed to the Russian social networking company V Kontakte, which hosts nearly 13,000 computers. Almost all of these were running nginx last month, but 11,000 have since defected to Apache, leaving less than 2,000 of V Kontakte's computers still using nginx.

OVH is still the second largest hosting company in terms of web-facing computers (although DigitalOcean is hot on its heels), but demand for its own relatively new .ovh top-level domain appears to be waning. Last month, we reported that the number of sites using the new .ovh TLD had shot up from 6,000 to 63,000. These sites were spread across just under 50,000 unique .ovh domains, and the number of domains grew by only 2,000 this month.

Only the first 50,000 .ovh domains were given away for free, while subsequent ones were charged at EUR 0.99. Despite being less than a third of the planned usual price of EUR 2.99, this shows how even a tiny cost can have a dramatic impact on slowing down the uptake in domain registrations.

Other new top-level domains which have shown early signs of strong hostname growth include .click, .restaurant, .help, .property, .top, .gifts, .quebec, .market and .ooo, each of which were almost non-existent last month but now number in their thousands.

The proliferation of new top level domains is evidently generating a lot of money for registrars and ICANN, but for some parties it has caused expenditure that was previously unnecessary. Take the new .hosting TLD for example: you would expect this domain to only be of interest to hosting companies, but US bank Wells Fargo has also registered some .hosting domains, including, and These domains are not used to serve any content, and instead redirect customers to Wells Fargo's main site at The sole purpose of registering these domains appears to be to stop any other party from doing so, which protects the bank's brand and prevents the domains being used to host phishing sites.

In a similar move, Microsoft has also registered several .hosting domains including,,,, and Browsing to any of these domains causes the user to be redirected to, which displays search results for the second-level string (i.e. "xbox", "windows", etc.).

Of course, with many other new TLDs continually popping up, brand protection becomes an increasingly costly exercise. Microsoft has also recently registered hundreds of other nonsensical domains which are used to redirect browsers to, such as,,,,,,,,,, and so on.

However, the race to register domain names is not always won by Microsoft — is a prime example of a domain that someone else got to first. This domain is currently offered for sale, highlighting the fact that it's not just ICANN and the registrars that stand to gain money from the influx of new TLDs.

Total number of websites

Web server market share

DeveloperDecember 2014PercentJanuary 2015PercentChange
Continue reading

December 2015 Web Server Survey

In the December 2015 survey we received responses from 901,002,770 sites and 5,579,077 web-facing computers, reflecting a loss of 2.0 million sites, but a gain of 39,900 computers.

Apache suffered the largest loss of 13.4 million sites, followed by Microsoft, which lost 5.0 million. A good part of this month's overall losses were caused by expired .xyz domains, which resulted in nearly 9 million .xyz websites disappearing from the internet. Despite the widespread losses caused by the demise of these websites, nginx managed to gain 7.1 million sites overall, which was the largest growth seen by any web server vendor.

The .xyz top-level domain was made available to the general public on 2 June 2014 and immediately received strong support from Network Solutions, which registered nearly 100,000 .xyz domains during the first ten days of operation. Controversially, Network Solutions gave away many .xyz domains for free to customers who already had the corresponding domain under the .com TLD. This was done on an opt-out basis, and the domains were only free for the first year, leaving some customers surprised when each domain became due for renewal at a cost of $38 this year.

Google's parent company, Alphabet Inc, is one of the most notable users of the .xyz TLD with the domain, while some of the other popular .xyz sites include adult sites and torrent search engines. The .xyz TLD has also proven reasonably popular with fraudsters: Netcraft found phishing sites on 150 .xyz domains throughout November 2015.

This month's changes have caused Apache's leading market share to fall by 1.41 points to 35.6%, while nginx's site share has increased to 17.4%. A little over a year ago, Microsoft was in the lead, but has recently been floating around in second place, currently 9.2 percentage points ahead of nginx, and 9.0 behind Apache.

As well as gaining the largest number of sites this month, nginx also showed the largest growth in terms of web-facing computers, growing by 17,000 to reach a total of 765,000. Despite their site losses, Apache and Microsoft also gained a reasonable number of web-facing computers (10,400 and 6,100), while Lighttpd and Google suffered small losses.

A relatively unknown web server, Safedog, was found serving nearly ten times as many websites as last month, making it now the 7th most commonly used web server software with 6.3 million hostnames. However, the number of web-facing computers with Safedog installed is very low – less than 300 – and nearly all of these are running the deprecated Windows Server 2003 operating system. All websites using this Chinese server software claim to be running Safedog 4.0.0, which appears to be a cloud security system.

2015 has been a turbulent year in terms of hostnames, with the total number of sites rising from 877 million in January, to 901 million in December, but dipping as low as 849 million in April. Apache has continued to lead the market throughout the year, with Microsoft following in second place, getting to within 4.1 percentage points of Apache's share in October. In web-facing computers, nginx has shown remarkably consistent growth in its market share, while both Apache and Microsoft have declined. nginx is now installed on 13.71% of all web-facing computers, compared with 11.03% at the start of the year, and its market share within the top million sites has also grown noticeably from 21.09% to 24.29%.

Total number of websites

Web server market share

DeveloperNovember 2015PercentDecember 2015PercentChange
Continue reading

January 2016 Web Server Survey

In the January 2016 survey we received responses from 906,616,188 sites and 5,753,264 web-facing computers, reflecting a modest increase of less than six million sites, but a significant gain of 174,000 computers.

Microsoft gained 22.5m sites (+9.40%), which has taken its market share up by 2.32 points. Meanwhile, Apache lost 16.4m sites, and nginx fell by 15.6m. Apache's market share is now less than 5 points ahead of Microsoft; this difference was more than twice as large just two months ago.

The web-facing computers metric is typically much more stable, but this month's overall gain of 174,000 computers is unusually large as a result of a 7.6% increase in the number of web-facing computers running Apache.

This large gain comprised of nearly 195,000 Apache computers, and the majority of these are Western Digital My Cloud personal storage devices. These consumer devices run web servers and can be accessed using public hostnames with a format similar to Consumers can remotely access their files via the My Cloud web application, a mobile app, or via third-party applications that make use of the relatively new My Cloud OS 3 platform.

Consumers can remotely access their files via the My Cloud web application (shown), or via a mobile app.

Consumers can remotely access their files via the My Cloud web application (shown), a mobile app, or third-party tools.

More than 240,000 of these hostnames point directly to a variety of consumer broadband connections, which is where the My Cloud devices are physically located.

Network Attached Storage (NAS) devices are rarely exposed to the internet on such a large scale, and so this provides some otherwise invisible insights into the usage of these particular devices. Although consumers do not have to enable the Cloud Access feature, the 240,000+ devices that are directly exposed to the internet are likely to be a fairly representative sample of all similar Western Digital devices.

Nearly half of the My Cloud devices that are exposed directly to the internet are located in the US, while the UK has the next largest share of 13%, and France follows with 6%. This suggests that nearly two-thirds of Western Digital's consumer NAS sales take place in these three countries alone.

As well as the My Cloud devices that are exposed directly to the internet, a further 273,000 hostnames resolve to fewer than 200 IP addresses hosted by Amazon AWS. These hostnames likely represent additional My Cloud devices that have been cloud-enabled using Relay mode. In this mode, requests bound for the device are relayed via the Amazon-hosted web service, which makes it possible for a consumer to gain remote access even when they are not able to set up port forwarding on their router.

However, whilst certainly convenient, exposing a My Cloud device to the internet (either directly or in relay mode) could undermine a consumer's security by revealing the device's internal IP address to the whole world. Each of the 500,000+ My Cloud devices that can be accessed via hostnames like also have corresponding DNS entries that reveal their local IP addresses:

$ host has address 78.72.xx.x
$ host has address

These "-local" DNS entries allow a remote attacker to discover the local IP address of a consumer's My Cloud device (in this case,, which would make it easier to carry out CSRF attacks against it. Even if the consumer has taken the precaution of changing the device's name so that his browser cannot reach it via the default local address (http://wdmycloud), it could still be reached by browsing directly to its local IP address. Devices that have not been updated recently might still be vulnerable to remote code execution via CSRF attacks.

The local IP address of the My Cloud device can also be used to infer the address of the consumer's broadband router, which may well be vulnerable to similar types of attack. Knowing some likely IP addresses of the router makes CSRF attacks much more feasible – for example, if the My Cloud device has an IP address of, the attacker could deduce that the router's IP address might be or, rather than any of the other 17+ million IANA-reserved private network addresses. A successful exploit against a vulnerable router could give an attacker full control over the router's settings, which could ultimately lead to data theft or financial losses through pharming attacks.

While the influx of these My Cloud devices has resulted in strong growth for Apache, nginx continued its steady progress by gaining a further 23,300 (+3.0%) web-facing computers. Apache's market share in terms of computers now stands at 47.9% (+2.0), while Microsoft lost 20,600 computers, contributing to its share falling to 27.1%. Despite maintaining the consistent growth it has demonstrated for several years, nginx also suffered a minor loss in share by virtue of Apache's exceptional growth.

Total number of websites

Web server market share

DeveloperDecember 2015PercentJanuary 2016PercentChange
Continue reading

Certificate revocation: Why browsers remain affected by Heartbleed

More than 80,000 SSL certificates were revoked in the week following the publication of the Heartbleed bug, but the certificate revocation mechanisms used by major browsers could still leave Internet users vulnerable to impersonation attacks. Little has changed since Netcraft last reported on certificate revocation behaviour.

Why is revocation necessary?

The Heartbleed bug made it possible for remote attackers to steal private keys from vulnerable servers. Most web server access logs are unlikely to show any evidence of such a compromise, and so certificates used on previously-vulnerable web servers should be replaced without delay.

However, even if the certificate is replaced, the secure site could still be vulnerable. If the pre-Heartbleed certificate had been compromised, it will remain usable by an attacker until its natural expiry date, which could be years away. A correctly positioned attacker, with knowledge of the old certificate's private key and the ability to intercept a victim's internet traffic, can use the old certificate to impersonate the target site.

Certificate authorities can curtail the lifetime of the compromised certificate by revoking the certificate. In principle, a revoked certificate should not be trusted by browsers, which would protect users from misuse of the certificate. The realities of revocation behaviour in browsers, however, could leave some internet users vulnerable to attack with compromised certificates.

The Heartbleed bug is currently the largest cause of certificate revocations, but other reasons for revoking certificates can include the use of weak signature algorithms, fraudulent issuance, or otherwise breaching the requirements laid out by the CA/Browser Forum.

How does revocation checking work?

There are two main technologies for browsers to check the revocation status of a particular certificate: the Online Certificate Status Protocol (OCSP) and Certificate Revocation Lists (CRLs). OCSP provides real-time revocation information about an individual certificate from an issuing certificate authority, whereas CRLs provide a list of revoked certificates which is typically retrieved by clients less frequently.

Of the major browsers, only Internet Explorer and Opera behave correctly in a wide variety of revocation scenarios, including where end-entity and intermediate certificates had been revoked only via a CRL or only via OCSP. The remaining browsers — Google Chrome, Safari, and Firefox — all have less consistent behaviour when checking the revocation status of SSL certificates.

Firefox blocks access to certificates which have been revoked via OCSP.

OCSP, the more recent standard, is effectively the revocation method of choice on the internet: providing the URL to a CRL in individual certificates is optional in the Baseline Requirements, and only Opera and Internet Explorer consistently check them when OCSP is not available. The latest version of Firefox removed the last vestiges of CRL checking: previously CRLs were checked only for EV certificates when OCSP failed.

Although CRLs have some disadvantages — their size for one — they do offer some key advantages over OCSP: CRLs can be downloaded ahead of time on a trusted network and, unlike OCSP, CRLs do not reveal which sites you are visiting to the certificate authority. Google's decision to disable OCSP checking by default was also partly due to these privacy concerns.

OCSP stapling is an alternative approach to distributing OCSP responses. By including a recent OCSP response in its own TLS/SSL handshake, a website can remove the need for each visitor to make a separate connection to the certificate authority. As well as improving performance, stapled responses remove the privacy concerns surrounding standard OCSP leaking user IPs to certificate authorities. However, only 24% of all SSL certificates found in the most recent Netcraft SSL survey were used on websites that stapled an OCSP response.

Google has shunned the traditional methods of revocation: whilst Chrome does check the status of EV certificates, revocation checking is not enabled by default for any other type of certificate. Instead, Chrome uses its own updating mechanism to maintain an aggregated list of revoked certificates gathered by crawling CRLs. This is a subset of all revocations and is intended to cover only the most important.

Is revocation checking useful for certificates potentially compromised by Heartbleed?

As explained by Adam Langley, online revocation checking can easily be blocked if the compromised certificate is being used in a man-in-the-middle attack. An attacker able to intercept traffic to the targeted website will likely also be able to block OCSP requests. If the victim is using a browser which does not hard-fail (which is the default setting of all major browsers) when an OCSP response isn't received, the attacker will be able to use a revoked certificate as normal.

However, the same logic does not apply to CRLs: if the CRL was downloaded earlier when on a trusted network, a revoked certificate used in a man-in-the-middle attack will not be trusted. This requires the certificate to have been revoked before the CRL was downloaded; however, many CRLs can be cached for a significant length of time (up to 10 days in the Baseline Requirements). Although, if a new CRL is needed, its download can be blocked just as effectively as OCSP's can be. When CRLs are used, an attacker cannot rely on the certificate passing validation: a subset of users, those with cached CRLs, will be prevented from continuing on the attacker's site. The same logic also applies to Google's CRLSets, including the ability to block updates.

As such, despite the difficulties of revocation checking in the MITM scenario, it is still critical for site owners to revoke certificates. If the certificate is revoked, an attackers job is made that much more difficult: he must chose sites with certificates issued without a CRL distribution point (which is permissible under the Baseline Requirements) or that are not covered by Google's CRLSets, and his victims must be using a browser that checks neither. Certificates that are not revoked are unlikely to ever be included in more effective revocation methods such as CRLSets.

Should I enable revocation checking in Chrome?

Whilst OCSP is easily blocked in man-in-the-middle attacks, if revocation checking is enabled, Chrome (on both Windows and Linux) will check CRLs for certificates that do not support OCSP. It is likely that you will have cached CRLs for websites you have visited recently — if you move onto an untrusted network, you will be protected by the CRLs that were downloaded earlier. Over 4% of currently valid certificates are only revocable by CRL, including Unfortunately, for the majority of sites where OCSP is available CRLs will not be downloaded, any OCSP requests made can be blocked, and the attacker can continue as if the certificate is not revoked.

Perfect OCSP checks: A chicken and egg problem

By default, all browsers take the "soft-fail" approach to OCSP checks. A revoked certificate will be regarded as valid if the OCSP request fails. While this sounds like unsafe behaviour, browser vendors are reluctant to force a hard-fail approach because of the problems it can cause. For example, paid-for internet connections, such as WiFi hotspots or hotel room connections, that use captive portals are one of the major chicken-and-egg scenarios. Before a user can access the internet, he must visit a secure payment page, but this would fail because the OCSP responder used by the site's certificate cannot be reached until after he has paid. There are methods to resolve this problem, including OCSP stapling and less restrictive blocking; however, such solutions are unlikely to adopted quickly.

Firefox can be forced to use a hard-fail approach to OCSP checking, but this setting is not enabled by default.

It is critical that OCSP responders have 100% uptime, as any outage whatsoever could provide a window of opportunity to misuse compromised revoked certificates. Netcraft publishes a list of OCSP responder sites ordered by failures over the past day. Partly due to the reliability concerns, the Mozilla Foundation suggests that there is some way to go before a hard-fail approach can be enabled by default.

Despite the drawbacks of soft-fail OCSP checking, there are circumstances in which a soft-fail approach can still be useful. For example, it might be desirable to revoke a domain-validated certificate which had been issued to a deceptive domain name (e.g., or when a domain changes hands. In the absence of any man-in-the-middle attackers, soft-fail OCSP is likely to be effective.

Irrevocable certificates

Browsers that do not support CRLs, such as Firefox, are not able to determine whether or not the 4% of certificates without OCSP responder URLs have been revoked. Only if an OCSP response has been stapled to the TLS connection can such browsers check the revocation status. Given the majority of certificates (76%) are served without a stapled OCSP response, such certificates are effectively irrevocable for a large proportion of internet users. As a result, the compromised certificates can be misused for fraud up until their natural expiry dates. A smaller number of certificates fail to specify URLs for either method of revocation, which makes them completely irrevocable in all browsers which rely on these technologies.

It is likely that browser vendors will be forced to take additional steps to ensure that irrevocable certificates are correctly regarded as invalid. Such measures were taken in 2011, when Mozilla released new versions of Firefox which explicitly blacklisted some of the fraudulent certificates generated by the Comodo Hacker, even though the affected certificates had already been revoked by the issuer. One of the fraudulent certificates released to the public impersonated Firefox's addons site at Google's CRLSet gives it the ability to distribute such revocations without relying on any certificate authority to revoke the certificate.

Accenture was using a CRL-only Extended Validation certificate on its website at using a vulnerable version of OpenSSL (1.0.1e). The potentially compromised certificate was subsequently replaced with a new certificate issued on 14 April, and the previous certificate (serial number 0x0100000000013b03d6adfeff5c37) was revoked. The serial number was added to the CRL at If an attacker had managed to compromise the private key used by the old certificate, he can continue impersonating with a seemingly valid SSL certificate until its natural expiry date in November 2014 for victims using browsers which do not check CRLs, which includes Firefox 28. The only indication that revocation checking has not been completed is the lack of the EV browser cues. This certificate is present in Google's CRLSet, and so Google Chrome users are protected against its misuse.

A currently deployed EV certificate without OCSP in Firefox 28 (left). The EV browser cues are not displayed in Firefox as the revocation status has not been checked. Internet Explorer (right), which has checked the revocation status on the CRL, does display the additional green bar with the company's name.

Apple's Safari web browser also does not perform any CRL revocation checks for Extended Validation certificates despite doing so for non-EV certificates. This behaviour may be based on the Baseline Requirements and the EV guidelines, which have mandated that EV certificates contain an OCSP responder URL for some time. As a consequence, the certificate previously used on is also irrevocable in Safari. In addition, despite making no revocation checks, Safari retains the EV browser cues rather than downgrading to standard SSL.

Problems revoking intermediate certificates

Digital certificates are verified using a chain of trust. At the top of the chain is the root CA's public key, which is built into the browser. The corresponding private keys can be used by the root CA to sign an intermediate certificate one step down the chain. At the very bottom of the chain is the certificate for the website itself, which is signed by the sub-CA whose intermediate certificate is immediately above the site's certificate. A single chain of trust can have multiple intermediate certificates chained together in order to form a path from the website's certificate to a trusted root.

An example of an SSL certificate's chain. This one is used by

Browsers must trust each level of the chain: all intermediate certificates in the chain must ultimately be signed by a root CA in order for the website's certificate to be trusted. Most root certificate authorities are understandably paranoid about the security of their private keys, and so root certificates are rarely compromised directly. Smaller certificate authorities, however, may not have as much funding or expertise, and may be more likely to suffer from security breaches which could result in the disclosure of an intermediate certificate's private key.

If the private key of a sub-CA's intermediate certificate is leaked, it has serious implications for the whole internet. A fraudster could use the certificate's private key to issue arbitrary publicly trusted certificates, essentially allowing him to impersonate any website on the planet. It is imperative that compromised intermediate certificates are immediately revoked, but it difficult to achieve this in practice.

For example, when a Firefox user visits, a website which has a non-EV certificate, Firefox will only make an OCSP request for the website's certificate. This means that the revoked intermediate certificate (McAfee Public CA v1) will continue to be trusted by Firefox, and the only way to resolve this would be for Mozilla to release a new version of Firefox. The same behaviour is seen in Google Chrome unless revocation checking is enabled, as the intermediate certificate is not in Google's CRLSet. When Chrome has revocation checking turned on, the certificate is correctly marked as revoked.

    Serial Number: 55A1BA093A529CB41F12EB6A1FF71EF6
        Revocation Date: Oct  7 14:03:19 2013 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code:
                Cessation Of Operation
            Invalidity Date:
                Oct  7 14:03:09 2013 GMT

The entry for McAfee Public CA v1 in uses a certificate which has been signed by a revoked intermediate certificate (McAfee Public CA v1). Firefox displays the site without showing any warnings.

Google Chrome revocation bug

Although Google Chrome does not perform OCSP checks by default, it does perform them in the case of Extended Validation certificates (unless the certificate is already covered by the CRLSet). However, the Linux version of Google Chrome does not prevent access to sites using a revoked EV certificate when not covered by the CRLSet. Despite the browser sending an OCSP request and receiving a 'revoked' response, it mishandles the results and fails to block access. Instead, the EV browser cues (the green bar) is removed. Netcraft reported this apparent bug to Google in August 2013, but it was classed as low severity and has yet to be fixed on Linux.

The Windows version of Chrome (on left) behaves correctly and blocks access to a site with a revoked EV certificate. However, Chrome on Linux (on right) does not display any errors when a site uses a revoked EV certificate; it merely downgrades the UI from EV to standard SSL.

Where can we go from here?

Each of the currently available revocation methods has significant disadvantages: CRLs are potentially very large; OCSP can be blocked easily; and CRLSets are not intended to provide complete coverage. To those looking to move towards hard-fail, despite being far from pervasive, OCSP stapling could offer the answer. When combined with must-staple, currently an Internet draft, it would enable per-site, opt-in hard-fail behaviour. However, this solution is limited by the length of time (the Baseline Requirements limit the validity to 10 days) an attacker can use a cached 'good' OCSP response saved just before the certificate was revoked.

In the meantime, CRLSets, if they provided wider coverage, would be a more robust alternative to soft-fail OCSP checking. Mozilla is also looking to join Google by move towards a CRLSet-like mechanism for some of the revocation checking in Firefox.

Even soft-fail OCSP checking can be made more robust by removing any secure indicators (such as padlocks) when visiting a site without up-to-date revocation information.