Certificate revocation and the performance of OCSP

Certificate revocation is a critical aspect of maintaining the security of the third-party Certificate Authority (CA) infrastructure which underpins secure communication on the internet using SSL/TLS. A certificate may be worth revoking when it has had its private key compromised, the owner of the certificate no longer controls the domain for which it was issued, or the certificate was mistakenly signed. Without the ability to revoke certificates, a CA has no direct means of marking a certificate as untrusted before the expiry of the certificate, which could be several years away. In particularly urgent cases a browser vendor may have the ability to block certain individual certificates, trusted roots, or intermediate certificates, but this is rarely performed and is not suitable for lower-risk issues where revocation is necessary but not urgent.

There are two main technologies for browsers to check the revocation status of a particular certificate: using the Online Certificate Status Protocol (OCSP) or looking up the certificate in a Certificate Revocation List (CRL). OCSP provides real-time revocation information about an individual certificate from an issuing CA, unlike CRLs which provide a list of revoked certificates and may be received by clients less frequently.

The graph below shows a comparison of the time taken for the TLS handshake, both with and without OCSP checking enabled. The data was collected using packet traces taken while using Firefox 20 on Linux from an IP address in the UK. Measurements were taken three times (each time with a fresh cache) after discarding an initial request.

The relationship between whether OCSP checking is enabled and the time taken to complete the TLS handshake is not straightforward. In order for the browser to display the "green bar" to distinguish an Extended Validation (EV) certificate, OCSP requests must be made for every certificate in the chain whereas in many browsers, if an OCSP request is made at all, intermediate certificates are not checked. The increased time taken for the TLS handshake when using an EV certificate can be attributed to Firefox's sequential OCSP checking behaviour. However, where an OCSP check can be performed within the round-trip time to the server — for example, if the OCSP responder is served via a content delivery network or CDN — the check does not dramatically affect the time taken for the TLS handshake. When both the web server and the OCSP responder are topologically close to the client, as is the case with www.globalsign.com, the short round-trip time to the server isn't sufficient to mask the the time taken to receive OCSP responses for both the web site's certificate and the intermediate certificate presented. The slight difference between Paypal and GlobalSign's performance can at least partially be attributed to the additional OCSP request made for GlobalSign: GlobalSign's certificate chain requires three OCSP requests whereas Paypal's requires just two.

Reliability of RapidSSL's OCSP responder — December 2012

Netcraft has extracted around 40 OCSP responder URLs from certificates seen in the Netcraft SSL server survey, and has been monitoring them since late November 2012. The performance and reliability of the services varies significantly: Symantec's VeriSign OCSP responder has had consistently solid reliability, only a handful of connections failed over a 4 month period; whereas, in the same period more than 6% of requests to one of StartCom's responders failed. The reliability and performance of StartCom's OCSP responders have improved significantly since the end of February 2013 when it switched to using Akamai. Geotrust, another Symantec brand, did not have as strong a performance as either Thawte or VeriSign — all three of GeoTrust’s OCSP servers were down for between 48 and 104 minutes in a single event. Performance and reliability is measured from 11 points spread around Europe and North America: outages require at least one failed response from all measurement nodes within the 15-minute measurement interval.

Shift in reliability and performance for StartCom — late February 2013

For those browsers performing a synchronous OCSP request during the TLS handshake, the performance of the OCSP responder is often crucial. Any delay in responding to the request may noticeably slow down the handshake. For example, comparing GlobalSign's CloudFlare-accelerated OCSP responder with Entrust's, you find that GlobalSign's responder is significantly faster than Entrust's which uses Akamai's CDN. However, despite GlobalSign's performance advantage, its reliability has been affected by a number of CloudFlare outages — since Netcraft began monitoring OCSP, GlobalSign's responders have had at least 45 minutes of downtime whereas Entrust has had none.

GlobalSign (blue) and Entrust (green) OCSP responder performance.

OCSP responses can be stapled to a response from a web server when negotiating the TLS handshake to avoid the need for the browser to make a secondary request to a third party server. CloudFlare has claimed that enabling OCSP stapling has led to a 30% speed improvement for HTTPS sites. OCSP stapling support is present in newer versions of nginx — an increasingly popular open source web server — as a result of a development project sponsored by GlobalSign, DigiCert, and Comodo. OCSP stapling is not supported in the most popular version of Apache, 2.2.x, nor is it supported in current versions of Firefox (although support is in the pipeline), so it must remain only part of the solution for the foreseeable future. Frustrated by some of the limitations of OCSP, some CAs have lent support to a proposed an alternative revocation method using short lived certificates.

Browser support for the both OCSP and CRLs is mixed: currently, Firefox does not automatically download the CRLs from trusted CAs, so Firefox users must rely on OCSP alone; Google uses a proprietary mechanism to distribute CRLs to users of Google Chrome which aggregates per-CA CRLs into a single update which is distributed using its automatic update channel. Many browsers default to a "soft-fail" approach, leaving users vulnerable to eavesdroppers able to block or tamper with OCSP traffic. For as long as the CAs running OCSP responders do not have a strong record for both the performance and the reliability of their OCSP responders, browsers will find it difficult to justify switching to synchronous "hard-fail" behaviour.

Updated 18/04/2013

Mt.Gox “victim of own success” as Bitcoins fall in value

After days of intense growth, Bitcoins peaked at an unprecedented value of $266 last night, shortly before a crash which saw some investors selling them for as little as $105.


Value of 1 Bitcoin (BTC) in USD, midday 10 April - midday 11 April 2013 BST. [Source: Mt.Gox]

The Bitcoin market showed signs of recovery the following morning, but started falling again during an outage at Mt.Gox, which handles the majority of all Bitcoin trade.

Mt.Gox announced on Facebook that last night's crash was not caused by a DDoS (distributed denial of service) attack, but rather as a result of increased trade and new users signing up. The increased trade caused the Mt.Gox trading system to lag, which caused panic amongst some investors who started "cashing out" their Bitcoins, further exacerbating the situation until the trade engine froze.

Mt.Gox also revealed that the number of trades had tripled in a 24 hour period, and the number of new accounts jumped from 60,000 in March to 75,000 in just the first few days of April. Around 20,000 accounts are now being created each day, which is not surprising, given the potential investment value that has become widely evident over the past few weeks.

One investor was fortunate enough to have sold nearly 70,000 Bitcoins ahead of the crash. These would have been worth more than $18 million if sold at the very peak of the market, which demonstrates just how remarkable the growth has been — less than 3 years ago, 10,000 Bitcoins were used to buy $25's worth of pizza.

Mt.Gox went down for a short period late this morning (Thursday) while it performed some system maintenance and added several new servers to its system; however, as soon as this maintenance was completed, Mt.Gox was subjected to another DDoS attack.

mtgox.com is hosted by Prolexic, a company specialising in DDoS protection and mitigation, whilst the read-only APIs on data.mtgox.com are served via CloudFlare's content delivery network.

Dynamically updating performance graphs of the most popular Bitcoin trading sites are available here.

Most Reliable Hosting Company Sites in March 2013

Rank Company site OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Datapipe FreeBSD 0.000 0.058 0.009 0.019 0.030
2 ServerStack Linux 0.000 0.026 0.051 0.103 0.103
3 iWeb Linux 0:00:00 0.005 0.079 0.066 0.134 0.134
4 GoDaddy.com Inc Windows Server 2008 0:00:00 0.005 0.092 0.069 0.303 0.617
5 Server Intellect Windows Server 2008 0:00:00 0.005 0.016 0.085 0.172 0.430
6 Swishmail FreeBSD 0:00:00 0.008 0.066 0.051 0.101 0.241
7 Kattare Internet Services Linux 0:00:00 0.008 0.148 0.126 0.252 0.520
8 Hyve Managed Hosting Linux 0:00:00 0.010 0.100 0.036 0.072 0.073
9 Pair Networks FreeBSD 0:00:00 0.013 0.186 0.059 0.121 0.461
10 www.cwcs.co.uk Linux 0:00:00 0.013 0.265 0.114 0.230 0.645

See full table

Datapipe was the most reliable hosting company in March 2013, with both the fastest average connection time and no failed requests. Even more impressive is its remarkable 100% uptime record, which now stretches back for more than 7 years, and its connection times are regularly among the fastest we see each month.

The second most reliable hosting company in March 2013 – also with no failed requests – was ServerStack. Since Netcraft started monitoring ServerStack in October 2012, their site has had an uptime record of 99.990%. The company's 100% uptime SLA offers 5% credit for every half hour of sustained downtime, although this excludes periods of scheduled maintenance and its only outage so far lasted just 24 minutes.

iWeb ranked third after failing to respond to only one request during the whole of March. This performance was closely followed by Go Daddy and Server Intellect, each of which also failed to respond to just one request, but demonstrated marginally slower connection times than iWeb. Go Daddy's appearance in fourth place came despite a series of distributed denial of service (DDoS) attacks against its European webhosting operations, based in the Netherlands, which caused some of its customers' websites to become temporarily unavailable.

The previous month's winner, Hyve Managed Hosting, ranked eighth this time with three failed requests, but demonstrated very good average connection and total response times. These metrics are purportedly taken into account by Google's search algorithms, resulting in better rankings. Hyve's customers can gain similar advantages by using its high speed cloud platform with "light-speed" disk access, which allow its virtual servers to outperform traditional dedicated servers.

Datapipe runs its website on FreeBSD, which was also used by two other top-ten hosting companies during March: Swishmail and Pair Networks. Two sites were using Windows Server 2008, while the remaining five – including ServerStack – used Linux.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

Most Reliable Hosting Company Sites in February 2013

Rank Company site OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Hyve Managed Hosting Linux 0:00:00 0.007 0.164 0.084 0.172 0.174
2 Kattare Internet Services Linux 0:00:00 0.007 0.111 0.103 0.207 0.455
3 Netcetera Windows Server 2012 0:00:00 0.010 0.025 0.071 0.143 0.286
4 Pair Networks FreeBSD 0:00:00 0.017 0.144 0.038 0.078 0.253
5 Datapipe FreeBSD 0:00:00 0.024 0.071 0.016 0.032 0.049
6 Hosting 4 Less Linux 0:00:00 0.024 0.064 0.059 0.120 0.170
7 XILO Communications Ltd. Linux 0:00:00 0.024 0.154 0.071 0.451 0.619
8 www.hostway.ro Linux 0:00:00 0.024 0.524 0.154 0.473 0.795
9 New York Internet FreeBSD 0:00:00 0.027 0.091 0.031 0.691 0.835
10 iWeb Linux 0:00:00 0.027 0.062 0.055 0.111 0.111

See full table

In its third month being publicly monitored by Netcraft, Hyve Managed Hosting had an almost perfect record: only two requests failed out of the 30,000 requests we made in February. www.hyve.co.uk is served by nginx, a web server well-known for its performance. Hyve's primary data centre is in Global Switch London 2, a well-located modern facility in London's Docklands, close to key business centres. Hyve specialise in Cloud, Dedicated, and Secure FTP hosting, with clients including British Airways, Tesco, and American Express.

Kattare Internet Services also had just two failed requests in February, but was ranked in second place by using the average connect time as the tie-breaker. Kattare — a Java specialist based in Oregon — has been monitored by Netcraft since October 2003. Kattare, named for Kättare (Swedish for "heretic"), is a keen advocate of open-source solutions including FreeBSD and Linux: more than 97% of the web-facing computers found at the hosting company are powered by Linux.

Netcetera, up from 8th place in January to 3rd in February, is the only hosting company with a site hosted on Windows in the top 10: the remainder, where known, are all powered by Linux or FreeBSD. Netcetera has data centres in London and the Isle of Man, a jurisdiction which welcomes online gambling, linked by a comprehensive network.

Datapipe, Hosting 4 Less, XILO, and Hostway Romania all had seven failed requests, split only by average connect time: Datapipe's impressive connect time, 16ms, is evidence of the benefits of their globally disperse hosting platform. February was only the second month where Hostway.ro has been in the top 10, only three months after their first appearance in the top 10 in November 2012.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

Most Reliable Hosting Company Sites in January 2013

Rank Company site OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Datapipe FreeBSD 0:00:00 0.000 0.022 0.016 0.033 0.051
2 Qube Managed Services Linux 0:00:00 0.000 0.025 0.025 0.049 0.050
3 www.uk2.net Linux 0:00:00 0.003 0.147 0.089 0.183 0.202
4 New York Internet FreeBSD 0:00:00 0.006 0.058 0.016 0.648 0.727
5 Kattare Internet Services Linux 0:00:00 0.006 0.236 0.089 0.180 0.382
6 www.logicworks.net Linux 0:00:00 0.009 0.039 0.022 0.354 0.403
7 Server Intellect Windows Server 2008 0:00:00 0.009 0.023 0.060 0.125 0.305
8 Netcetera Windows Server 2012 0:00:00 0.009 0.016 0.078 0.167 0.327
9 www.codero.com Linux 0:00:00 0.012 0.120 0.049 0.306 0.536
10 www.memset.com Linux 0:00:00 0.015 0.082 0.082 0.162 0.330

See full table

Datapipe had the most reliable website of all monitored hosting providers, responding to every single request made by Netcraft from its 11 monitoring points distributed across North America and Europe. Datapipe.net achieves such rapid average connection times — meaning that it often wins the top spot even when otherwise tied on failed requests — by serving content from the server topologically closest to the client. Datapipe's Stratosphere platform is available in five global data centres allowing its clients to benefit from similar performance.

Qube, ranked second this month, also with an exemplary record, with no failed requests recorded but a slightly longer average connection time. Qube has a number of notable clients, including both BetFair, an online gambling exchange, and blinkbox, a video streaming service in the UK, which trust it to provide a dependable network from its three data centres in London, New York, and Zurich.

UK2.net finished in third place, having just a single failed request. One of UK2.net's flagship brands, VPS.net, released a newly redesigned website and logo on the last day in January. At the same time, UK2.net also announced a set of new promises for VPS.net customers including a 99.9% SLA for unmanaged customers and 100% for those with managed services.

Server Intellect are joined in the top ten this month by a fellow Windows-based hosting provider, Netcetera, appearing in the top ten for the 57th time, but the first time since September 2012.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

Most Reliable Hosting Company Sites in December 2012

Rank Company site OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 ServerStack Linux 0:00:00 0.000 0.039 0.027 0.053 0.054
2 Swishmail FreeBSD 0:00:00 0.003 0.037 0.025 0.051 0.105
3 New York Internet FreeBSD 0:00:00 0.006 0.078 0.025 0.677 0.774
4 Server Intellect Windows Server 2008 0:00:00 0.006 0.035 0.066 0.132 0.328
5 Datapipe FreeBSD 0:00:00 0.009 0.102 0.015 0.032 0.049
6 Pair Networks FreeBSD 0:00:00 0.009 0.092 0.041 0.087 0.294
7 Virtual Internet Linux 0:00:00 0.009 0.072 0.061 0.182 0.321
8 www.uk2.net Linux 0:00:00 0.009 0.121 0.066 0.134 0.220
9 www.codero.com Linux 0:00:00 0.009 0.183 0.086 0.370 0.747
10 ReliableServers.com Linux 0:00:00 0.016 0.206 0.027 0.059 0.065

See full table

Serverstack had the most reliable hosting company site during December, responding to every request from our monitoring system. We have only been monitoring Serverstack for three months, but it has quickly established itself as one of the hosting company sites with the fewest failed requests over that period, despite being located in the area affected by Hurricane Sandy.

Swishmail (second), New York Internet (third), Datapipe (fifth) and Reliable Servers (tenth) are also hosted within the area in which Hurricane Sandy made landfall and the presence of five such affected companies in the top ten reinforces Datapipe founder Robb Allen’s assertion that the recent history of the US North East with events including grid blackouts, Hurricane Irene and the 9/11 attacks has helped improve the resilience of the internet connectivity and hosting industry in that area.

December saw New York Internet (third) named NJBIZ's "Emerging Business of the Year" for 2012. NJBIZ profiled New York Internet's New Jersey datacentre in 2011, and praised the company's renovation and retrofitting of an older property in order to accommodate modern technology.

December's top ten list is dominated by FreeBSD and Linux, with the exception of Windows specialists Server Intellect (fourth) who have the only site running Windows. Server Intellect, which now offers Windows Server 2012 as standard on all dedicated and cloud servers, was second last month and regularly features among the top ten most reliable hosting company sites.

During December we added a new performance measurement point hosted at Webair's datacentre, in Amsterdam, bringing the total number of measurement points to 11.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.