US Government aiding spying… against itself

Partly as a consequence of the US Government shutdown, there are presently more than two hundred .gov websites using expired SSL certificates. Although the shutdown is expected to be a short term measure, the widespread use of expired certificates on .gov sites may cause long term harm. The US Government is effectively training its citizens and employees to click through SSL warnings, and once the users of a website treat SSL error messages as normal, attackers may be able to perform otherwise difficult man-in-the-middle attacks.

The situation is exacerbated by the behaviour of some mainstream browsers which do not faithfully warn the user of the most serious problem in scenarios where two or more errors are present.

An SSL error message presented on EV-enabled www.usaspending.gov in Google Chrome.

When an SSL error occurs, some browsers only display a single error message, sometimes not the most serious, or even a generic error message for all types of SSL error. An attacker can exploit this vulnerable browser behaviour on SSL sites with expired certificates to perform an almost seamless man-in-the-middle attack. By signing his own expired SSL certificate for a US government website, the SSL error message displayed for the attacker's SSL certificate is indistinguishable (in some browsers) from the error message produced by the real SSL certificate belonging to the US Government. Citizens accustomed to seeing the "expired" error message will happily proceed with a connection using the attacker's expired (and untrusted) certificate, unwittingly communicating with the attacker instead of the US Government.

By testing an expired certificate signed by an expired untrusted issuer, Netcraft found that whilst some browsers are vulnerable, Internet Explorer is not as it correctly displays both error messages. Google Chrome on Windows and OS X displays the more serious error message but does not display a warning about the expiry. All other tested browsers displayed either a generic error message or did not mention that the issuing CA is not widely trusted. Generic error messages are dangerous if they hide the severity of the SSL error from the user: a change in the type of the SSL error (from expiry to an untrusted issuer) will not be noticed. The tested website contained in the screenshots below is not on a .gov domain, but demonstrates browser behaviour with an untrusted and expired CA certificate with an expired end-entity certificate.

Google Chrome displaying an error message for an expired SSL certificate issued by an untrusted CA. From left to right: Windows, Mac OS X, Linux, and Android.

Google Chrome's behaviour is not consistent across its supported platforms: on Windows and Mac OS X it displays the most serious SSL error message, namely that an untrusted issuer has signed this SSL certificate. On Linux and Android, however, Google Chrome displays an error message about the expired certificate and does not mention the untrusted issuer. By reading the error message and accepting the risks of trusting an expired certificate, a user may unwittingly trust an SSL certificate that was not issued by a widely trusted CA.

Internet Explorer and Opera displaying an error message for an expired SSL certificate issued by an untrusted CA.

On Windows, Internet Explorer correctly presented both applicable error messages. Opera presented the more serious error message though only after viewing an additional dialogue box. Once a user is accustomed to accepting Opera's generic error message, any other type of SSL error on the same website is unlikely to be noticed. Internet Explorer, Google Chrome, and Opera all use Microsoft's CryptoAPI on the Windows platform which may explain their similar behaviour.

Firefox displaying an error message for an expired SSL certificate issued by an untrusted CA.

Firefox, which displays a generic error message for most SSL errors, has further information hidden by default. For an expired certificate issued by an untrusted and expired CA, Firefox's error message refers only to expired certificates (both the CA and end-entity certificates) and does not make any mention of the issuer not being a widely-trusted CA. Hidden details mean that a user having seen the same error message on the .gov website may not notice a change in the category of the SSL error message.

Safari (on OS X and iOS) displaying an error message for an expired SSL certificate issued by an untrusted CA.

Safari on OS X, like both Firefox and Opera displays a generic error message. If the message is expanded, Safari displays an error message based on the expired certificate and will also highlight the lack of trust in the issuer. Safari on iOS 7 displays a generic error message, "Not trusted", for many types of SSL certificate error — it is difficult to tell what is wrong with the SSL certificate without examining the certificate in detail.

Even without the "training" from the US Government, the click-through rate of different SSL messages has been demonstrated to be very high. For Firefox, which doesn't display full error messages by default, Akhawe and Porter Felt found SSL error messages were bypassed in 85% of cases: 87% for untrusted issuer messages and 81% for expired certificate errors. Paradoxically, in Google Chrome expired certificate error messages were dismissed 57% of the time whereas error messages for an untrusted issuer (the more serious problem) were dismissed in 82% of studied cases.

Phishers using CloudFlare for SSL

Some Content Delivery Networks (CDNs) enable fraudsters to deploy phishing attacks with valid SSL certificates. Not only does this make the fraudulent sites appear more credible, but they also benefit from the fast response times provided by the CDN.

A Turkish phishing site using CloudFlare (site has since been taken down)

The phishing site on odemerkezi.com is targeted at Turkcell customers — visitors to the phishing site are asked for their phone number, bank name, credit card details, and password. As CloudFlare's SSL feature is only available on paid accounts (which start at $20/month), the fraudster may have used an early victim's credit card to purchase the Pro plan.

Netcraft is currently blocking hundreds of phishing attacks which use CloudFlare's content delivery network, including some which use CloudFlare-provided SSL certificates. So far this year, Netcraft has blocked more than 2,000 phishing attacks using Cloudflare's infrastructure, of which approaching 200 used SSL.

CloudFlare's SSL certificates make use of the Subject Alternative Name (SAN) extension, which allows an edge node to use a single certificate for multiple domains. In the case of www.odemerkezi.com, the edge node presented a certificate which had a common name (CN) of "ssl2796.cloudflare.com", but also included the odemerkezi.com domain along with the domains of many other CloudFlare customers.

An SSL certificate used by a CloudFlare edge node server. It is valid for multiple domains belonging to its customers.

The multi-domain SSL certificates used by CloudFlare edge nodes are issued by GlobalSign. Rather than using Server Name Identification (SNI) — which would allow an individual certificate to be used for each website on a single IP address — CloudFlare uses GlobalSign's Cloud product to work around a lack of support for SNI in Internet Explorer on Windows XP and some mobile browsers. The two companies announced their partnership less than a year ago, and GlobalSign's own website uses CloudFlare, as do its OCSP and CRL services.

Some of the SSL phishing sites on CloudFlare that have been blocked by Netcraft have used deceptive domain names, such as paypal-germany.de.com, paypal-kundensicherheit.net and paypal-verifikation.com. Last month, a similarly deceptive domain name and SSL certificate issued by Network Solutions was used in a phishing attack against customers of Chase Bank.

Domain registrars and certificate authorities can reduce the likelihood of new domains and certificates being used for fraudulent activities. Netcraft's Domain Registration Risk service identifies domains which are deceptively similar to legitimate websites run by banks and other institutions that are commonly targeted by phishing attackers.

October 2013 Web Server Survey

In the October 2013 survey we received responses from 767,234,152 sites, an increase of 28.2M.

Apache experienced another significant loss, 1.8M hostnames, and saw its market share drop to 45% — the lowest it has been for over 15 years. The last time Apache's market share stood at 45% was in January 1998, at which time it served 830k hostnames. Despite gaining 18M new sites this month, 16M sites previously served by Apache no longer exist and a further 4M sites have moved to other web servers, including 1.5M sites to nginx and 420k sites to Microsoft. This loss was also seen in the SSL survey in which Apache lost 5.8k sites, but it continues to be the largest web server with 43% of the SSL market.

Microsoft saw the largest growth, gaining a net 16.5M hostnames. In May, it seemed inevitable that nginx would shortly match Microsoft in the number of sites served — there was just a single percentage point separating them. Microsoft’s growth has since accelerated however, and Microsoft now stand almost 5 percentage points above nginx. The next version of Microsoft's web server, IIS/8.5, has now been released to MSDN and TechNet subscribers ahead of the public release on the 18th October. More than 600 sites are already served by IIS/8.5, up 60% over last month.

Nginx gained 11.4M hostnames overall this month. Included in this was a net gain of 1.5M hostnames from Apache and a net loss of 340k hostnames to Microsoft. Among the top million, nginx gained 4.4k hostnames and the server is used by 15% of the top million sites.

All major server vendors suffered losses in the active sites metric resulting in a net decrease of 780k active sites, 664k of which belongs to Apache. In June 2000, when Netcraft started measuring active sites, 44% of sites were deemed active. The gap between the number of hostnames and active sites has been steadily increasing, and now the number of active sites account for less than a quarter of all sites.

ICANN signed 30 registry agreements for new top level domains in September. New gTLDs added this month include .reviews and .technology. General availability for domain registration of new gTLDs will begin in early 2014. The last new TLD to be seen in the wild is .post, a sponsored TLD first seen in October 2012. There were just 4 .post domains seen in the survey this month.





DeveloperSeptember 2013PercentOctober 2013PercentChange
Apache346,288,70646.86%344,408,38744.89%-1.97
Microsoft160,691,76321.74%177,216,29623.10%1.35
nginx111,680,07815.11%123,114,80016.05%0.93
Google34,806,5024.71%34,127,4824.45%-0.26
Continue reading

President Obama forgets to renew SSL certificate

At the start of the first US Government shutdown since 1996, an SSL certificate used on barackobama.com has expired. Issued by Go Daddy in September 2012, the SSL certificate for *.barackobama.com and barackobama.com was used by Organizing for Action, a non-profit grassroots organisation aligned with Obama's political policies. Whilst not directly associated with the US Government, the expiry of the SSL certificate for barackobama.com during a US Government shutdown is nonetheless a curious coincidence.

Warning in Google Chrome when visiting a website using the SSL certificate for *.barackobama.com.

Several SSL certificates controlled by the US Government expired today and are still being used — for example, the SSL certificates used on both ui.tn.gov and webmail.coop-uspto.gov have expired and may not be replaced any time soon. Furthermore, there are at least 30 US Government sites still using SSL certificates that are scheduled to expire before Friday.

SSL certificates expiring may be least of the problems for US Government websites, some websites have been taken offline: www.nasa.gov now redirects to notice.usa.gov.

Most Reliable Hosting Company Sites in September 2013

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Qube Managed Services Linux 0:00:00 0.000 0.125 0.066 0.134 0.134
2 Kattare Internet Services Linux 0:00:00 0.003 0.193 0.125 0.250 0.515
3 Hosting 4 Less Linux 0:00:00 0.003 0.179 0.128 0.251 0.634
4 www.uk2.net Linux 0:00:00 0.006 0.158 0.087 0.179 0.309
5 krystal.co.uk Linux 0:00:00 0.009 0.151 0.100 0.208 0.208
6 Netcetera Windows Server 2012 0:00:00 0.022 0.073 0.088 0.185 0.357
7 iWeb Linux 0:00:00 0.022 0.152 0.089 0.177 0.177
8 Hivelocity Hosting unknown 0:00:00 0.022 0.156 0.101 0.201 0.201
9 ServerStack Linux 0:00:00 0.025 0.099 0.081 0.161 0.161
10 INetU Windows Server 2003 0:00:00 0.025 0.143 0.088 0.221 0.484

See full table

Qube Managed Services had the most reliable hosting company site in September 2013, with not a single failed request throughout the whole month, and an average connection time of only 0.066 seconds. Qube is based in London, but they also host services from data centers in New York and Zurich. Their New York data center is at 111 8th Avenue, which is adjacent to a trunk dark fiber line. This building is the city's third largest in terms of floor area and was bought by Google for $1.9 billion in 2010. Qube provides managed and colocated hosting services from each of its data centers, as well as virtual data centers based on VMware vCloud Director.

Including September, Qube ("Qualified By Experience") has made five appearances within the top ten so far this year, and also attained another first place result in May.

In second place, with just one failed request, was Kattare Internet Services. It is among the most reliable sites monitored by Netcraft, managing 99.993% uptime over the past year and 99.97% over the past seven years. On September 21st, it was predicted that a damaging storm would hit the Pacific Northwest (Kattare's base of operations), causing thunder and wind storms, with 50mph gusts strong enough to take down trees. The next day this storm took out Kattare's power supply; however, the use of generators meant there were no outages recorded during the storm.

Hosting 4 Less narrowly missed out on second place, as although it only had one failed request during September, its average connection time was 3 milliseconds slower than Kattare's.

Seven of September's top ten hosting company sites were running on Linux operating systems. Five of these (including Qube, Kattare and Hosting 4 Less) used the Apache web server software, while ServerStack and krystal.co.uk used nginx.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

Wildcard EV certificates supported by major browsers

Extended Validation, or EV, certificates are designed to provide evidence of a greater level of verification by the Certificate Authority of the legal identity of the company in control of the SSL certificate and domain name. By way of contrast, the most common type of certificate, domain-validated, only requires the CA to verify control of the domain name. Browsers display EV-specific cues within the user interface to highlight this additional verification: most notably, the company name is displayed in the address bar, often with a green padlock or a green bar.

An Extended Validation certificate for login.live.com in Google Chrome

EV certificates are subject to additional requirements, over and above those specified in the Baseline Requirements. As with the Baseline Requirements, the EV guidelines were drawn up by the CA/B forum, an industry group of both browser vendors and CAs. The EV guidelines prohibit EV certificates from using wildcards (i.e. www.example.com, mail.example.com, and paypal.example.com would all match *.example.com) and explicitly mention this restriction twice "Wildcard certificates are not allowed for EV Certificates".

Nevertheless, Verizon Business has chosen to test browsers' approach to wildcard EV certificates by issuing a certificate to Accenture for *.cclearning.accenture.com. Verizon Business — which is not a member of the CA/B forum — is known for its maverick approach to certificate issuance having issued certificates (including EV certificates) which violate the Baseline Requirements.

Despite the EV guidelines prohibiting wildcard EV certificate issuance, presently most major browsers fail to enforce this restriction. Google Chrome, Firefox, Internet Explorer, Opera, and Safari (Desktop) all retain the EV browser cues when visiting a website using this EV certificate.

Clockwise from top left: Google Chrome, Internet Explorer, Opera, and Firefox. All display the conventional EV browser cues.

The only exception was Safari — Desktop Safari displays the EV browser cues as normal, as do the remainder of the desktop browsers; however, Safari on iOS 7 does not display the EV UI.

Safari (Desktop)

Safari on iOS 7 does not display the conventional EV UI for the wildcard EV certificate. An example of the EV UI in iOS 7.

Netcraft offers a Baseline Requirements checking service for CAs to provide third-party verification of Baseline Requirements conformance. For more information contact sales@netcraft.com