North Korean websites still barely reachable since Christmas
10th February, 2015
North Korea's presence on the internet has remained extremely patchy for more than a month, with little improvement since a suspected DDoS attack that took place just before Christmas.
The state-run Korean Central News Agency website at www.kcna.kp has been barely reachable since Christmas day. Only 13% of requests to the site succeeded during the past month, with the worst period being around the end of January when the site became completely unavailable for several days in a row from our network of performance monitors.
Although the articles on www.kcna.kp are written in multiple languages, the KCNA clearly acknowledges that North Korea has never been an ideal location to host material that is intended for global consumption — for greater dissemination, the agency continues to publish articles to a secondary site at www.kcna.co.jp, which is hosted at a much more reliable location in Japan.
Even so, both of these sites remain deliberately inaccessible from some parts of the world. Access to both has been blocked in South Korea, and addresses in New Zealand were blocked after scraping content to be used on the KCNA Watch website, which tracks North Korean media.
When they do succeed, most requests to www.kcna.kp are met with an HTTP 1.0 response, which renders as a blank page. These responses can take a few minutes to be received:
$ curl -i http://www.kcna.kp HTTP/1.0 200 OK Connection: Close Pragma: no-cache cache-control: no-cache Refresh: 0.1 Content-Type: text/html; charset=iso-8859-1 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/strict.dtd"> <!-- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> --> <HTML> <HEAD> <META HTTP-EQUIV="Refresh" CONTENT="0.1"> <META HTTP-EQUIV="Pragma" CONTENT="no-cache"> <META HTTP-EQUIV="Expires" CONTENT="-1"> <TITLE></TITLE> </HEAD> <BODY><P></BODY> </HTML>
Occasionally, www.kcna.kp will return its proper content in a HTTP 1.1 response which uses JavaScript to redirect the browser to http://www.kcna.kp/kcna.user.home.retrieveHomeInfoList.kcmsf, but this page — as well as all of the images and scripts it uses — suffers from similar performance issues, making the site practically unusable from many locations outside of North Korea.
When it is accessible, this is what kcna.kp looks like.
Roughly half of the small number of websites hosted in North Korea, including www.kcna.kp, use Apache 2.2.15 running on the Linux-based Red Star 3.0 operating system. The Korea Computer Center (which also administers the .kp top-level domain) released this version of Red Star in 2013, but it was not until the end of last year that the rest of the world gained hands-on experience with it after an ISO image of the installation disk was distributed via bittorrent.
One of the sites using Apache 2.2.15 and Red Star 3.0 is the Korea Elderly Care Fund website at www.korelcfund.org.kp, which seemed to disappear completely for a few weeks after Christmas.
The rest of North Korea's websites are served by Apache running on CentOS, which is a free operating system derived from the sources of Red Hat Enterprise Linux. Websites using this platform in North Korea include the Korea National Insurance Corp site at www.knic.com.kp and the Committee for Cultural Relations with Foreign Countries at www.friend.com.kp, which ironically failed to respond to 84% of requests from our network of performance monitors.
Two years ago, Netcraft noted that kcna.kp used to run on Apache 2.2.3 with Red Hat Enterprise Linux 5. As this Linux distribution is owned, distributed and supported by an American multinational company, it is subject to U.S. export controls, which specifically prohibit its use in North Korea. As a result, this installation was likely unlicensed and so may not have received security updates, and would certainly not have received any official support.
North Korea normally has a very small presence on the internet, even when everything is working properly. Before the alleged attacks, Netcraft's Web Server Survey found 916 million websites around the globe, but only 24 of these sites were hosted in North Korea. To put that in perspective, you would have more chance of winning the UK's National Lottery jackpot than you would of randomly picking a North Korean website out of our survey.
Despite having an estimated population of 25 million people, North Korea has relatively few IP addresses of its own - just 1,024 in total. A third of the websites hosted in North Korea are served from a single IP address within this block, so a successful DDoS attack against this address is likely to take out several sites at once.
Hosted on an IP address assigned to North Korea, cqztjx.com is plastered with adverts for online gambling services.
In addition to North Korea's 1,024 native IP addresses, a block of 256 IP addresses in the range 5.132.126.0 – 5.132.126.255 has also been assigned to an End User in North Korea. These addresses appear to be used solely for hosting online gambling websites on virtual private servers. This block is marked as ASSIGNED PA, which means it is not permanently allocated to North Korea; the range will be lost if the local issuing internet registry, Outside Heaven, terminates its services.
inetnum: 5.132.126.0 - 5.132.126.255 netname: OUTSIDEHEAVEN_MUTI-IP_VPS descr: OUTSIDEHEAVEN_MUTI-IP_VPS infrastructure country: KP admin-c: OHS18-RIPE tech-c: OHS18-RIPE status: ASSIGNED PA
North Korea's other additional assigned network block at 77.94.35.0 – 77.94.35.255 does not currently appear to be used for hosting websites.
is.gd goes down, takes a billion shortened URLs with it
20th May, 2014
The popular is.gd URL shortening service has been offline for more than two days, taking with it more than a billion shortened URLs. Shortly before the site disappeared on Sunday, the homepage reported that its links have been accessed nearly 50 billion times.
The shortened links generated are usually not more than 18 characters long, including the protocol http://. These links are commonly used in tweets, emails, and text messages where long URLs are impractical. Despite the fact the shortened links do not work, many previously-created is.gd shortened URLs are still appearing on Twitter.
is.gd is owned by and supported by UK hosting provider Memset, who planned to support it as a free service indefinitely. Notably, its sister site, v.gd, is still up and running. Other free services provided by Memset include TweetDownload, TweetDelete and the statistics calculator Tweetails.
For security reasons, both is.gd and v.gd disallow the shortening of URLs which use the data: and javascript: protocols. Nevertheless, the service is still abused by fraudsters who use the shortened URLs to direct victims to phishing sites. Some fraudsters have appended a query string to the shortened URL in an attempt to make it look similar to those used by the phishing target. For example, the following is.gd URL was used to redirect victims to a Taobao phishing site:
http://is.gd/Tb###U?2.taobao.com/item.htm?spm=2007.1000337
Throughout April, is.gd was the fifth phishiest URL shortening service. By far the phishiest was tinyurl.com, which pointed to 17 times as many phishing sites, making it account for 60% of all phishing activity amongst the top five URL shortening services. Privately-held bit.ly, Google's goo.gl and GoDaddy's x.co also pointed to more phishing sites than is.gd.
Three years ago, the is.gd service suffered a shorter outage of a few hours. This was caused by the failure of some of the virtual machines in its frontend cloud, which were responsible for accepting HTTP requests from a load balancer.
Update 21/05/2014: is.gd is now back online. An explanation for the outage can be found at http://is.gd/news.php
GCHQ website falls after threats from Anonymous
12th February, 2014
GCHQ's website at www.gchq.gov.uk is exhibiting some noticeable performance issues today, suggesting that it could be suffering from a denial of service attack.
Last week, documents from whistle-blower Edward Snowden revealed that GCHQ carried out denial of service (DoS) attacks against communications systems used by the hacktivist group Anonymous during their own Operation Payback, which itself involved carrying out denial of service attacks against high profile websites such as MasterCard, Visa, Amazon, Moneybookers, and PostFinance.
This caused some furore amongst supporters of Operation Payback, some of whom were tried and convicted for carrying out denial of service attacks. Denial of service attacks are illegal in the UK under the Police and Justice Act 2006, yet the leaked slides suggest that GCHQ may have used such techniques against Anonymous, resulting in 80% of IRC users leaving within a month.
Part of a statement published by Anonymous on AnonNews.
Following these revelations, a statement on GCHQ's war against Anonymous was posted on the AnonNews website. The statement ends with a suggestion that some kind of retaliation could be expected: "Now that we truly know who it was who attacked us, Expect all of us."
Twitter accounts associated with Anonymous also fuelled suggestions that they could be responsible for GCHQ's website woes, with some referring to the #TheDayWeFightBack hashtag.
http://t.co/FCYJFlYAHr is still #TANGODOWN
We are anonymous.
It is far to late to expect us. pic.twitter.com/PVbTunXjqt
— AnonOpsCenter (@AnonOpsCenter) February 12, 2014Curiously, a much larger amount of downtime has been observed from Netcraft's Romanian performance monitor since the leaked slides were made public. That could indicate much more extreme DDoS mitigation techniques are being applied to these requests, and this in turn suggests that if an attack is occurring, perhaps Romania is one of the countries from which the attacks are being launched.
The www.gchq.gov.uk website is served from a content delivery network run by Limelight Networks, who claim to be one of the world's largest, best performing, and most highly available content delivery networks. Although it remains hosted at the same location, the website changed its Server header from "WebServer" to "EdgePrism/4.1.2.0" earlier this week. Limelight Networks first unveiled EdgePrism in 2001, so any similarities to the name of the NSA's PRISM mass electronic surveillance program are presumably coincidental.
OCSP Server Performance in April 2013
23rd May, 2013
| Rank | Performance Graph | OS | Outage hh:mm:ss |
Failed Req% |
DNS | Connect | First byte |
Total |
| 1 | ocsp.starfieldtech.com | Linux | 0:00:00 | 0.013 | 0.111 | 0.023 | 0.043 | 0.043 |
| 2 | ocsp.trendmicro.com/tmca | Citrix Netscaler | 0:00:00 | 0.019 | 0.043 | 0.099 | 0.200 | 0.200 |
| 3 | ocsp.entrust.net | Linux | 0:00:00 | 0.022 | 0.251 | 0.014 | 0.249 | 0.249 |
| 4 | ocsp.godaddy.com | Linux | 0:00:00 | 0.022 | 0.164 | 0.021 | 0.041 | 0.041 |
| 5 | ocsp.digicert.com | Linux | 0:00:00 | 0.022 | 0.027 | 0.026 | 0.051 | 0.051 |
| 6 | ocsp.quovadisglobal.com | Windows Server 2003 | 0:00:00 | 0.032 | 0.021 | 0.116 | 0.222 | 0.222 |
| 7 | ocsp.verisign.com | Citrix Netscaler | 0:00:00 | 0.038 | 0.050 | 0.084 | 0.168 | 0.168 |
| 8 | evsecure-ocsp.verisign.com | Citrix Netscaler | 0:00:00 | 0.041 | 0.239 | 0.085 | 0.168 | 0.168 |
| 9 | ocsp.thawte.com | Citrix Netscaler | 0:00:00 | 0.044 | 0.041 | 0.083 | 0.165 | 0.165 |
| 10 | ocsp.startssl.com/sub/class4/server/ca | Linux | 0:00:00 | 0.047 | 0.086 | 0.011 | 0.041 | 0.041 |
Starfield Technologies had the most reliable OCSP responder during April, failing to respond to only 4 of Netcraft's OCSP requests. Starfield also had the most reliable responder in March, but showed a slight improvement to its average connection times in April. Starfield was founded as the technology and research branch of Go Daddy in 2003, and Go Daddy customers can choose to have their SSL certificates issued by either Starfield or Go Daddy.
Trend Micro had the second most reliable OCSP responder, which failed to respond to only 6 requests. However, this could be one of the survey's least busy responders: Netcraft's April 2013 SSL Survey discovered only 113 valid SSL certificates issued by Trend Micro, all of which are organisation validated. 29 of these certificates are used by a single organisation, Florida Hospital.
StartCom (which operates StartSSL) once again exhibited the fastest connection times, taking only a hundredth of a second to establish a TCP connection for one of its OCSP URLs. However, its reliability was only just good enough to make it into the top ten — in total, 15 requests to http://ocsp.startssl.com/sub/class4/server/ca failed during April.
Linux is the most popular choice of operating system on which to run an OCSP responder, and it certainly seems to perform well with regard to connection times: all of the top 25 fastest OCSP responders used Linux in April. In terms of failed requests, though, the distribution of Citrix Netscaler appliances is skewed towards the more reliable end of the spectrum — of the five responders that were using Netscaler, four of them feature in the top ten. QuoVadis's OCSP responder, which was sixth most reliable in April, is one of only two responders that ran on Windows.
On April 24, nginx 1.4.0 stable was released, incorporating several new features that had previously only been released in development branches of the web server. One of the most important performance features is that nginx now support OCSP stapling. This feature is designed to improve performance by allowing secure websites to "staple" a cached OCSP response to the TLS handshake, removing the need for the client browser to make a second, separate connection to the certificate authority's OCSP responder.
The Online Certificate Status Protocol (OCSP) is an alternative method to Certificate Revocation Lists (CRLs) for obtaining the revocation status of an individual SSL certificate. Fast and reliable OCSP responders are essential for both Certificate Authorities (CAs) and their customers — a slow OCSP response will introduce an additional delay before many browsers can start sending and receiving encrypted traffic over an HTTPS connection.
OCSP Server Performance in March 2013
22nd April, 2013
| Rank | Company site | OS | Outage hh:mm:ss |
Failed Req% |
DNS | Connect | First byte |
Total |
| 1 | ocsp.starfieldtech.com | Linux | 0:00:00 | 0.003 | 0.076 | 0.024 | 0.043 | 0.043 |
| 2 | ocsp.verisign.com | Citrix Netscaler | 0:00:00 | 0.006 | 0.051 | 0.081 | 0.162 | 0.162 |
| 3 | ocsp.thawte.com | Citrix Netscaler | 0:00:00 | 0.006 | 0.041 | 0.083 | 0.164 | 0.164 |
| 4 | ocsp.godaddy.com | Linux | 0:00:00 | 0.015 | 0.161 | 0.025 | 0.044 | 0.044 |
| 5 | ocsp.startssl.com/sub/class4/server/ca | Linux | 0:00:00 | 0.018 | 0.068 | 0.011 | 0.056 | 0.056 |
| 6 | evsecure-ocsp.verisign.com | Citrix Netscaler | 0:00:00 | 0.018 | 0.228 | 0.082 | 0.163 | 0.163 |
| 7 | ocsp.trendmicro.com/tmca | Citrix Netscaler | 0:00:00 | 0.018 | 0.050 | 0.099 | 0.200 | 0.201 |
| 8 | evintl-ocsp.verisign.com | Citrix Netscaler | 0:00:00 | 0.024 | 0.261 | 0.082 | 0.162 | 0.162 |
| 9 | ocsp.startssl.com/sub/class2/server/ca | Linux | 0:00:00 | 0.027 | 0.049 | 0.011 | 0.057 | 0.057 |
| 10 | ocsp.xi.tcclass2-ii.trustcenter.de | Linux | 0:00:00 | 0.027 | 0.199 | 0.090 | 0.197 | 0.197 |
The Online Certificate Status Protocol (OCSP) is an alternative method to Certificate Revocation Lists (CRLs) for obtaining the revocation status of an individual SSL certificate. Fast and reliable OCSP responders are essential for both Certificate Authorities (CAs) and their customers — a slow OCSP response will introduce an additional delay before many browsers can start sending and receiving encrypted traffic over an HTTPS connection.
Starfield Technologies, a Go Daddy brand, had the most reliable OCSP responder last month with only a single failed request and an average connection time of 24ms. Starfield Technologies was founded in 2003 as the technology research branch of Go Daddy. Go Daddy customers have the option to choose which issuing organization to use when buying an SSL certificate. Although both Go Daddy and Starfield appear to share the same OCSP responder infrastructure, ocsp.godaddy.com had five failed requests, however this was still fewer than StartCom, Symantec, and Trend Micro. Both Go Daddy and Starfield issue certificates in all three certificate assurance categories: Domain Validation (DV), Organisation Validation (OV), and Extended Validation (EV). Starfield is most prominent in the EV sector — more than 15% of all EV certificates issued within the group are issued by Starfield — but it remains only a small part of Go Daddy's SSL certificate business: Starfield accounts for just 10% of certificates issued.
StartCom had the shortest average connect time (11ms) of all monitored CAs last month after having moved its OCSP infrastructure at the end of February. StartCom, as well as Entrust, now delivers its OCSP responses via the Akamai CDN (Content Delivery Network), reducing the OCSP connection overhead to a minimum by serving content from as topologically close as possible to the client. GlobalSign is a CloudFlare evangelist, using CloudFlare's CDN platform for its OCSP and CRL infrastructure as well as their own corporate website.
Many of the monitored OCSP responders are served by Citrix Netscaler devices. Citrix Netscaler is a hardware appliance that provides, amongst other features, load balancing and firewall functions. The use of such load balancing technology is no surprise — a single certificate on a popular site that does not use OCSP stapling could generate a significant number of OCSP requests, causing a CA's responder to experience high volumes of traffic.
In many circumstances each connection to an HTTPS site could trigger multiple OCSP requests: a request for the server's certificate and one for each intermediate certificate. OCSP responses are typically valid for a week, so some caching is possible. Caching can reduce both the burden on OCSP responders and increase the perceived performance of HTTPS websites to users, but is limited to repeat visits. OCSP Stapling is designed to improve performance by allowing the web site's server to “staple” the OCSP response to the TLS handshake, removing the need for the client to connect to the CA's OCSP responder.
Netcraft measures and makes available the OCSP and CRL end point response times of all the major Certificate Authorities (CAs). The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.
Certificate revocation and the performance of OCSP
16th April, 2013
Certificate revocation is a critical aspect of maintaining the security of the third-party Certificate Authority (CA) infrastructure which underpins secure communication on the internet using SSL/TLS. A certificate may be worth revoking when it has had its private key compromised, the owner of the certificate no longer controls the domain for which it was issued, or the certificate was mistakenly signed. Without the ability to revoke certificates, a CA has no direct means of marking a certificate as untrusted before the expiry of the certificate, which could be several years away. In particularly urgent cases a browser vendor may have the ability to block certain individual certificates, trusted roots, or intermediate certificates, but this is rarely performed and is not suitable for lower-risk issues where revocation is necessary but not urgent.
There are two main technologies for browsers to check the revocation status of a particular certificate: using the Online Certificate Status Protocol (OCSP) or looking up the certificate in a Certificate Revocation List (CRL). OCSP provides real-time revocation information about an individual certificate from an issuing CA, unlike CRLs which provide a list of revoked certificates and may be received by clients less frequently.
The graph below shows a comparison of the time taken for the TLS handshake, both with and without OCSP checking enabled. The data was collected using packet traces taken while using Firefox 20 on Linux from an IP address in the UK. Measurements were taken three times (each time with a fresh cache) after discarding an initial request.
The relationship between whether OCSP checking is enabled and the time taken to complete the TLS handshake is not straightforward. In order for the browser to display the "green bar" to distinguish an Extended Validation (EV) certificate, OCSP requests must be made for every certificate in the chain whereas in many browsers, if an OCSP request is made at all, intermediate certificates are not checked. The increased time taken for the TLS handshake when using an EV certificate can be attributed to Firefox's sequential OCSP checking behaviour. However, where an OCSP check can be performed within the round-trip time to the server — for example, if the OCSP responder is served via a content delivery network or CDN — the check does not dramatically affect the time taken for the TLS handshake. When both the web server and the OCSP responder are topologically close to the client, as is the case with www.globalsign.com, the short round-trip time to the server isn't sufficient to mask the the time taken to receive OCSP responses for both the web site's certificate and the intermediate certificate presented. The slight difference between Paypal and GlobalSign's performance can at least partially be attributed to the additional OCSP request made for GlobalSign: GlobalSign's certificate chain requires three OCSP requests whereas Paypal's requires just two.
Netcraft has extracted around 40 OCSP responder URLs from certificates seen in the Netcraft SSL server survey, and has been monitoring them since late November 2012. The performance and reliability of the services varies significantly: Symantec's VeriSign OCSP responder has had consistently solid reliability, only a handful of connections failed over a 4 month period; whereas, in the same period more than 6% of requests to one of StartCom's responders failed. The reliability and performance of StartCom's OCSP responders have improved significantly since the end of February 2013 when it switched to using Akamai. Geotrust, another Symantec brand, did not have as strong a performance as either Thawte or VeriSign — all three of GeoTrust’s OCSP servers were down for between 48 and 104 minutes in a single event. Performance and reliability is measured from 11 points spread around Europe and North America: outages require at least one failed response from all measurement nodes within the 15-minute measurement interval.
For those browsers performing a synchronous OCSP request during the TLS handshake, the performance of the OCSP responder is often crucial. Any delay in responding to the request may noticeably slow down the handshake. For example, comparing GlobalSign's CloudFlare-accelerated OCSP responder with Entrust's, you find that GlobalSign's responder is significantly faster than Entrust's which uses Akamai's CDN. However, despite GlobalSign's performance advantage, its reliability has been affected by a number of CloudFlare outages — since Netcraft began monitoring OCSP, GlobalSign's responders have had at least 45 minutes of downtime whereas Entrust has had none.
OCSP responses can be stapled to a response from a web server when negotiating the TLS handshake to avoid the need for the browser to make a secondary request to a third party server. CloudFlare has claimed that enabling OCSP stapling has led to a 30% speed improvement for HTTPS sites. OCSP stapling support is present in newer versions of nginx — an increasingly popular open source web server — as a result of a development project sponsored by GlobalSign, DigiCert, and Comodo. OCSP stapling is not supported in the most popular version of Apache, 2.2.x, nor is it supported in current versions of Firefox (although support is in the pipeline), so it must remain only part of the solution for the foreseeable future. Frustrated by some of the limitations of OCSP, some CAs have lent support to a proposed an alternative revocation method using short lived certificates.
Browser support for the both OCSP and CRLs is mixed: currently, Firefox does not automatically download the CRLs from trusted CAs, so Firefox users must rely on OCSP alone; Google uses a proprietary mechanism to distribute CRLs to users of Google Chrome which aggregates per-CA CRLs into a single update which is distributed using its automatic update channel. Many browsers default to a "soft-fail" approach, leaving users vulnerable to eavesdroppers able to block or tamper with OCSP traffic. For as long as the CAs running OCSP responders do not have a strong record for both the performance and the reliability of their OCSP responders, browsers will find it difficult to justify switching to synchronous "hard-fail" behaviour.
Updated 18/04/2013