December 2014 Web Server Survey

In the December 2014 survey we received responses from 915,780,262 sites and 5,034,578 web-facing computers.

This is the second month in a row where there has been a large drop in the total number of websites, giving this month the lowest count since January. As was the case in November, the loss has been concentrated at just a small number of hosting companies, with the ten largest drops accounting for over 52 million hostnames. The active sites and web facing computers metrics were not affected by the loss, with the sites involved being mostly advertising linkfarms, having very little unique content. The majority of these sites were running on Microsoft IIS, causing it to overtake Apache in the July 2014 survey. However the recent losses have resulted in its market share dropping to 29.8%, leaving it now over 10 percentage points behind Apache.

Despite losing more than six million hostnames this month, nginx outpaced all other major server vendors by gaining 22,300 web-facing computers. nginx is now used by nearly 11% of all web-facing computers – twice the share that it had two years ago.

Overall, the total number of web-facing computers in our survey increased by just over 40,000 this month, making nginx responsible for more than half the increase. Despite an increase of over 11,000 computers for Apache, and 1,700 for IIS, both continue to lose market share.

Thanks to continued strong growth at Amazon Web Services, Amazon is the largest hosting company by a considerable margin in terms of our web-facing computers metric (which includes web-facing virtual machines, providing that each has its own kernel and IP address). With nearly 300,000 web-facing computers in total, Amazon has just over twice as many as second-place OVH. In October, we reported that DigitalOcean had become the 4th largest hosting company in under 2 years, but it quickly reached third place in November and is continuing to close the gap on OVH.

Cloud growth

Both Amazon Web Services and Microsoft Azure expanded their cloud hosting footprints recently. Amazon opened a new European AWS region in Frankfurt, which augments its existing EU region in Ireland. Besides being able to host services closer to the center of Europe, the new region means that customers can now build multi-region applications with the assurance that their data will stay within the EU. The new Frankfurt region houses two EC2 availability zones and three AWS edge locations.

Microsoft's new Azure "geo" is in Australia, and consists of two geographically redundant regions in New South Wales and Victoria. This will help Microsoft to compete with Amazon's two EC2 availability zones in Sydney.

New TLDs

More new top-level domains showing strong growth in this month's survey include .nyc, which is targeted for use by New Yorkers, and .realtor, which is only allowed to be used by members of the National Association of Realtors or the Canadian Real Estate Association. These have grown from virtually nothing to a total of 40,000 and 80,000 sites respectively.

Ironically, one of this month's fastest growing new top-level domains started off as an April Fool's Day joke in 2009, when the founder of OVH announced the creation of the .ovh TLD – years before such things were actually possible. This joke resulted in over 22,000 requests to register .ovh domains within a few hours, demonstrating the potential demand for such domains. OVH eventually entered into a Registry Agreement with ICANN in January 2014, and the sunrise period for the .ovh TLD began in September. This month's survey saw the number of sites using .ovh domains grow from 6,000 to 63,000, likely due to the first 50,000 .ovh domains being given away for free, and with subsequent growth being fuelled by attractive pricing: new .ovh domains can currently be registered for only EUR 0.99 per year and renewed for EUR 1.99.

Total number of websites

Web server market share

DeveloperNovember 2014PercentDecember 2014PercentChange
Apache350,853,79837.05%358,159,40539.11%2.06
Microsoft306,029,30732.31%272,967,29429.81%-2.51
nginx139,130,99214.69%132,467,76314.47%-0.23
Google19,560,2062.07%20,011,2602.19%0.12
Continue reading

ICANN hit by successful spear phishing attack

The Internet Corporation for Assigned Names and Numbers (ICANN) has fallen victim to a phishing attack which resulted in the attackers gaining administrative access to some of ICANN's systems, including its Centralized Zone Data Service (CZDS).

In an email alert sent this morning, ICANN said it believes a spear phishing attack in November resulted in several ICANN staff members' email credentials being compromised. The stolen passwords were then used to gain unauthorised access to multiple ICANN systems, which could have resulted in other usernames and passwords being compromised.

Although CZDS passwords are stored as salted hashes, ICANN has taken the precaution of deactivating passwords and API keys used on the compromised CZDS service. ICANN implemented some security enhancements earlier this year, which it believes limited the extent of the unauthorised access, and has implemented further measures since this attack.

The spear phishing emails involved in this attack were crafted to appear to originate from ICANN's own domain, which is a common tactic for phishers as it lends a fair amount of credibility to the emails. This domain spoofing could well have played an important part in the successfulness of the attack, but icann.org still does not feature any Sender Policy Framework records to specify who can send mail on its behalf.

Organisations concerned about these types of attack can use Netcraft's Fraud Detection service, which processes DMARC (Domain-based Message Authentication, Reporting and Conformance) reports on your behalf. These reports are sent by ISPs and e-mail receivers when they see any emails which claim to be from one of your own domains. A web interface shows the status of all of your own domains, any configuration changes required, and highlights unprotected domains being used by fraudsters attacking your customers.

Banks allow phishers to log in using Tor

The Financial Crimes Enforcement Network (FinCEN), a department of the US Treasury that combats financial crimes such as fraud and money laundering, recently released a report stating that "nearly $24 million in likely fraudulent activity" involved known Tor network nodes. The proportion of fraud that involves Tor is increasing rapidly: according to the report, October 2007 to March 2013 saw an increase of 50% in Tor-related fraud reports, whereas the most recent and much shorter period of March 2013 to July 2014 saw an increase of 100%. The report, which is not public, was obtained by computer security journalist Brian Krebs.

Tor is a piece of open-source software that attempts to provide online anonymity using a technique known as "onion routing". Messages sent by the user, such as HTTP requests from the user's web browser, are sent across the Tor network, instead of being sent directly to the destination server. Before a user sends a message, it is encrypted several times, along with information describing how the message should be routed through a virtual circuit across the Tor network. Circuits consist of a series of three randomly-selected Tor nodes: an entry node, a middle node and an exit node. The user's traffic enters the Tor network at the entry node. Each successive node is able to remove a single layer of encryption, which also reveals the next node to send the message to – akin to peeling the layers of an onion. When the message reaches the exit node, the final layer of encryption is removed and it is sent out across the Internet to its final destination. A similar procedure applies to messages travelling in the opposite direction back to the user, such as HTTP responses.

A diagram showing the nodes and the links between them in a Tor circuit. Although Tor does not encrypt the communication between the exit node and destination itself, it can be encrypted by the applications using Tor – for example, the user's web browser could use HTTPS instead of HTTP.

At no single point in the circuit are the source IP address, destination IP address and contents of the message all known to an eavesdropper simultaneously. To reduce the chance that users can be de-anonymized, Tor attempts to avoid picking nodes that share the same operator when creating circuits. This makes it difficult, but perhaps not impossible, for the identity of a particular user to be discovered. For example, an attacker who can observe a user's traffic as it both enters and leaves the Tor network can carry out a traffic confirmation attack, in which they correlate characteristics such as the timing or volume of the user's traffic, to link the user to the destination server.

Unsurprisingly, the anonymity provided by Tor makes it an attractive tool for fraudsters. For example, a phisher who has tricked users into handing over their online banking credentials might use Tor to log in to the bank's website with the compromised credentials. The bank's log files will show the IP address of the Tor exit node, rather than the phisher's own IP address, making it more difficult for the bank and law enforcement agencies to trace the fraud back to the phisher.

The report from FinCEN examined 6,048 suspicious activity reports (SARs) filed by banks and other financial companies between 2001 and 2014. Of those, 975 involved Tor, totalling $24 million of "likely fraudulent activity". The report goes on to state that "in the majority of the SAR filings, the underlying suspicious activity – most frequently account takeovers – might have been prevented if the filing institution had been aware that their network was being accessed via Tor IP addresses." Even if blocking Tor does not deter phishers from committing fraud entirely, it may cause them to switch to using services that are easier for the authorities to trace, such as open proxy servers or anonymous VPN services.

According to FinCEN's report, banks were only aware that Tor was involved in 3% of cases. Netcraft has visited the websites of the ten financial companies most targeted by phishing in the last six months, using a variety of Tor exit nodes located around the world, to check if any of the companies block Tor.

Position Company Blocks Tor traffic
1 PayPal No, but Tor users must solve a CAPTCHA
2 USAA No
3 AXA Banque No
4 SFR No
5 Wells Fargo No
6 Bank of America No
7 Chase No, but Tor users must use two-factor authentication
8 Lloyds Bank No
9 Banco do Brasil No
10 Cielo No

As shown in the table above, none of the login pages we visited blocked Tor traffic outright. For example, the following screenshot shows the appearance of PayPal's login page fetched from a variety of Tor exit nodes:

Screenshots of PayPal's login page fetched from several Tor exit nodes located across the world.

However, some of the websites we tested do treat Tor users differently during or after the login process – instead of blocking Tor users outright, they use Tor as an indicator for performing more stringent anti-fraud checks. (It is also possible that some companies perform additional checks that are not visible to end users.)

For example, Chase forces the use of two-factor authentication – by either email, text message or phone call – over Tor. PayPal requires Tor users to solve a CAPTCHA during the login process, which protects against automated attacks such as brute force login attempts, but would not prevent a phisher from manually logging into a victim's account. On the other hand, Lloyds Bank does not appear to visibly treat Tor users any differently to normal users.

A screenshot of the CAPTCHA that PayPal displays to users who attempt to log in over the Tor network.

The Tor Project considers services blanket blocking Tor traffic due to abusive and illegal behaviour by a proportion of its users to be a "threat to Tor's success". It advocates a range of other measures for sites to tackle abusive Tor traffic, including CAPTCHAs, two-factor authentication and establishing trust on a per-user rather than a per-IP basis. However, with the exception of two-factor authentication, most of these measures are targeted at abusive behaviour such as spam and are unlikely to prevent fraudsters from logging into compromised accounts.

Netcraft provides a wide range of countermeasures against phishing to many customers, including two of the world's top ten banks, as well as some smaller institutions at the sharp end of Internet crime – such as three of the largest Bitcoin exchanges and four Nigerian banks. For more information, please contact sales@netcraft.com.

Most Reliable Hosting Company Sites in November 2014

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Qube Managed Services Linux 0:00:00 0.004 0.104 0.039 0.078 0.078
2 XILO Communications Ltd. Linux 0:00:00 0.012 0.215 0.066 0.131 0.224
3 Datapipe Linux 0:00:00 0.019 0.103 0.018 0.035 0.050
4 Pair Networks FreeBSD 0:00:00 0.019 0.221 0.080 0.162 0.540
5 ServerStack Linux 0:00:00 0.027 0.090 0.079 0.155 0.155
6 Host Europe Linux 0:00:00 0.031 0.109 0.072 0.171 0.172
7 Bigstep Linux 0:00:00 0.039 0.134 0.062 0.125 0.125
8 INetU Windows Server 2003 0:00:00 0.039 0.140 0.076 0.207 0.486
9 Kattare Internet Services Linux 0:00:00 0.039 0.191 0.122 0.253 0.532
10 Memset Linux 0:00:00 0.042 0.128 0.067 0.154 0.266

See full table

Qube had the most reliable hosting company site in November with just a single failed request. This is the fifth time Qube has made it to first place in 2014, and the fourteenth time it has featured in the top 10 since September 2013. Qube offers a Managed Operating System Service, where its engineers cover day-to-day tasks such as continuous monitoring, tuning and patching.

XILO had the second most reliable hosting company website, with three failed requests. This is the second month in a row that XILO has taken the title of the second most reliable hosting company website. XILO offers a Premium Hosting service with automatic fail-over in the event of a server failure and 1:1 volume mirroring between cloud nodes to ensure redundancy.

Datapipe had the third most reliable hosting company website with five failed requests. Datapipe uses a high performance network over multiple Tier-1 backbone providers to ensure reliability and scalability.

Linux was again the most popular operating system of choice, being used by 8 out of the top 10 hosting company websites. FreeBSD and Windows Server 2003 both had single entrants, Pair Networks and INetU respectively.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

Typosquatters cashing in on .uk domains

Typosquatters are cashing in by registering new .uk domains which look similar to those used by existing high-traffic .co.uk websites. By simply registering a .uk domain that ends in "co", the squatters have obtained dangerously deceptive domains such as paypalco.uk and americanexpressco.uk in an attempt to steal traffic from the real domains, paypal.co.uk and americanexpress.co.uk.

Many of these typosquatting domains are being monetized by displaying ads related to the legitimate domains they are impersonating, or by using referral schemes to redirect visitors to the corresponding legitimate site — or even driving visitors towards competing services.

The typosquatting site at paypalco.uk features monetized adverts for both PayPal and its competitors.

The typosquatting site at paypalco.uk features monetized adverts for both PayPal and its competitors.

However, the potential for abuse is not limited to making money through advertising and referral schemes. With the only difference being a single additional dot in the real domain name, this form of typosquatting could be exploited to make extremely potent phishing attacks.

First introduced in 1985, the .uk country code top-level domain (ccTLD) has only recently allowed ordinary consumers to register domains directly under .uk (such as stephenfry.uk). Before 10 June 2014, practically all UK domains had to be registered under second-level domains, which categorised the activity of the site. By far the most popular of these second-level domains is .co.uk, which is intended for commercial and general use.

Even the BBC has been targeted: www.bbcco.uk  redirects browsers to a sponsored listings page at bringthenews.co.uk

Even the BBC has been targeted: www.bbcco.uk
redirects browsers to a sponsored listings page at bringthenews.co.uk

To limit the most obvious potential for domain squatting, existing owners of .co.uk domains were given automatic rights to the corresponding .uk domain (for example nationalrail.uk) on 10 June 2014, providing there was no other equivalent .org.uk, .me.uk, .net.uk, .ltd.uk and .plc.uk domain in existence. The reservation period runs for a period of five years, during which time no other party can register the domain, even if the rightful party chooses not to.

However, these measures are inconsequential to the typosquatters, who seem to have found no barriers in registering deceptive domains such as nationalrailco.uk, barclaysco.uk and hsbcco.uk. The latter two deceptive domains are registered to a corporation in Sweden, and currently display a set of sponsored listings with titles such as "Need a New Bank Account?". Other registered domains which target high-traffic financial institutions include nationwideco.uk, lloydsbankco.uk, bankofscotlandco.uk, halifax-onlineco.uk, natwestco.uk, and westernunionco.uk.

The potential for financial fraud is immense, particularly as many online banking transactions are now carried out using mobile devices, on which typographical errors are naturally more common.

Some of the .uk typosquatting sites are clearly optimised for use on mobile devices, such as nationalrailco.uk, which displays a small form to search for train tickets. However, rather than taking users to the real National Rail website at nationalrail.co.uk, the search form uses the TradeDoubler affiliate scheme to monetize the typo-traffic by directing users to a train ticket sales website at thetrainline.com.

Some co.uk typosquatting sites are optimised to be viewed on mobile devices.

Some co.uk typosquatting sites are optimised to be viewed on mobile devices.

Flagrant typosquatting of popular sites amongst the .uk top-level domain is rife. Another brazen example is mbnaco.uk, which is clearly trying to scoop up typo-traffic from credit card provider MBNA, which uses mbna.co.uk for its main website. The typo domain presents adverts which invite visitors to apply for credit cards at various competitors, including American Express and Capital One.

Sponsored listings for competing credit card providers on mbnaco.uk

Sponsored listings for competing credit card providers on mbnaco.uk

Companies concerned about typosquatting attacks against their customers can use Netcraft's Fraud Detection service to pre-emptively identify fraudulent domain name registrations. Domain name registrars can use Netcraft's Domain Registration Risk service to analyse the likelihood of a new domain being used for fraudulent activity.

November 2014 Web Server Survey

In the November 2014 survey we received responses from 947,029,805 sites and 4,994,577 web-facing computers.

Despite the dramatic loss of almost 82 million sites since October, the number of web-facing computers increased by over 23 thousand (+0.47%) this month. Automatically generated content and wildcard sites are often used for activities such as domain holding and search-engine results manipulation, where the majority of the sites will not receive human visitors and therefore the resources required to run the sites are minimal. This is clearly demonstrated this month, with over half of the net loss of sites being attributed to a single IP address that was previously hosting parked websites.

The most significant contributor to the increase in web-facing computers was nginx, with almost 16k more computers using the web server this month. Apache and Microsoft both experienced small losses in market share as a result, continuing the trend seen over the past few years.

nginx gained a number of notable high-traffic websites in this month's survey. After several years of using lighttpd, the SourceForge download site (downloads.sourceforge.net, traffic rank #411) has switched to nginx, and gaming news site pcgamer.com (#1339) has also started using nginx with Rackspace's Cloud Load Balancing as a Service product.

Amongst the top million websites, Apache's market share has continued its slow decline. Ever since Netcraft started publishing these figures, Apache has commanded more than half of this market sector. This month, however, Apache has reached its lowest market share ever: 50.01%. Although it still clings on to more than half of the market this month, it looks unlikely to retain it for long. Microsoft and Google have also been losing share amongst the top million sites, all making room for nginx, which has muscled its way up to 20.4%.

New top level domains are continuing to add a modest number of sites to the survey each month. One of this month's fastest growing new TLDs is .audio, which is run by Uniregistry, where it is pitched as a dedicated online space for sound, musicians, engineers and producers. More than 17,000 sites are already using this TLD.

German states and cities are also contributing to the growth in new TLDs. For example, .koeln is now used by 14,000 sites, and .bayern is already used by more than 10,000 websites since it entered general availability on 30 September 2014.

Just in time for this year's Black Friday events on November 28th, the .blackfriday TLD was also launched recently by Uniregistry. More than 10,000 sites have adopted this domain in the space of a month, including amazon.blackfriday and target.blackfriday, which redirect to each company's main US website. Uniregistry also provides another seasonal TLD, .christmas, which was found to be used by 12,000 sites in this month's survey.

NCC Group released its 124-page .trust Technical Policy [pdf] last month, which outlines a stringent set of criteria which must be met by registrants of .trust domains. Many of these policies relate to security, dictating both what must and must not be done – the table of contents alone contains 71 instances of "Do not". The policies go far beyond the typical fraud prevention requirements set by most other registries, with a heavy emphasis on web application security. For example, websites which use .trust domains must serve all content over an encrypted HTTPS connection, must not use inline JavaScript or the eval() function, must provide an appropriate Content Security Policy header, and must be free from open redirects.

NCC Group bought the .trust TLD earlier this year from Deutsche Post, which had originally obtained it from ICANN in 2013. With NCC Group claiming that security compliance is externally verifiable, it anticipates that all sites using the .trust TLD will be recognised as trustworthy.

Total number of websites

Web server market share

DeveloperOctober 2014PercentNovember 2014PercentChange
Apache385,354,99437.45%350,853,79837.05%-0.40
Microsoft345,485,41933.58%306,029,30732.31%-1.26
nginx148,330,19014.42%139,130,99214.69%0.28
Google19,431,0261.89%19,560,2062.07%0.18
Continue reading