March 2013 Web Server Survey

In the March 2013 survey we received responses from 631,521,198 sites.

Microsoft showed a noticeable gain of 9M sites this month, increasing its market share by 1.42 percentage points to 18.01%. Much of this growth was seen at Go Daddy, which alone gained 2.6M sites powered by Microsoft web servers. Microsoft also fared well amongst the top million busiest sites, where its market share grew by 0.74 percentage points to 13.60%, increasing its narrow lead over its closest rival, nginx.

More than 130,000 of the sites in Netcraft's survey are hosted on Microsoft's Windows Azure cloud platform, but not all are running Microsoft web servers: Since Microsoft Open Technologies launched its VM Depot preview in January, more than 100 ready-to-use images have been added to the community-driven catalogue of virtual machines. This makes it easier for customers to deploy Linux-based images running preconfigured applications and frameworks such as WordPress, Joomla!, Drupal, Django and MongoDB. The majority of these images are based on Ubuntu Linux, and many of them use Apache and PHP to serve their content.

nginx also saw a reasonable gain in market share this month, with an additional 4.4M sites taking its share up by 0.68 percentage points to 13.53%. The most recent development release of nginx (1.3.13) introduced support for proxying WebSocket connections – an HTML5 technology which provides full-duplex communications between a browser and a web application over a single TCP connection. WebSockets are supported by all modern desktop browser software, for which the protocol specification defines two URI schemes: ws: for unencrypted connections, and wss: for secure ones. The development of WebSocket support in nginx was sponsored by CloudBees and Apcera, who will both be making use of the new feature in their own services.

nginx performed less well amongst the top million sites, where it had looked set to overtake Microsoft this month. Instead, a net loss of 910 nginx sites resulted in its share falling to 12.72%.





DeveloperFebruary 2013PercentMarch 2013PercentChange
Apache344,915,10554.68%341,021,57454.00%-0.68
Microsoft104,647,42516.59%113,712,29318.01%1.42
nginx81,074,69412.85%85,467,55513.53%0.68
Google22,717,9843.60%22,605,6463.58%-0.02
Continue reading

Phishing by proxy

Netcraft's toolbar community has reported an increase in the deployment of malicious scripts which direct webmail and online banking traffic through rogue proxy servers. These proxies allow attackers to steal usernames and passwords when forms are submitted, or use victims' cookies to hijack already-authenticated sessions.

The attacks rely on malicious proxy auto-config (PAC) scripts, which are remotely hosted and instruct a victim's web browser to proxy certain requests according to the specified configuration. Other requests are left untouched and end up being transmitted directly to the intended websites. The selective behaviour could perhaps be an attempt to limit the amount of traffic an attacker would need to process to extract sensitive information; alternatively, it could be an attempt to make detection more difficult — the results from services such as whatismyip.com may not be indicative of whether or not traffic was being intercepted.

Part of a malicious PAC script, which uses a proxy server hosted in Brazil

The PAC script shown above defines a JavaScript function – FindProxyForURL(url, host) – which is called by the browser. The full implementation of this function lets the attacker specify which URLs or hostnames should be requested directly, and which should be proxied. In the above example, requests to Banco do Brasil's website will be transmitted via the attacker's proxy server.

By using the Web Proxy Autodiscovery Protocol, a correctly positioned attacker could plausibly trick victims into using his phishing proxy without their knowledge. Although this feature is not enabled by default, many corporate environments may enable it in order to reduce the administrative overhead of manually configuring employees' laptops and other mobile devices to use proxies. If these devices are subsequently connected to an untrusted wireless network – which is controlled by an attacker – the WPAD discovery process would provide the attacker with a mechanism through which he can introduce arbitrary proxy scripts into browsers.

Alternative methods of attack include somehow enticing users to manually edit their proxy settings (perhaps by falsely claiming that it would result in performance benefits), or manipulating the settings via malware running on the user's computer. Similar malware-driven attacks have been around since 2008 and offer the attacker the additional advantage of being able to ensure that the malicious proxy settings cannot be tampered with.

Previous attacks using this technique originally targeted customers of Brazilian banks, but the fraudsters have since widened their scope and now also proxy traffic destined for webmail services such as Hotmail and Gmail, American banks, and one of the world's most popular phishing targets – PayPal.

To mitigate such attacks, it would be wise to avoid using automatic proxy detection settings on untrusted networks, and to also ensure your browser's automatic proxy configuration URL does not contain an unexpected address.

Most Reliable Hosting Company Sites in January 2013

Rank Company site OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Datapipe FreeBSD 0:00:00 0.000 0.022 0.016 0.033 0.051
2 Qube Managed Services Linux 0:00:00 0.000 0.025 0.025 0.049 0.050
3 www.uk2.net Linux 0:00:00 0.003 0.147 0.089 0.183 0.202
4 New York Internet FreeBSD 0:00:00 0.006 0.058 0.016 0.648 0.727
5 Kattare Internet Services Linux 0:00:00 0.006 0.236 0.089 0.180 0.382
6 www.logicworks.net Linux 0:00:00 0.009 0.039 0.022 0.354 0.403
7 Server Intellect Windows Server 2008 0:00:00 0.009 0.023 0.060 0.125 0.305
8 Netcetera Windows Server 2012 0:00:00 0.009 0.016 0.078 0.167 0.327
9 www.codero.com Linux 0:00:00 0.012 0.120 0.049 0.306 0.536
10 www.memset.com Linux 0:00:00 0.015 0.082 0.082 0.162 0.330

See full table

Datapipe had the most reliable website of all monitored hosting providers, responding to every single request made by Netcraft from its 11 monitoring points distributed across North America and Europe. Datapipe.net achieves such rapid average connection times — meaning that it often wins the top spot even when otherwise tied on failed requests — by serving content from the server topologically closest to the client. Datapipe's Stratosphere platform is available in five global data centres allowing its clients to benefit from similar performance.

Qube, ranked second this month, also with an exemplary record, with no failed requests recorded but a slightly longer average connection time. Qube has a number of notable clients, including both BetFair, an online gambling exchange, and blinkbox, a video streaming service in the UK, which trust it to provide a dependable network from its three data centres in London, New York, and Zurich.

UK2.net finished in third place, having just a single failed request. One of UK2.net's flagship brands, VPS.net, released a newly redesigned website and logo on the last day in January. At the same time, UK2.net also announced a set of new promises for VPS.net customers including a 99.9% SLA for unmanaged customers and 100% for those with managed services.

Server Intellect are joined in the top ten this month by a fellow Windows-based hosting provider, Netcetera, appearing in the top ten for the 57th time, but the first time since September 2012.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

February 2013 Web Server Survey

In the February 2013 survey we received responses from 630,795,511 sites.

Both Apache and Microsoft IIS were used on fewer sites this month, losing more than five million hostnames between them. Conversely, nginx grew its market share to 12.85%, serving 1.4 million more hostnames than last month. Amongst the million busiest sites, nginx is now almost neck-and-neck with Microsoft IIS — both have a market share of just under 13% and there are now fewer than 500 individual sites separating them.

Tengine, an nginx derivative maintained by Taobao, a large Chinese internet retailer, is now used on almost 4 million hostnames, including at the Internet Archive. Alibaba, the parent company of Taobao, has the second largest number of hostnames in China and accounts for more than 11% of the hostnames we find in China. Though China accounts for 19% of the world’s population, only 5.8% of the world's websites are hosted in China. Microsoft leads the way in China, with 38% of Chinese-hosted sites using IIS; just 26% use Apache, while usage of nginx — 19% — is significantly above-average.

Taobao is a magnet for Phishing attacks — Netcraft is currently blocking almost six thousand URLs targeting Taobao customers. After Facebook, Taobao.com is one of the busiest websites powered by PHP and also makes heavy use of JavaScript, though not using one of the more popular frameworks, instead using an open-sourced in-house developed library, KISSY.





DeveloperJanuary 2013PercentFebruary 2013PercentChange
Apache348,119,03255.26%344,915,10554.68%-0.58
Microsoft106,619,17716.93%104,647,42516.59%-0.34
nginx79,640,47212.64%81,074,69412.85%0.21
Google22,573,8583.58%22,717,9843.60%0.02
Continue reading

PHP just grows & grows

Netcraft began its Web Server Survey in 1995 and has tracked the deployment of a wide range of scripting technologies across the web since 2001. One such technology is PHP, which Netcraft presently finds on well over 200 million websites.

PHP Trend

The first version of PHP was named Personal Home Page Tools (PHP Tools) when it was released by Rasmus Lerdorf in 1995. PHP 1 can still be downloaded today from museum.php.net. Weighing in at only 26 kilobytes in size, php-108.tar.gz is diminutive by today's standards, yet it was capable of allowing users to implement guestbooks and other form-processing applications.

PHP 2 introduced built-in support for accessing databases, cookie handling, and user-defined functions. It was released in 1997, and by the following year, around 1% of sites on the internet were using PHP.

However, PHP 3 was the first release to closely resemble today's incarnation of PHP. A rewrite of the underlying parser by Andi Gutmans and Zeev Suraski led to what was arguably a different language; accordingly, it was renamed to simply PHP, which was a recursive acronym for "PHP: Hypertext Preprocessor". This was released in 1998 and the ease of extending the language played a large part in its tremendous success, as this aspect attracted dozens of developers to submit a variety of modules.

Andi Gutmans and Zeev Suraski continued to rewrite PHP's core, primarily to improve performance and increase the modularity of the codebase. This led to the creation of the Zend Engine, which was used by PHP 4 when it was released in 2000. As well as offering better performance, PHP 4 could be used with more web servers, supported HTTP sessions, output buffering and several new language constructs.

By September 2001, Netcraft's Web Server Survey found 1.8M sites running PHP.

PHP 5 was released in 2004, and remains the most recent major version release today (5.4.11 was released on 17 January 2013). Zend Engine 2.0 forms the core of this release.

By January 2013, PHP was being used by a remarkable 244M sites, meaning that 39% of sites in Netcraft's Web Server Survey were running PHP. Of sites that run PHP, 78% are served from Linux computers, followed by 8% on FreeBSD. Precompiled Windows binaries can also be downloaded from windows.php.net, which has helped Windows account for over 7% of PHP sites.

Popular web applications that use PHP include content management systems such as WordPress, Joomla and Drupal, along with several popular ecommerce solutions like Zencart, osCommerce and Magento. In January 2013, these six applications alone were found running on a total of 32M sites worldwide.

PHP also demonstrates a strong installation base across web-facing computers that are found as part of Netcraft's Computer Counting survey. Just as an individual IP address is capable of hosting many websites, an individual computer can also be configured to have multiple IP addresses. This survey allows us to identify unique web-facing computers and which operating systems they use regardless of how many sites or IP addresses they have. As of January 2013, 2.1M out of 4.3M web-facing computers are running PHP.

PHP has also become a victim of its own success in some respects: With so many servers running PHP, and with so many different web applications authored in PHP, hackers are presented with a huge and rather attractive attack surface. Because it is so easy to get started with programming in PHP, it attracts all levels of developers, many of whom may produce insecure applications through lack of experience and attention to detail. Netcraft's anti-phishing services find wave upon wave of phishing attacks hosted on compromised PHP applications, and the U.S. NVD (National Vulnerability Database) contains several thousand unique vulnerabilities that relate either to PHP itself, or to applications written in PHP.

Methodology

The full list of hostnames from the Netcraft Web Server Survey forms the basis of our technology tracking. We make requests to each of these sites, or if there is a large number of sites hosted on a single IP address, we employ a proportional sampling technique. The content of each page and its HTTP headers are analysed to determine which technologies are being used. For PHP, we look for references to .php filename extensions or the existence of HTTP response headers like "X-Powered-By: PHP". Additional signature tests are used to identify particular PHP applications, such as WordPress.

Each metric is then calculated as follows:

Hostnames

For each IP address, we estimate the total number of PHP sites it serves by calculating the product of the proportion of sampled hostnames that are running PHP and the total number of hostnames on that IP address. In cases where the IP address is serving 100 or fewer sites, all sites will be sampled and thus be representative of the entire population for that IP address.

Active sites

To provide a more meaningful metric which counts the number of human-generated sites actively using PHP, our active site count excludes spam sites or other computer-generated content. This methodology is described in more detail here.

IP addresses

This metric counts the number of unique IP addresses where at least one hostname in its sample set was found to be running PHP.

Computers

A single physical or virtual computer may have more than one IP address. We are able to identify unique computers that are exposed to the internet via multiple IP addresses. If an IP address is running PHP, then the computer associated with it is marked as running PHP. Further details of this methodology are explained in our Hosting Provider Server Count.

Netcraft removes phishing attacks in less than half the industry average time

Netcraft’s phishing site countermeasures service helps organisations targeted by phishing attacks remove the fraudsters’ forms as quickly as possible.

Recently we became aware that our median times for takedowns are very much better than the industry average calculated by the Anti-Phishing Working Group (APWG) in its most recent Global Phishing Survey. The APWG found that phishing attacks have a median lifetime of 5 hours and 45 minutes. In contrast, banks and other companies using our countermeasures service have experienced a median phishing attack availability of 2 hours and 12 minutes calculated over our most recent 100 takedowns, with the attacks removed in just 38% of the industry average time.

The graph below shows the availability times of our most recent 100 phishing attacks.

Last 100 Takedown Times

The difference between the first and final outages reflect the fact that phishing attacks will sometimes fluctuate up & down on compromised hosts where the fraudster may still have access to the system and be able to replace his content after the site owner removes it. In this scenario it is important to continue monitoring sites for some time after they go offline and restart takedowns if & when the phishing content reappears. For example, 87% of phishing attacks we attended to had their first outage within 24 hours, and 90% had their final outage within 48 hours.

Takedown times do vary significantly from country to country. For example, all of our last 100 takedowns in the US were completed within three days, and 90% had their first outage within 12 hours. In contrast, takedown times in Russia are rather longer, albeit with 90% going down within three days, and 70% having their first outage within twelve hours.

Russia and the US are by no means the long and short of phishing attacks. Phishing attacks we dealt with in the UK & Ireland have a shorter median lifetime than those hosted in the US, whilst phishing attacks we have taken down in Iran have a median lifetime of just under 30 hours, around five times longer than Russia.

In addition to providing fast takedown of the fraudulent content, the countermeasures service is also linked to our phishing site feed, which is licensed by all of the main web browsers, together with many of the largest anti-virus and content filtering products, firewall and network appliance vendors, mail providers, registrars, hosting companies and ISPs. Consequently, as soon as the phishing attack is verified, access to it will be blocked for hundreds of millions of people shortly afterwards, significantly reducing the effectiveness of the attack even before it has been removed.

More information regarding our countermeasures service can be found here.