May 2016 Web Server Survey

In the May 2016 survey we received responses from 1,033,790,346 sites and 5,946,961 web-facing computers. This reflects a gain of 147,000 computers, coupled with a loss of 49 million sites.

While last month's survey recorded the largest number of sites ever, many of the Chinese sites running Microsoft IIS that appeared last month have since disappeared. Combined with other departures, Microsoft suffered a net loss of 75 million sites this month, which has played a major part in its market share falling by more than 5 percentage points to less than 36%. Nevertheless, it is still the most common server vendor by number of sites, with a total of nearly 370 million hosted on IIS servers.

Despite Microsoft's loss of 75 million sites, the number of active sites using IIS actually grew by 450,000, which is indicative of the low quality of the sites it lost. Most of the lost sites were engaged in link farming activity, with large numbers of these sites being served from relatively few computers. The loss of these sites therefore had little impact on the number of web-facing computers using Microsoft IIS, which grew by 14,000.

Microsoft's closest competitor, Apache, gained 8.4 million sites, with its increased market share of 29.1% putting it within 6.4 percentage points of Microsoft's leading share.

Although it has yet to reach the same level as Microsoft and Apache, nginx made the largest gains, growing by 21 million sites and increasing its market share by 2.6 points to 15.9%.

nginx also showed the strongest growth in the survey's other metrics: it gained nearly 7.5 million active sites, 74,100 web-facing computers, and increased its presence within the top million sites by 16,000. The most significant of these gains was nginx's active site count increasing by a whopping 27%, largely as a result of Tumblr sites now exhibiting the Server: nginx header (in previous months, most Tumblr sites did not reveal which server software they were using).

While Microsoft has shaken off many of its low-quality sites, Alibaba's nginx fork, Tengine, gained around 10 million. Most of the new sites served by Tengine this month make use of domains under the .science gTLD, which has proved popular with many Chinese link farms and webspam sites – most likely due to the sub-dollar registration costs. Tengine suffered a small net loss in active sites this month, which corroborates the low quality of the 10 million new sites.

Only 2.4% of the sites served by Tengine now qualify as active sites, which highlights just how many of them are used for displaying automatically generated content. Microsoft is still also fairly popular with link farm operators (particularly in China), with only 4.6% of its sites showing active content. In contrast, more than 26% of Apache sites, and nearly 22% of nginx sites feature active content.

Total number of websites

Web server market share

DeveloperApril 2016PercentMay 2016PercentChange
Microsoft441,470,89440.75%366,964,00935.50%-5.26
Apache292,043,54826.96%300,447,47029.06%2.10
nginx143,349,43913.23%163,902,97115.85%2.62
Google20,597,6051.90%21,567,2522.09%0.18
Continue reading

Bangladesh government exporting live phish

Bangladesh is one of the world's largest producers of fish; but lately, its government has also become an inadvertent exporter of phish.

Over the past week, several phishing sites have popped up on Bangladeshi government websites, under the .gov.bd second-level domain. These fraudulent sites have been used in phishing attacks against customers of Wells Fargo bank, Google, AOL, and other email providers.

One of the phishing sites currently using a .gov.bd domain is hosted on a website belonging to the Bandarban Technical Training Center in Bangladesh. This site imitates Google Docs in an attempt to steal victims' email credentials, whichever mail providers they use.

One of the phishing sites currently using a .gov.bd domain is hosted on a website belonging to the Bandarban Technical Training Center in Bangladesh. The fraudulent content imitates Google Docs in an attempt to steal victims' email credentials, whichever mail providers they use.

Domain name registrations under .gov.bd are restricted to government-related entities in Bangladesh, although it is unlikely that the government is directly responsible for these attacks. As with most phishing sites, the fraudulent content has probably been placed on these government sites by remote hackers; nonetheless, this would make the Bangladesh government at least responsible for poor security.

The vast majority of websites under .gov.bd are hosted within Bangladesh, but the apparently-compromised server involved in these attacks is one of a few that are hosted in the United Kingdom, on a static IP address used by the hosting company Nibs Solutions. No Bangladeshi servers are currently serving phishing sites from .gov.bd domains.

After more than a week since this spate of phishing attacks started appearing on UK-hosted .gov.bd sites, none of the fraudulent content has been removed. The presence of multiple live phishing sites on the affected server, and the fact that the previous compromises have not yet been cleaned up, suggests that whatever security vulnerabilities might have affected the server are yet to be resolved.

Detected just over a week ago, the oldest phishing site in this spate of attacks targets Wells Fargo customers and remains accessible today on the Jessore Technical Training Center website at jessorettc.gov.bd. This training center was established by the Government of the People's Republic of Bangladesh in 2004, hence its eligibility to use the .gov.bd domain.

Detected just over a week ago, the oldest phishing site in this spate of attacks targets Wells Fargo customers and remains accessible today on the Jessore Technical Training Center website at jessorettc.gov.bd. This training center was established by the Government of the People's Republic of Bangladesh in 2004, hence its eligibility to use the .gov.bd domain.

Bangladesh has a relatively small presence on the web, with just over 30,000 websites making use of the entire .bd country code top-level domain. However, the ratio of phishing incidents to sites is quite high at roughly 1 in 100.

Users of the Netcraft anti-phishing extension are already protected from these attacks, including the examples shown above, even though the fraudulent content has not yet been removed by the sites' administrators.

Most Reliable Hosting Company Sites in April 2016

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Datapipe Linux 0:00:00 0.000 0.160 0.012 0.024 0.031
2 Qube Managed Services Linux 0:00:00 0.000 0.153 0.058 0.117 0.117
3 CWCS Linux 0:00:00 0.000 0.189 0.070 0.142 0.143
4 Pair Networks FreeBSD 0:00:00 0.004 0.246 0.070 0.143 0.143
5 GoDaddy.com Inc Linux 0:00:00 0.009 0.257 0.010 0.024 0.025
6 XILO Communications Ltd. Linux 0:00:00 0.009 0.222 0.063 0.127 0.127
7 Kattare Internet Services Citrix Netscaler 0:00:00 0.009 0.517 0.114 0.228 0.228
8 LeaseWeb Linux 0:00:00 0.013 0.351 0.029 0.055 0.055
9 Hyve Managed Hosting Linux 0:00:00 0.013 0.222 0.060 0.120 0.120
10 Aspserveur Linux 0:00:00 0.013 0.343 0.077 0.326 0.471

See full table

Datapipe had the most reliable hosting company site in April, responding to all of Netcraft's requests. Datapipe's performance has seen it appear in the top ten every month so far in 2016, continuing a streak which has placed the company in the top ten 11 times in the past 12 months. Datapipe provides hosting services out of a number of data centres in Europe, Asia and North America. In April, Datapipe announced that it would be partnering with Singapore's largest electronics retailer to ensure the scalability of the latter's online infrastructure.

Qube had the second most reliable hosting company site. As with Datapipe, Qube's site responded to all of Netcraft's requests, but was fractionally slower to do so. Qube's performance is also consistent: the company has appeared in the top ten 9 times over the last 12 months.

CWCS achieved third place in April, also with a 100% response rate, albeit with an average connect time that was marginally slower than both Datapipe and Qube's. CWCS provides shared and managed hosting solutions, with data centre facilities in England and North America, and counts organisations such as KPMG and the University of York amongst its clients.

Linux is once again the most popular choice of operating system with hosting companies: eight out of the top ten companies in April hosted their websites on Linux machines. Kattare Internet Services and Pair Networks were the only two exceptions, using a Citrix Netscaler device and FreeBSD respectively.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

Hook, like and sinker: Facebook serves up its own phish

Fraudsters are abusing Facebook's app platform to carry out some remarkably convincing phishing attacks against Facebook users.

A phishing site displayed on the real Facebook website.

A phishing site displayed on the real Facebook website.

Masquerading as a Facebook Page Verification form, this phishing attack leverages Facebook's own trusted TLS certificate that is valid for all facebook.com subdomains. This makes the page appear legitimate, even to many seasoned internet users; however, the verification form is actually served via an iframe from an external site hosted by HostGator. The external website also uses HTTPS to serve the fraudulent content, so no warnings are displayed by the browser.

The phishing attack does not require the victim to be already logged in.

The phishing attack does not require the victim to be already logged in.

This phishing attack works regardless of whether the victim is already logged in, so there is little chance of a victim being suspicious of being asked to log in twice in immediate succession.

The source code of the phishing content reveals that it sends the stolen credentials directly to the fraudster's website.

The source code of the phishing content reveals that it sends the stolen credentials directly to the fraudster's website.

To win over anyone who remains slightly suspicious, the phishing site always pretends that the first set of submitted credentials were incorrect. A suspicious user might deliberately submit an incorrect username and password in order to test whether the form is legitimate, and the following error message could make them believe that the credentials really are being checked by Facebook.

The phishing site always pretends the first submitted credentials are incorrect.

The phishing site always pretends the first submitted credentials are incorrect. Note that it now also asks for the victim's date of birth.

Those who were slightly suspicious might then believe it is safe to enter their real username and password. Anyone else who had already entered the correct credentials would probably just think they had made a mistake and try again. After the second attempt, the phishing site will act as if the correct credentials had been submitted:

On the second attempt, the phishing site will ask the victim to wait up to 24 hours.

On the second attempt, the phishing site will ask the victim to wait up to 24 hours.

The final response indicates that the victim will have to wait up to 24 hours for their submission to be approved. Without instant access to the content they were trying to view, the victim will probably carry on doing something else until they receive the promised email notification.

But of course, this email will never arrive. By this point, the fraudster already has the victim's credentials and is just using this tactic to buy himself some time. He can either use the stolen Facebook credentials himself, or sell them to others who might monetize them by posting spam or trying to trick victims' friends into helping them out of trouble by transferring money. If more victims are required, then the compromised accounts could also be used to propagate the attack to thousands of other Facebook users.

Some of Facebook's security settings.

Some of Facebook's security settings.

However, Facebook does provide some features that could make these attacks harder to pull off. For example, if login alerts are enabled, the victim will be notified that their account has been logged into from a different location – this might at least make the victim aware that something untoward is going on. Although not enabled by default, users can completely thwart this particular attack by activating Facebook's login approvals feature, which requires a security code to be entered when logging in from unknown browsers. Only the victim will know this code, and so the fraudster will not be able to log in.

April 2016 Web Server Survey

In the April 2016 survey we received responses from 1,083,252,900 sites and 5,800,222 web-facing computers. This reflects a gain of nearly 80 million sites and 18,100 computers.

This is the largest number of sites the survey has ever seen, beating the previous maximum of 1,028,932,208 in October 2014. The number of web-facing computers is also at its largest, although this total has generally risen much more steadily than the number of sites.

Microsoft was the only major vendor to gain sites this month, and so it was solely responsible for this month's total reaching its highest value ever. Apache lost 33 million sites, while nginx and Google suffered much smaller losses. Many of the 124 million additional sites using Microsoft IIS are aimed at a Chinese audience. Several million are served from just a handful of IP addresses, using either IIS 6.0 or 7.5.

However, this proliferation of new Microsoft-powered websites is largely driven by automated processes. Many are "spam" sites that use link farming techniques to attract traffic. Although Microsoft's website count grew by a remarkable 38.9% in April, it lost 12,100 web-facing computers. High quality websites that attract genuine repeat traffic tend to have a very low number of sites per computer compared with the computers that are involved in link farming, which sometimes host millions of automatically-generated sites each. Corroborating this further, Microsoft suffered a loss of 341,000 active sites this month, taking its total down by 2.0%.

Meanwhile, nginx continued its relentless growth. It gained 19,500 web-facing computers this month (+2.4%), was the only major vendor to increase its active sites count, and increased its share within the top-million websites by 0.49 percentage points.

nginx is particularly prominent at Amazon and DigitalOcean, with the two hosting companies accounting for more than 25% of all nginx computers. In particular, nginx is the most commonly used server at DigitalOcean, being used by just under half of its web-facing droplets. At Amazon, despite its large share of all nginx computers, Apache is more than twice as common, with nginx only used on a quarter of EC2 instances.

Total number of websites

Web server market share

DeveloperMarch 2016PercentApril 2016PercentChange
Microsoft317,761,31831.65%441,470,89440.75%9.10
Apache325,285,18532.40%292,043,54826.96%-5.44
nginx143,464,29314.29%143,349,43913.23%-1.06
Google20,790,7672.07%20,597,6051.90%-0.17
Continue reading

Most Reliable Hosting Company Sites in March 2016

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 Qube Managed Services Linux 0:00:00 0.000 0.148 0.058 0.118 0.118
2 Netcetera Linux 0:00:00 0.000 0.078 0.082 0.166 0.166
3 Datapipe Linux 0:00:00 0.008 0.158 0.012 0.024 0.031
4 Kattare Internet Services Citrix Netscaler 0:00:00 0.008 0.518 0.116 0.231 0.231
5 GoDaddy.com Inc Linux 0:00:00 0.013 0.244 0.006 0.018 0.018
6 ServerStack Linux 0:00:00 0.013 0.130 0.066 0.132 0.132
7 www.dinahosting.com Linux 0:00:00 0.013 0.267 0.081 0.163 0.163
8 Hyve Managed Hosting Linux 0:00:00 0.017 0.230 0.059 0.118 0.119
9 Pair Networks FreeBSD 0:00:00 0.017 0.239 0.070 0.142 0.142
10 Anexia Linux 0:00:00 0.017 0.197 0.085 0.181 0.181

See full table

Qube had the most reliable hosting company site in March, responding to all of Netcraft's requests. This continues Qube's strong performance from last year, where it placed in the top ten in eight months of 2015, and both January and February of 2016. Qube is based in London and offers a range of managed services, including private cloud hosting, from data centres in London, New York and Zurich.

In a very close second place, Netcetera also responded to every Netcraft request, but with a slightly longer average connection time. Netcetera owns and operates a carbon neutral data centre on the Isle of Man, and recently celebrated its 20th birthday, having been in the hosting business since 1996.

Datapipe takes third place in March, with two failed requests. Datapipe has an exceptionally consistent record for reliability, with its site maintaining 100% uptime over 10 years, and appearing in the top ten list on 42 occasions over the last 48 months.

Linux remains the most common choice of operating system amongst the most reliable hosting company sites, powering eight out of the top ten. Citrix Netscaler and FreeBSD also make an appearance, being employed by Kattare Internet Services and Pair Networks respectively.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.