May 2017 Web Server Survey

In the May 2017 survey we received responses from 1,814,996,345 sites and 6,404,290 web-facing computers. Although the total number of sites has fallen slightly, by 1.4 million since April, the number of computers has grown by 83,000.

Large shifts in site counts

Although it fared well in other metrics (see below), nginx suffered a massive 30% loss of hostnames this month. With a net loss of more than 100 million sites, its market share has fallen by 5.71 percentage points to 13.5%. The majority of the sites that disappeared were Chinese-language spam sites hosted at Amazon Web Services in the United States and Japan.

Meanwhile, Microsoft gained 79 million sites, which has taken its market share up to 49.1%. This is Microsoft's highest market share in the 22-year history of the Web Server Survey, and also reflects a significant change in fortunes over the past year: Last June, Apache had the largest market share of websites, but now Microsoft's share is more than twice as large as Apache's, which fell by a further 1.73 percentage points this month to 20.95%.

nginx leads active site growth

Despite the large loss of hostnames, nginx gained nearly a million active sites, which has taken its active sites share back above 20%. This suggests most of the 104 million hostnames it lost did not have distinct content, and therefore were of little interest to ordinary web users, despite the apparent size of the change.

Apache continues to lead the active sites market quite comfortably with a 45.6% share, although a loss of 1.4 million active sites has brought this down by 0.67 percentage points. nginx stays firmly in second place, with more than twice as many active sites as Microsoft.

nginx's computer growth continues unabated

nginx's consistent computer growth has continued, making it look ever more likely to overtake Microsoft later in 2017. Its gain of 48,500 computers – combined with Microsoft's loss of 6,000 – has reduced the difference in their market shares by nearly a whole percentage point.

nginx is now within 3.46 percentage points of Microsoft's share of 23.8%; but Apache also maintains a comfortable lead in this market – it increased its web-facing computer count by 28,800 this month, keeping Apache's share above 43%.

nginx pushing others out of the top million

nginx is also continuing to make strong progress amongst the top million websites, where it has been ahead of Microsoft for the past few years. It was the only major vendor to increase its presence this month, resulting in thousands of competing vendors' websites being pushed out of the top million. Apache suffered most, with 3,200 Apache-powered sites departing the top million, but it still leads this market with a 40.5% presence.

Apache has exhibited a slow and steady decline over the past several years. Coupled with nginx's consistent growth within the top million sites, the gap between the two is ever decreasing; however, it looks like it will be a good year or two until nginx seriously starts to threaten Apache's lead.

New nginx 1.13 mainline branch

nginx 1.13.0 was released on 25th April. This is the first release on the new, actively-developed 1.13.x mainline branch, adding bugfixes and new features to what was essentially the most recent stable version of nginx (1.12.0).

The most notable new feature in nginx 1.13.0 is its support for TLS 1.3, which aims to be the latest and most secure version of the Transport Layer Security protocol – although not many underlying crypto libraries actually offer TLS 1.3 yet. The TLS 1.3 specification has not yet been finalised, although the working draft has been sufficient for some servers and clients to implement it. For example, Mozilla's NSS cryptographic library – which is used by Firefox – enables TLS 1.3 support by default.

Microsoft IIS Administration API enters general availability

The Microsoft IIS Administration API that we mentioned in February is now Generally Available. The newest 1.1.0 release also facilitates management of the IIS Central Certificate Store, and includes an improved certificate API. These features are intended to make it easier to manage certificates across entire farms of web servers.

LiteSpeed enters the Amazon cloud

LiteSpeed Technologies is now a Technology Partner in the Amazon Web Services Partner Network (APN), coinciding with the release of the LiteSpeed Web Server AMI. This Amazon Machine Image is based on CentOS 7 with the LiteSpeed Web Server pre-installed. AWS users can now use this AMI to quickly deploy ready-to-use virtual machines that run the LiteSpeed Web Server in the cloud.

This month's Web Server Survey found 6.4 million websites running LiteSpeed, which encompasses 2.3 million active sites, more than 2 million unique domain names, and nearly 11,000 web-facing computers. One of the busiest sites currently using LiteSpeed is FanFiction.Net, which is an automated fan fiction archive.

Enter the Beaver

China saw a large number of new sites being served by the relatively unknown "Beaver" web server. Just over a million sites now exhibit the Beaver Server header, and these make use of more than 110,000 unique domain names – mostly under the .cn top-level domain. Most of the sites are hosted by Aliyun, which is China's largest cloud hosting provider, while the majority of the rest are hosted by other Chinese companies. Only a single Beaver site is hosted outside of China – this solitary instance is hosted in Japan, at Amazon's Tokyo AWS region.

The behaviour of these sites suggests that Beaver might not be an entirely new web server, but possibly an application based on Microsoft's HTTP Server API (HTTPAPI 2.0). This API lets C/C++ programmers receive HTTP requests and send responses without using Microsoft IIS. At least 38 million other websites also use HTTPAPI 2.0.

But most of the Beaver sites are currently inaccessible and display the following message: "According to the filing requirements of China's Ministry of Industry and Information Technology (MIIT), the website is accessible only if the ICP information is accurate and the ICP license is filed". An ICP Filing ("Bei'an") is required by all content providers in China before they can use hosting and CDN products, but this only allows them to be used for informational purposes. A Commercial ICP Licence ("Zheng") is required for any website that sells goods or services that directly generate revenue online.

Pepyaka making the web more friendly

The little-known Pepyaka web server has also been quietly growing. This server is used predominantly by Wix, an Israeli company that provides a friendly website building platform to millions of users. Wix provides free website hosting under the *.wix.com domain, or customers can get a free custom domain name when upgrading to one of Wix's yearly premium plans.

This business model has led to a high ratio of domains to hostnames amongst the Pepyaka install base. Wix-hosted websites alone account for 1.4% of all unique domain names in use on the web, which is no mean feat. All of these sites are hosted within the Amazon Web Services cloud, using Pepyaka version 1.11.3. This version number, coupled with Wix's previous uses of nginx, suggests that it could be based on last year's mainline version of nginx.

Developer	April 2017	Percent	May 2017	Percent	Change
Microsoft	812,157,808	44.71%	891,000,721	49.09%	4.38
Apache	412,130,526	22.69%	380,321,106	20.95%	-1.73
nginx	349,092,975	19.22%	245,114,317	13.50%	-5.71
Google	19,121,684	1.05%	20,033,229	1.10%	0.05

Continue reading →

Posted by Netcraft on 25th May, 2017 in Web Server Survey

Web Shells: The Criminal’s Control Panel

Web shells are an overlooked aspect of cyber crime and do not attract the level of attention of either phishing or malware. Nevertheless, Netcraft found more than 6,000 web shells during April 2017, which works out at around 1 new shell installation every 5 minutes. When web shells first appeared, the limit of their functionality was to transfer files and execute arbitrary shell commands. However, the best engineered web shells now provide well presented, sophisticated toolkits for diverse crimes, with facilities for password cracking, privilege elevation, network reconnaissance, phishing, spamming and DDoS, not solely available through a web based user interface but also accepting commands as part of a botnet.

An example of the hugely popular and feature-rich WSO (Web Shell by Orb) shell.

A number of shells offer the creation of a botnet in as little as a click, launching standalone processes that either connect to a command and control server or listen for commands over an insecure TCP connection. Some allow performing port scans to find potentially exploitable services. Others enable fraudsters to schedule denial of service attacks. There are shells dedicated to sending bulk spam emails, testing stolen credentials against popular websites (such as PayPal or Amazon), cracking passwords, and automatically defacing websites. With such a wide array of powerful features, it is unsurprising how popular web shells are with cyber criminals.

The WSO shell offers both bind shell and back connect options. Selecting one of these options will launch a standalone process that will connect to or listen for a connection from a remote command and control server - an easy method for the creation of a botnet.

WSO offers both bind shell and back connect options. Selecting one of these options will launch a standalone process that will connect to or listen for a connection from a remote command and control server - an easy method for the creation of a botnet.

The prevalence of these backdoors allows easy—and potentially persistent—access to thousands of compromised machines. If the web shell is missed during the webmaster's cleanup after an attack, removing the original phishing or malware content will be in vain, as the fraudster can use the web shell to upload new malicious material, or re-purpose the machine as an accessory to alternative forms of cyber crime.

Continue reading →

Posted by George Field on 18th May, 2017 in Around the Net, Netcraft Services, Security

Phishing sites react promptly to new browser changes

The number of phishing sites making use of HTTPS has increased noticeably since January, coinciding with the introduction of a new feature in the Mozilla Firefox and Google Chrome web browsers.

Both Firefox and Chrome now display warnings when an unencrypted (HTTP) webpage contains a password field. This behaviour is intended to protect users from man-in-the-middle attacks, and also encourages the affected websites to start using secure HTTPS connections when handling sensitive data.

This German PayPal phishing site uses the unencrypted HTTP protocol, causing the latest version of Firefox to display an unmissable warning message when the user interacts with the login form.

These warning messages could scupper many phishing sites: Most are served over unencrypted HTTP connections, and so another positive consequence of the new browser behaviour is that potential victims are less likely to fall for phishing attacks.

However, fraudsters may have quickly realised this, as there has been a dramatic increase in the number of phishing sites making use of HTTPS. If the new browser behaviour has driven this change — and the timing suggests it might have — then it may have also had the unintended side effect of increasing the efficacy of some phishing sites. Phishing sites that now use HTTPS and valid third-party certificates can appear more legitimate, and therefore increase the likelihood of snaring a victim.

Firefox 51 and Chrome 56 were the first stable browsers to flag HTTP websites as insecure if they contained password fields. Their release dates appear to coincide with the increase in HTTPS phishing sites.

Firefox 51 and Chrome 56 were the first stable browsers to flag websites as insecure if they contained password fields. Their release dates appear to coincide with the increase in HTTPS phishing sites.

Another plausible hypothesis is that many legitimate websites have migrated to HTTPS in response to the new behaviour in Firefox and Chrome. Phishing sites are often hosted on compromised websites, and so this would naturally cause the number of HTTPS phishing sites to increase accordingly; or it could be that some fraudsters are now targeting HTTPS websites in preference to HTTP sites.

While the majority of today's phishing sites still use the unencrypted HTTP protocol, a threefold increase in HTTPS phishing sites over just a few months is quite significant. Regardless of what caused this change, phishing sites that use the unencrypted HTTP protocol could still prove effective against some victims, as not all browsers share the behaviour implemented in Firefox and Chrome. In particular, Microsoft's Internet Explorer and Edge browsers do not yet display any warnings when users interact with insecure forms.

Posted by Paul Mutton on 17th May, 2017 in Security

Panama’s abundance of phish!

According to tradition, the country of Panama was named after a former indigenous fishing village and its nearby beach called Panamá, meaning "an abundance of fish"; but today, it looks like Panama has an abundance of phish!

Netcraft has blocked nearly 5,000 phishing sites in Panama over the past three months, which is an astounding amount considering Panama hosts fewer than 13,000 active websites in total.

Nearly 4,000 phishing sites are still blocked, making Panama the phishiest country in the world at the present moment. To give these figures some perspective, only 0.007% of the world's active sites are hosted in Panama, yet it hosts 1.0% of all phishing sites that are currently blocked.

An Apple ID phishing site currently hosted by Offshore Racks in Panama.

Around 1.9 million people are estimated to use the internet in Panama, but most of the phishing sites hosted there are clearly aimed at foreigners, as the majority are not written in Panama's official language of Spanish. In fact, most of the currently blocked phishing sites target customers of Italian banks, and a large proportion of new phishing sites found in Panama over the past month were written in English and targeted Apple customers.

Most of the Apple phishing attacks make use of domain names that have been registered specifically to carry out these attacks, with many containing obvious references to Apple, Apple ID, or iCloud.

A handful of the domains used by Apple phishing attacks last month.

A few examples of the domains used by Apple phishing attacks last month.

The majority of these phishing sites are hosted by Offshore Racks, a Panamanian hosting company that offers "high privacy" anonymous hosting and accepts payment in Bitcoins – ideal for fraudsters who do not want to be traced easily.

As the phishing sites make use of domain names that have been registered specifically for phishing, this suggests the fraudsters have purposely sought their own hosting arrangements, rather than adopting the more common method of deploying phishing kits on compromised web servers. While this eliminates the risk of the phishing content being deleted by the disgruntled owner of a compromised site, the obvious disadvantage for the fraudster is that he may have to pay for both domain registrations and hosting.

Offshore Racks' Acceptable Use Policy has said nothing more than "In development..." since 2010.

While it is clear that the company responsible for hosting most of these phishing sites could be doing more to prevent the attacks, domain name registrars and domain registries are also well positioned to nip this activity in the bud. Netcraft's Deceptive Domain Score service can be used to analyse the likelihood of a domain name being used for fraudulent activities, giving an opportunity to prevent the registration, flag for human inspection, or immediately suspend fraudulent domains, before malicious content can be uploaded. Domains that have already been registered can be suspended by TLD operators as soon as phishing activity is detected.

Consumers can boost their browsers' standard security features by installing the Netcraft anti-phishing extension. As well as blocking access to known phishing sites, it will display the hosting location, Risk Rating and other information that can help establish the authenticity of every site visited.

Posted by Paul Mutton on 11th May, 2017 in Security

Most Reliable Hosting Company Sites in April 2017

Rank	Performance Graph	OS	Outage hh:mm:ss	Failed Req%	DNS	Connect	First byte	Total
1	One.com	Linux	0:00:00	0.004	0.185	0.040	0.116	0.116
2	Swishmail	FreeBSD	0:00:00	0.004	0.132	0.060	0.120	0.164
3	Hyve Managed Hosting	Linux	0:00:00	0.004	0.075	0.062	0.127	0.127
4	Webair	Linux	0:00:00	0.009	0.144	0.054	0.109	0.110
5	XILO Communications Ltd.	Linux	0:00:00	0.009	0.210	0.067	0.134	0.134
6	CWCS	Linux	0:00:00	0.009	0.194	0.077	0.176	0.176
7	GoDaddy.com Inc	Linux	0:00:00	0.013	0.219	0.012	0.031	0.032
8	Bigstep	Linux	0:00:00	0.013	0.129	0.063	0.128	0.128
9	Qube Managed Services	Linux	0:00:00	0.013	0.131	0.064	0.129	0.129
10	www.viawest.com	Linux	0:00:00	0.017	0.265	0.005	0.194	0.194

See full table

One.com comes in first place in April, up from seventh place in March. One.com responded to all but one of Netcraft's requests, with an average TCP connection time of 40 milliseconds. The company was founded in 2002 and is based in Denmark, with staff also based in Dubai and India.

Swishmail and Hyve follow in second and third places respectively. These hosting company sites also responded successfully to all but one request, but had marginally slower average connection times than One.com's site. Swishmail is based in the United States and provides business email and web hosting services, whilst Hyve is based in the United Kingdom and provides a variety of cloud and dedicated server hosting services. Hyve reached the finals of the 2017 European IT and Software Excellence Awards under both the Managed Service Solution of the Year and Service Provider of the Year categories.

Xilo has consistently featured in the top 10 so far in 2017. Xilo provides a variety of services, ranging from shared hosting up to managed and self-managed dedicated servers, as well as domain name services and SSL certificates.

Of the top 10 hosting providers for April, all but one of their websites are running on Linux, with the exception being Swishmail's, which runs on FreeBSD.

Netcraft measures and makes available the response times of around thirty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

Posted by Netcraft on 3rd May, 2017 in Hosting, Performance

April 2017 Web Server Survey

In the April 2017 survey we received responses from 1,816,416,499 sites and 6,320,910 web-facing computers. This reflects a gain of 56 million sites and 49,800 computers.

Microsoft had a noticeable gain of 108 million sites (+15.4%), recouping last month's loss and expanding its market share by nearly 5 percentage points to 44.7%. Microsoft is 10 million sites down from the start of the year, but its share is now nearly twice as large as Apache's. The only major vendor to suffer a loss of hostnames in April was nginx, which lost 1.4 million sites – this took its market share down by 0.41 points to 19.2%, leaving it slightly further behind Apache's share of 22.7%.

Despite nginx's loss of sites, it was once again the only major vendor to increase its presence within the top million sites, increasing its count by 2,164 sites, while Apache lost 1,962; but Apache stays in the lead with a 40.9% share, while nginx's increased to 28.7%.

nginx's growth is also reflected well in the web-facing computers market, where it had the largest increase of 29,900 computers (+2.44%). This took its share up by 0.32 points to 19.9%, while Apache's leading share fell by 0.27 points to 43.5%. Microsoft is still in second place with 24.2% of the web-facing computer market, but this position looks set to be taken by nginx within the next year if the trend of nginx's gains and Microsoft's losses continue.

Apache continues to reign supreme in terms of active sites, where it increased its market share slightly to 46.3%, putting it further ahead of nginx, which has a share of 19.6%; however, the long term trend over the past several years has seen the two vendors getting closer, with nginx slowly gaining market share while Apache has slowly declined. Microsoft's share of active sites is only 8.28%, but this is enough to keep it in third place, ahead of Google.

A new mainline version of nginx (1.11.13) was released on 4 April. This release included several bugfixes and formed the basis of the current stable version, nginx 1.12.0, which was later released on 12 April. As a consequence of including all bug fixes and new features from the entire 1.11.x branch, nginx 1.12.0 includes support for configuring multiple SSL certificates of different types, better support for dynamic modules, and several other new features.

Earlier this month, Netcraft examined the success of ICANN's New gTLD program, as well as the impact it has had on brand owners such as LEGO. The .loan gTLD saw the largest domain growth of any type of TLD this month, gaining 287,000 unique domains, yet losing more than 10 million websites. Only 3% of all .loan domains are considered active by Netcraft, indicating that large numbers share near-identical content, such as monetized domain holding pages.

From the TLD operator's perspective, the rise in .loan domains is much more significant than the large reduction in sites, as each unique domain will correspond to a domain registration, which invariably involves some type of transaction. The cost of registering a single .loan domain can vary between $20-$40 per year (including ICANN fees), depending which registrar is used, although some registrars – such as Namecheap – offer low introductory prices of $0.88 for the first year only. The prevalence of domain holding pages suggests that many of these domains may have been bought at introductory prices, so the estimated revenue from the 980,000 unique .loan domains currently in use on the web is likely to be much closer to $1m than $40m.

Developer	March 2017	Percent	April 2017	Percent	Change
Microsoft	704,000,530	39.99%	812,157,808	44.71%	4.73
Apache	383,707,112	21.79%	412,130,526	22.69%	0.90
nginx	350,540,372	19.91%	349,092,975	19.22%	-0.69
Google	18,849,171	1.07%	19,121,684	1.05%	-0.02

Continue reading →

Posted by Netcraft on 21st April, 2017 in Web Server Survey

1 2 3 4 5 6 … 210 Next »

Large shifts in site counts

nginx leads active site growth

nginx's computer growth continues unabated

nginx pushing others out of the top million

New nginx 1.13 mainline branch

Microsoft IIS Administration API enters general availability

LiteSpeed enters the Amazon cloud

Enter the Beaver

Pepyaka making the web more friendly

Share:

Share:

Share:

Share:

Share:

Share:

Advertisers Directory