Netcraft Hosting Provider Server Count Available

Netcraft has developed a technique for identifying the number of computers [rather than IP addresses] acting as web servers on the internet, and attributes these computers to hosting locations by reverse DNS lookups.

This provides an independent view, with a consistent methodology worldwide, on the numbers of web servers, the rate of growth over time, and the operating systems and web server technology used at each hosting provider worldwide.

The dataset is useful to hosting providers for competitive analysis, mergers and acquisitions, identifying international markets for organic expansion, and also to any organization selling to the hosting industry.

It is presented as an Excel spreadsheet where the user may pivot by country, operating system, hosting model [dedicated, shared, bulk/domain registry as calculated from the ratio of sites to servers], and drill down to inspect absolute numbers, rate of growth and technology deployed at individual companies.

Introduction

One of the common observations made about the Web Server Survey is that it counts hostnames rather than physical computers, and so is not a suitable metric to indicate the number of servers at a particular ISP. Technically sophisticated hosting providers can run thousands of sites on a single computer, and the great majority of the world's web sites are located at hosting datacenters rather than on peripheral networks.

By arranging for a number of IP addresses to send packets to us near simultaneously, low level TCP/IP characteristics can be used to work out, within an error margin, if those packets originate from the same computer, by checking for similarities in a number of TCP/IP protocol header fields. To build up sufficient certainty that IP addresses on the same computer have been identified many visits to the sites in the Web Server Survey are necessary, which takes place over a period of over a month.

Limitations

• Only sites found by the Web Server Survey will be included. The number of hosts found running internet web sites by the Web Server Survey is large [over 40 million in April 2003], but not exhaustive.
• Sites are attributed to companies by performing a reverse DNS lookup on each responding IP address in the Web Server Survey. If reverse DNS lookups have not been configured or otherwise fail, the count for the company will correspondingly reduced. To mitigate this we provide an additional view of the data compiled by Netblock registration. In practice the most successful hosting companies seem to set up reverse DNS correctly.
• Backend machines such as database servers not running web sites will not be counted, as they are unseen from the Internet.
• At most one server will be counted for each site. Round robin DNS, reverse web proxies, load balancing products like Cisco Local Director and BIG-IP and some connection level firewalls hide multiple web servers behind a single hostname. Additionally with some of these products the operating system detected is that of the "front" device rather than the web server behind.

Error Margins

There are some other factors that create errors in our technique, the main ones being:

• Despite making multiple visits, there is still a low probability that two computers will be considered the same by chance similarities in low-level TCP/IP protocol header fields; this leads to under-counting.
• Some IP addresses do not respond on enough visits for the technique to be applied. This is mainly due to computers or networks being down or badly overloaded on several of the visits, in which case there are uncounted computers; by extrapolating using the IP address/computer ratio for sites running similar software we can roughly correct for this error.
• If a system changes or upgrades operating system during the course of the survey, a computer may be counted more than once; this leads to over-counting, but generally this should be a fairly small effect.

A useful piece of evidence that suggests there are not large levels of error is that the average ratio of sites to computers on hosting company networks is over 10, whereas the ratio of self hosted sites to computers is about 2.

How does it pan out in practice?

The natural comparison metric for our computer counting results is the number of servers in your own datacenter. Because our internet-based view will not see backend servers that do not run web servers, and will only count a load balanced system as a single computer, you would expect our numbers to undercount the numbers of machines in your datacenter.

How much lower depends on the type of site hosted: leading sites like Amazon and Google may have tens of thousands of computers of which our survey would only detect a few.

Conversely, in a dedicated server company, most of the computers would likely be running web servers, with only a small proportion of the customers having multiple machines. In this scenario, the number of computers reported is likely to be within 50% of the actual number of servers in the datacenter. Informal checking with two of the largest dedicated server companies determined that we report a total of about ⅔ – ¾ of the machines in their datacenters.

The technique is likely to be at its most useful in comparisons of companies operating shared or mixed shared & dedicated business models, where comparisons of numbers of IP addresses may lead to misleading results. Some US companies [including Alabanza, Interland, & Verio] have over 100,000 IP addresses each supporting shared hosting. Our technique brings the number of servers at those locations into the right order of magnitude.

Worldwide, IP address allocations are very uneven with the US having a much more generous allocation of IP addresses than non-US countries. The computer counting technique reduces the geographical skew of an IP address based view of the internet, and whereas 18 of the top 20 hosting locations are US companies when counting by IP address, when using the computer counting technique this changes to 12 of the top 20. This makes comparison of companies operating in geographical locations possible, and is a useful precursor to international expansion or acquisitions.

Netcraft has been performing this survey since February 1999, generally four times a year. The trends since then have been very smooth suggesting there is only a small amount of "random error" in this analysis.

More Information

The dataset is available on a company licence basis. Please contact us (sales@netcraft.com) for further information.