Netcraft has developed a technique to identify site operators who control large numbers of internet web sites, but host the sites with external companies rather than develop their own datacenters. These site operators may include web site development companies, shared hosting systems, domain registrars, managed service companies, and private label hosting resellers.

Because of their control over large numbers of sites, and their ability to move from one infrastructure provider to another, these site operators often make attractive prospects for hosting service providers, and hosting automation software companies.

Netcraft identifies such companies by comparing the results of a DNS lookup on each site found by the Web Server Survey with the results of a DNS lookup on its ip address. Typically, the name server in control of the hostname belongs to the site operator, and the name server in control of the ip address belongs to the infrastructure provider.

The dataset covers all 22,000 site operators we find with 20 or more active sites in the United States hosted at some 2,250 hosting locations with 200 or more active sites. We preserve the mappings between each, such that there is a clear identification of the relationships between infrastructure providers and the site operators present on their networks. An analysis by operating system platform, is available and similar datasets can be produced on demand for other countries.

Example - Top 10 Site Operators using Verio in the United States, May 2003

Limitations

• Only sites found by the Web Server Survey will be included. The number of hosts found worldwide running internet web sites by the Web Server Survey is large [over 40 million in April 2003], but is not exhaustive.
• Fully attributing a site requires successful DNS lookups on each of the hostname and ip address. If a DNS lookup fails then the site operator and/or hosting location will be unknown to us.
• Name servers that provide DNS for sites hosted on 5 or more Infrastructure providers’ networks are removed from the dataset. We have found that companies that provide a DNS service to sites situated on many providers’ networks are more likely to be specialist DNS providers than web site developers or hosting services resellers.
• Sites where the hosting provider name server is authorative for both the ip address and domain name are tagged so that they can be removed from view - this is the conventional situation where a hosting company runs DNS for its customers, and there is likely to be no third party involved in the relationship.

More Information

The dataset is updated on a monthly basis and is available on a company license basis.

Please contact us (sales@netcraft.com) for further information of costs.

The US Air Force has moved its most public site away from the Defense Technical Information Center to a commercial hosting datacenter at Cable & Wireless. Microsoft will be delighted to have replaced Sun as the server platform. In 1999 Microsoft suffered the indignity of the main US Army web site operations staff not only replacing Microsoft servers with Macs but also giving their opinion of Microsoft product security on the front page of www.army.mil.

However, Cable & Wireless have subsequently announced their intention to sell or close their US datacenters, which may mean that www.af.mil will move again. Since this was not completely unexpected, it makes the timing of the US Army's move seem curious.

The Netcraft Web Server Survey is a survey of Web Server software usage on Internet connected computers. We collect and collate as many hostnames providing an http service as we can find, and systematically poll each one with an HTTP request for the server name.

In the May 2003 survey we received responses from 40,444,778 sites.

Market Share for Top Servers Across All Domains August 1995 - May 2003

Top Developers

Developer	April 2003	Percent	May 2003	Percent	Change
Apache	25090844	62.57	25290478	62.53	-0.04
Microsoft	11007434	27.45	10989025	27.17	-0.28
Zeus	789324	1.97	780936	1.93	-0.04
SunONE	415999	1.04	420398	1.04	0.00

Some people expressed surprise that there could be thousands of sites running on Windows Server 2003 before it went on sale. The main reason for this is that some shared hosting providers who were part of the Windows Server 2003 beta program were prepared to run part of their sites on Windows Server 2003 prior to launch. The three providers with the largest number of active sites on Windows Server 2003 are all Microsoft Certified Partners while the fourth largest is Microsoft itself. No one else had more than 1000 active sites running on Windows Server 2003 at the start of May.

Top Microsoft Windows Server 2003 hosting companies by Active sites, May 2003 Company Active Sites   Interland 4,996   Schlund 3,125   Fasthosts 2,250   Microsoft 811

For comparison, Schlund and Interland [primarily due to its acquisition of Dialtone Internet] are also two of the largest Linux hosting providers, along with specialist Linux hosting providers Rackshack and Alabanza.

Top Linux hosting companies by Active sites, May 2003 Company Active Sites   Schlund 1,450,868   Rackshack 234,740   Alabanza 150,180   Interland 144,488   About 118,287   Valueweb 109,998   Verio 102,696

Ironically, Richard Stallman best known for the creation and relentless advocacy of the General Public Licence, has for many years run his own site using software produced under the more generous BSD licence. www.stallman.org switched from FreeBSD to Linux during April.

PC Magazine has awarded Netcraft a place in its 2003 Top 100 Classic Web Sites. Classic sites are defined as "perennial favorite, category leading sites", which one would most miss if stranded on a desert island.

PC Magazine is one of the world's leading computer publications with 6 million readers.