New York Internet rides out the storm

New York Internet looks set to make it through the aftermath of Hurricane Sandy with barely a scratch on its uptime or performance.

NYI's New York City datacenter is just a third of a mile away from the 75 Broad Street datacenter (in evacuation Zone A), which flooded during Hurricane Sandy. NYI is crucially further uphill, in Zone C, and was out of reach of the storm surge. Despite widespread power failures within the area, NYI has so far managed to continue operating without suffering from any outages:

Demand for fuel is very high in the areas affected by the storm. Phillip Koblence, founder of NYI, said that poor availability of gasoline for staff to get to and from datacenters was his worst problem, followed by sleeping and washing facilities. Queues to get gasoline were reportedly two miles long, and vehicles containing fewer than 3 people were not being allowed into Manhattan between 6am and midnight.

Koblence told Netcraft that he had no power or water at home, and difficulties getting food anywhere near work were also adding to the problems. NYI expected commercial power to be shut off as the storm hit, and its New Jersey and New York City datacenters have been running on diesel generators since Monday.

NYI's New York City datacenter is only a few blocks
away from the flooded 75 Broad Street datacenter.

NYI estimates that its 21st floor generator tank in Manhattan can last for 36 hours, while its 8th floor generator can run for more than 3 days. The Bridgewater datacenter in New Jersey is believed to be capable of lasting at least 10 days on generators, far longer than the times between each fuel delivery that the company has been sustaining this week.

Commercial power is expected to be restored to the datacenters late on Saturday night, although NYI plans to remain on generator power until the commercial power supply becomes stable.

Queuing for gasoline in Bridgewater.

Live performance graphs for can be viewed here.

Most Reliable Hosting Company Sites in October 2012

Rank Company site OS Outage
DNS Connect First
1 Datapipe FreeBSD 0.000 0.071 0.018 0.038 0.057
2 Qube Managed Services Linux 0:00:00 0.007 0.132 0.071 0.142 0.142
3 XILO Communications Ltd. Linux 0:00:00 0.007 0.331 0.132 0.388 0.668
4 iWeb Linux 0:00:00 0.010 0.137 0.085 0.171 0.171
5 INetU Windows Server 2008 0:00:00 0.013 0.092 0.090 0.266 0.535
6 Linux 0:00:00 0.017 0.212 0.089 0.444 0.683
7 Server Intellect Windows Server 2008 0:00:00 0.017 0.058 0.107 0.215 0.537
8 Multacom FreeBSD 0:00:00 0.017 0.143 0.117 0.236 0.600
9 Linux 0:00:00 0.020 0.371 0.097 0.200 0.282
10 Swishmail FreeBSD 0:00:00 0.024 0.127 0.071 0.141 0.285

See full table

Datapipe was the most reliable hosting company in October, impressively responding to all requests throughout the month, despite being in the area most affected by Hurricane Sandy.

Datapipe's corporate offices are based in the heart of Jersey City, very close to areas where analysts had predicted a 100% probability of a storm surge higher than 6 feet. The southern tip of Manhattan eventually experienced a storm surge of 13 feet. Datapipe founder and CEO Robb Allen told Netcraft that the effects of the storm were also felt as far inland as Somerset, where Datapipe's New Jersey datacenters are based.

After the grid became unstable on Monday, Datapipe switched to generators for their datacenter power supply. Even though their diesel reservoir was large enough to last at least four days, Datapipe also stationed additional fuel trucks on site in case roads became impassable.

Allen noted that previous events including grid blackouts, Hurricane Irene and 9/11 have all helped the internet connectivity and hosting industry in the US north east be better prepared for emergencies, though that there is currently a lot of strain on fuel delivery systems.

Datapipe's performance and uptime remained remarkably calm before, during, and after the storm, in stark contrast to destruction depicted in photos taken during the event.

When there isn't a storm brewing, Datapipe runs its United States data centers on entirely renewable energy and is recognized by the Environmental Protection Agency as a Green Power Partner. Datapipe's other US datacenters are located in San Jose and North Virginia.

Datapipe also has datacenters in Iceland, London, Shanghai and Hong Kong. The location of the Iceland datacenter allows for free cooling and runs on 100% renewable power using only geothermal and hydroelectric energy sources. The physical infrastructure has been built with steel instead of concrete, making much of it recyclable at the end of its life.

Netcraft measures and makes available the response times of around forty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

November 2012 Web Server Survey

In the November 2012 survey we received responses from 625,329,303 sites, a modest increase of 4.8 million sites since last month's survey.

Apache has continued its declining trend in market share, suffering the greatest loss this month of 0.77 percentage points. Although it still captures the majority of the market with a share of 57.23%, this has fallen from 65.00% since this time last year. Conversely, Microsoft enjoyed the largest gain this month, upping its share by 0.25 to 16.52% this month and by 1.07 over the past year.

In absolute terms, Apache lost 2.0 million sites this month, whereas Microsoft gained 2.3 million. nginx also showed significant growth, with an additional 1.2 million sites resulting in a small increase to its market share.

Studying only active sites, the changes in market share are reversed: Apache grew by 0.23 to 55.66%, while Microsoft fell by 0.83 to 11.53%. Within the million busiest sites, nginx was the only major developer to increase its market share, which now stands at 12.22%.

November's survey saw Tumblr suddenly become the tenth largest hosting company by number of sites. The microblogging service previously hosted its customers' sites with Softlayer, but now hosts 9.2 million sites on its own netblock. Softlayer was previously the largest hosting company in the world in terms of hostnames, but this move now places that title in the hands of Go Daddy, who host nearly 6 million sites more than Softlayer.

Many sites using the .tk country code top level domain (ccTLD) disappeared from the survey this month, as non-existent domain names under this ccTLD are no longer resolved by a wildcard DNS configuration. The .tk ccTLD belongs to Tokelau (a territory of New Zealand), and is run by Dot TK - a joint venture between the Government of Tokelau, the communication company Teletok, and BV Dot TK, which is a privately held company with offices in the Netherlands, the UK and the Isle of Man.

Dot TK offers free domain names under the .tk ccTLD, which is also used by its URL shortening service. The April 2012 report by the Anti-Phishing Working Group says that .tk domains were taken advantage of extensively by phishers, and ranked .tk as the top phishing TLD by phish per 10,000 domains in the second half of 2011.

To combat phishing and other fraud, Dot TK introduced an anti-abuse API to allow trusted partners to shut down sites using the .tk ccTLD. The report notes that this has resulted in lower-than-average uptimes for phishing sites, but has not prevented phishers from obtaining and using .tk domains in the first place.

Netcraft has produced a live table of the top 50 phishiest TLDs, based on the ratio of the number of phishing sites to the total number of sites hosted within each TLD. This ranks .tk as only the 22nd phishiest TLD today, while .to (Tonga) currently ranks as the phishiest. The .to ccTLD is run by the Tonga Network Information Center (Tonic), which is one of the few ccTLD operators that does not provide registration information in a WHOIS database. This fact alone must surely contribute to the appeal of using the .to TLD for phishing and other fraud.

DeveloperOctober 2012PercentNovember 2012PercentChange
Continue reading

Phishing Alerts for Domain Registries

Monitor phishing within your top-level domains

While some registries still perceive phishing as a content issue for hosting companies and registrars, detailed knowledge of phishing activity within their Top Level Domain(s) is very beneficial for registries. It is a key data source for identifying problematic, negligent, or fraud-friendly registrars, and an essential tool for maintaining the reputation of a TLD.

It is common for hosting companies and domain registrars to unknowingly allow their infrastructure to be used for phishing. Even seemingly respectable companies may develop a reputation as a haven for fraud though some systematic deficiency in their working practices, such as a low level of resourcing for abuse related workflow (particularly outside core working hours and during weekends), or inexperienced or less capable staff being unable to recognise and act on fraudulent content.

The most prolific hosts of .net phishing sites, October 2012

Conversely, some criminal registrars and hosting companies specialise in hosting fraudulent content, and even go so far as to advertise their services as "bullet-proof". Bullet-proof hosting companies are typically based in jurisdictions where laws may be hard to apply, and being in an informed position to decline further business from these registrars may greatly aid operational efficiency.

Professionally validated feed, relied upon throughout the Industry

Netcraft's continuously updated, professionally validated phishing feed is used throughout the Internet Infrastructure industry. In addition to Internet registries, all of the main web browsers, along with major anti-virus companies, firewall vendors, SSL Certificate authorities, large hosting companies and domain registrars use Netcraft's feed to protect their user communities. Since Netcraft first launched its anti-phishing system in 2005, over 5.2 million unique phishing sites have been detected and blocked as of September 2012.

Reporting and Analysis

Reports can be refreshed hourly, and also trended over time periods of many months, with analysis by registrar, hosting company, name server, country or phishing target.

.net phishing sites by country, October 2012

Real-time Alerts

When Netcraft validates a phishing report in your TLD, you can receive an alert and can also arrange for alerts to be passed through to registrars. Acting on these individual alerts will demonstrate that your top-level domains are not welcoming to fraud. Fraudsters adjust to these signals within a short period of time, and are themselves quite efficient at moving their operations away from parts of the DNS where they are clearly unwelcome.

A refreshable Excel spreadsheet includes details of the phishing sites under the .net TLD

Case Study - Nominet .uk

Nominet is the registry responsible for managing the .uk domain, which is one of the largest ccTLDs with over 10 million domains registered as of March 2012. Netcraft has provided Nominet with information on phishing using .uk domains since 2009, with alerts made available to individual registrars via an opt-in service.

More information

Please contact us ( for pricing or further details about any of our services.

Netcraft Taxonomy of Internet Services Companies

To provide a comprehensive view of the web hosting industry, Netcraft has researched all of the hosting locations with at least twenty web facing computers found by our Web Server Survey. Of these eleven thousand hosting locations, around seven thousand provide hosting and connectivity services, the remaining being enterprises, government or educational institutions.

Netcraft has noted the services provided by each Internet Services company and the dataset includes these classifications, together with the numbers of computers found in our Hosting Provider Server Count segmented by operating system.

Field Description
Parent Company Parent company, if applicable.
Company Company name
Number of Computers The total number of Web Facing Computers found by our Hosting Provider Server Count segmented by operating system.
Websites A list of the company's own websites.
Country The main country of the company, based on their headquarters address.
Services Notes which of the following services is offered by the company:
  • Domain Registration
  • Paid Shared Hosting
  • Free Shared Hosting
  • Dedicated Hosting
  • Reseller Hosting
  • Colocation
  • Managed Services - includes packages, software configuration, firewall maintenance, monitoring.
  • Cloud/Grid Services
  • Virtual Private Servers
  • Ecommerce & Shopping carts
  • Streaming / Podcast Hosting
  • Application Hosting / Software As A Service
  • Bespoke Web development
  • Blogs
  • E-mail hosting
  • Webmail
  • Broadband
  • IPv6 Addressing
  • Leased Lines
  • Traditional Telco Services - e.g. telephone calling plans, line rental, fax, and mobile contracts.
  • VoIP - Voice Over IP
  • SSL Certificates
Provisioning Information on online ordering including accepted payment methods and expected set-up periods.
Data Centre Locations Countries in which the company has data centres.
Control Panel Software A list of the solutions available, e.g. CPanel and Plesk
Virtualization Software A list of the solutions available, e.g. HyperV, Xen, VMware, Parallels.
Partners A list of the company's publically advertised partners, for example, Cisco, Microsoft, Dell.
Main Business The company's main business area.
Language The primary language used by the company website.
Multi-Lingual Whether the company website is available in more than one language.
Address The address of the company headquarters.
Company Contact Details The main telephone number, fax number and e-mail address of the company.
Company URLS URLs to the following pages on the company website: Contact Us, About Us, Management, Partners.
Executive Contacts The executive contacts published by the company.
Social Networking URLs to Twitter, Facebook and LinkedIn pages for the company.
Stock Market Information URL to Google Finance page for the company, if publicly quoted.

The dataset is available in Excel format, making it simple to filter and sort the information, and allowing companies offering similar services to be compared.


The dataset is available on a company license basis. We are able to provide subsets of the data, for example, all hosting companies that offer cloud services in North America or VPS providers in Europe, or any other segmentation by geography or technology.

On demand, the classification could be extended to include smaller hosting companies and resellers.

Please contact us ( for further information and costs.

Domain Registration Risk Service now available

The Domain Registration Risk Calculator is a tool for domain registrars to analyse the likelihood that new domains will be used for fraudulent activities. The service identifies domains which are deceptively similar to legitimate websites run by banks and other institutions commonly targeted by phishing attacks.

Since such registrations are often made using stolen credit cards, there are significant advantages to the registrar in refusing them.

Netcraft has blocked well over five million phishing attacks since 2005, and our phishing feed is used by all of the major web browsers, and also by leading anti-virus companies, domain registrars, registries, certificate authorities and hosting companies. Our extensive experience in identifying, validating and eliminating phishing sites has provided us with a wealth of knowledge of the tricks that are used by fraudsters to create a deceptive domain name. We analyse our database of over six thousand organisations which have been targeted by phishing attacks to extract a comprehensive set of homoglyphs that could be used to convert bona fide domains to fraudulent ones. Example transformations are the corresponding characters from an IDN alphabet, or ASCII character set substitutions such as replacing “o” (letter O) with “0” (zero), or replacing “l” (lower-case letter l) with “1” (digit one), or simply appending or prepending strings such as update or secure.

A Facebook phishing site, along with its Domain Registration Risk score

The service computes a registration risk score for a proposed domain, which gives a measure of the likelihood that this candidate domain may be used to host a phishing attack. We do this by using the results of two algorithms:

  • The first algorithm, Phish target score compares the candidate domain to each of the frequently-phished legitimate domains we have on record. This comparison is done on a per-character basis, and the score is formed by looking at the minimum set of edits required to map from one to the other.

    The algorithm recognises certain tricks commonly used in domain names to deceive victims, such as double letters ( or confusing characters or combinations of characters ( We also check against a list of deceptive prefixes and suffixes that are frequently used by phishing sites, including signin and verify.

    As well as using a set of fixed rules, this algorithm also retains the flexibility to match new mappings and edits that have not been seen before. Using the suggested cut-off of a minimum score of 5/10, this method identifies 278 (12.7%) out of the 2,191 phishing domains currently blocked by Netcraft.

  • The second algorithm, String entropy score, works entirely differently. Many phishing domains in our database are essentially random strings of alphanumeric digits, yet very few legitimate sites follow this pattern. The string entropy test looks to see if a domain looks like a combination of real dictionary words and plausible names, or whether it looks more like a randomised string. The higher the score, the more random a string appears to be.

    Although most dictionary strings score zero, the suggested cut-off is a minimum score of 5/10; any domain scoring higher than this is very likely to be random, but below this score false positives are increasingly likely.

    Using the suggested cut-off identifies 474 (21.6%) of the 2,191 identified phishing domains and these are substantially non-overlapping with those domains spotted by the first method.

These two methods work together to give sophisticated and largely independent indicators of the likelihood that a candidate domain may be used to host phishing attacks against a known legitimate target. Using the overall risk rating produced by combining the two scores would presently detect 742 (33.9%) of the 2,191 currently blocked phishing domains.

Example Domains

The domains in the table below have run phishing attacks and are shown together with their domain registration risk.

DomainTargetRegistration Risk

The Interface

A web-based interface to the system is available for evaluation purposes and ad-hoc queries. For automated processes and bulk queries an API is available to return domain registration risk information in JSON format. Bespoke formats can be made available on request.

Entering the domain into the test system produces the report shown below:

More Information

Please get in touch ( if you would like to try out this service or for subscription information.