Netcraft to conform to new Internet Security RFC

Since we started the Web Server Survey in 1995, a longstanding theme of Netcraft's internet exploration work has been the issue of how best to reassure webmasters and systems administrators that requests they may see originating from Netcraft's network are benign, and do not in any way convey aggressive intent.

Earlier today an RFC was published by Internet pioneer Steve Bellovin which addresses this scenario. Bellovin's idea is that the sender's intentions, whether good or bad, should be stated directly in the TCP header information using a security flag [termed the "evil bit" by Bellovin]. It is intended that network protection devices such as routers, firewalls and Intrusion Detection Systems should defend their networks against packets where the evil bit is set, but otherwise assume that traffic is benign. Groups aligning themselves with RFC 3514 include the FreeBSD project, [who have already coded an implementation] and the nmap scanner.

Supporting RFC 3514 is straightforward for Netcraft, as we have uniformly good intentions and can leave the evil bit unset in all circumstances. However, Nmap has a very diverse user community, and is likely to adopt a context-sensitive approach where the security flag is derived from the organisation and geographical location of the source address.

Network protection companies may be more reticent to support RFC 3514 as the security flag can easily be spoofed. Technically, they could adopt a solution where the evil bit is not taken at face value, but further qualified using an approach similar to one likely to be implemented in nmap. However, this has the potential to become very contentious, as there will be no easy way to determine how "good" is distiniguished from "evil" is without direct access to the source code [which is available in nmap's case].

In particular, it will be hard for network protection companies, many of whom are domiciled in the US and Israel, to convince international customers that the algorithm to determine whether to allow traffic through is not influenced by their respective governments, or in some way derived from the anticipated political or religious affiliation of the organisation that the device's ip address is registered to.

For example, one could envisage a scenario where US developed RFC 3514 compliant firewalls deployed on ip addresses registered to organisations outside the United States might allow through all traffic originating from US government agencies, believing it to be "good". Symmetrically, there is the potential for such devices to covertly set the "evil" bit on outbound traffic from governments not closely aligned with the United States.

Consequently RFC 3514 is likely to generate controversy as soon as the mainstream media picks up on it. Human rights, religious and political groups, government agencies, and anyone who thinks their network traffic might be classified as "evil" may have especially strong opinions on the RFC. Conversely, open source advocates will point out that providing people restrict themselves to using only products for which they have the source, there is no inherent problem with the new RFC itself, and that the potential for underhand behaviour by closed source companies should not be allowed to hinder the widespread adoption of a well intentioned proposal.