Manufacturing.gov and White House security suffer under U.S. shutdown

Dozens more U.S. government websites have become inaccessible since last week, when Netcraft highlighted the impact of security certificates expiring during the federal shutdown.

As of today, more than 130 TLS certificates used by U.S. government websites have expired without being renewed. Some of these sites are now completely inaccessible in modern browsers due to their strict transport security policies.

The latest sites to be affected include some particularly prominent examples.

manufacturing-snippet

Take https://manufacturing.gov, for instance. While Trump is keen to highlight the performance of U.S. manufacturing during his administration, the shutdown has meant that nobody was available to renew the site's TLS certificate when it expired on 14 January 2019. Consequently, https://manufacturing.gov is dead in the water, along with https://manufacturingusa.com which shares the same certificate.

manufacturing

Furthermore, as https://manufacturing.gov appears in Chromium's HSTS preload list, visitors are unable to bypass the browser's security warnings, rendering the site unreachable.

manufacturing.gov appears in Chromium's HSTS preload list, which ensures that the website's strict transport policy will always be enforced, even when a browser has never visited the site before.

manufacturing.gov appears in Chromium's HSTS preload list, which ensures that the website's strict transport policy will always be enforced, even when a browser has never visited the site before. www.manufacturing.gov uses a different certificate, which is currently valid.

A White House subdomain at https://pages.mail.whitehouse.gov has also become unreachable. The certificate used by this site expired on 15 January 2019 and has not been renewed. This site is also covered by an effective preloaded HSTS policy.

White House security warnings in Mozilla Firefox.

White House security warnings in Mozilla Firefox.

Other notable websites to have been affected by expired certificates over the past five days include two FAA (Federal Aviation Authority) websites, a National Archives customer portal, the FFIEC (Federal Financial Institutions Examination Council) Anti-Money Laundering Infobase, several Department of Agriculture sites, and several governmental remote access services.

When the federal government restarts, the White House will need to renew its certificate for pages.mail.whitehouse.gov. The list price for a replacement DigiCert organisation validated certificate — similar to the expired one — could be up to $399 per year, or about 70 Big Macs.

.gov security falters during U.S. shutdown

Dozens of U.S. government websites have been rendered either insecure or inaccessible during the ongoing U.S. federal shutdown. These sites include sensitive government payment portals and remote access services, affecting the likes of NASA, the U.S. Department of Justice, and the Court of Appeals.

The DigiCert certificate used by this U.S. Court of Appeals website expired on 5 January 2019 and has not yet been renewed. The site provides links to a document filing system and PACER (Public Access to Court Electronic Records).

The DigiCert certificate used by this U.S. Court of Appeals website expired on 5 January 2019 and has not yet been renewed. The site provides links to a document filing system and PACER (Public Access to Court Electronic Records).

With around 400,000 federal employees currently furloughed, more than 80 TLS certificates used by .gov websites have so far expired without being renewed. To compound the situation, some of these abandoned websites can no longer be accessed due to strict security measures that were implemented long before the shutdown started.

One such example is https://ows2.usdoj.gov, a U.S. Department of Justice website which uses a certificate that expired in the week leading up the shutdown. The certificate has been signed by a trusted certificate authority, GoDaddy, but it has not been renewed since it expired on 17 December 2018.

All U.S. Department of Justice subdomains are covered by an HSTS policy. Combined with an expired TLS certificate, this currently makes it difficult for regular users to ignore the warnings and use the website.

All U.S. Department of Justice subdomains are covered by an HSTS policy. Combined with an expired TLS certificate, this currently makes it difficult for regular users to ignore the warnings and use the website.

In a twist of fate, the usdoj.gov domain — and all of its subdomains — are included in Chromium's HSTS preload list. This is a prudent security measure which forces modern browsers to only use secure, encrypted protocols when accessing the U.S. DoJ websites; however, it will also prevent users from visiting the HTTPS sites when an expired certificate is encountered. In these cases, modern browsers like Google Chrome and Mozilla Firefox deliberately hide the advanced option that would let the user bypass the warning and continue through to the site.

While this behaviour is bound to frustrate some users, in this case, security is arguably better than usability when you can't have both. If users were to ignore such warnings, they would be vulnerable to the type of man-in-the-middle attacks that TLS certificates were intended to combat.

However, only a few of the affected .gov sites implement correctly-functioning HSTS policies. Just a handful of the sites appear in the HSTS preload list, and only a small proportion of the rest attempt to set a policy via the Strict-Transport-Security HTTP header – but the latter policies will not be obeyed when they are served alongside an expired certificate, and so will only be effective if the user has already visited the sites before.

Consequently, most of the affected sites will display an interstitial security warning that the user will be able to bypass. This introduces some realistic security concerns, as task-oriented users are more likely to ignore these security warnings, and will therefore render themselves vulnerable to man-in-the-middle attacks.

For example, https://rockettest.nasa.gov/ is not included in the HSTS preload list, and its certificate expired on 5 January 2019. This causes browsers to display an interstitial security warning that users can ignore.

This NASA website is still using an expired certificate, but the domain does not appear on the HSTS preload list.  Users can therefore ignore the browser's warnings and proceed to the site.

This NASA website is still using an expired certificate, but the domain does not appear on the HSTS preload list. Users can therefore ignore the browser's warnings and proceed to the site.

The following example clearly demonstrates the potential dangers of ignoring browser security warnings. The certificate used by this Berkeley Lab .gov website at https://d2l.lbl.gov expired on 8 January 2019 (although Berkeley Lab was not affected by the shutdown) and has not yet been replaced. As there is no effective HSTS policy, users can ignore the browser's warnings and proceed to the login form.

Encouraging users to ignore browser warnings could make them more susceptible to man-in-the-middle attacks.

Encouraging users to ignore browser warnings could make them more susceptible to man-in-the-middle attacks. In this example, clicking next to the browser's address bar will explicitly advise the user not to enter any sensitive information, such as passwords – but anyone who really needs to use the site may foolishly end up doing so anyway.

With Donald Trump seemingly unwilling to compromise on his demands for a wall along the border with Mexico, and Democrats refusing to approve a budget containing $5.7bn for the wall, the hundreds of thousands of unpaid federal employees might not be the only ones hurting. As more and more certificates used by government websites inevitably expire over the following days, weeks — or maybe even months — there could be some realistic opportunities to undermine the security of all U.S. citizens.

Most Reliable Hosting Company Sites in December 2018

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 EveryCity SmartOS 0:00:00 0.000 0.230 0.070 0.338 0.338
2 Bigstep Linux 0:00:00 0.000 0.233 0.071 0.146 0.146
3 Webair Linux 0:00:00 0.000 0.335 0.080 0.161 0.161
4 Hyve Managed Hosting Linux 0:00:00 0.000 0.170 0.081 0.162 0.162
5 CWCS Linux 0:00:00 0.000 0.293 0.083 0.160 0.160
6 Rackspace Linux 0:00:00 0.004 0.663 0.006 0.015 0.015
7 New York Internet (NYI) FreeBSD 0:00:00 0.004 0.540 0.063 0.125 0.125
8 www.dinahosting.com Linux 0:00:00 0.004 0.280 0.089 0.178 0.178
9 krystal.co.uk Linux 0:00:00 0.004 0.241 0.100 0.198 0.198
10 GoDaddy.com Inc Linux 0:00:00 0.008 0.433 0.007 0.021 0.022

See full table

EveryCity had the most reliable hosting company site in December 2018, with no failed requests and an average connection time of 70ms. EveryCity appeared eight times in the top 10 in 2018, and has maintained 99.9994% uptime over the last four years. The hosting company offers cloud hosting solutions and managed third-party services, with its primary data centre located in the heart of the City of London.

Four other hosting company sites also successfully responded to all of Netcraft's requests, and were ranked by their average connection times. In second place, Bigstep had an average connection time just a millisecond slower than EveryCity. Bigstep has maintained 99.97% uptime over the past five years. Webair, Hyve and CWCS complete the top five, with average connection times of 80ms, 81ms and 83ms.

Linux was the most popular operating system in December 2018, used by eight of the top 10 most reliable hosting company sites. SmartOS is used by EveryCity, which had the most reliable hosting company site. SmartOS is an open-source community fork of OpenSolaris designed specifically for cloud computing. FreeBSD also appears in the top 10 for December.

Netcraft measures and makes available the response times of around twenty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

December 2018 Web Server Survey

In the December 2018 survey we received responses from 1,656,556,205 sites, 227,839,033 unique domains, and 8,147,795 web-facing computers. This reflects a gain of 4.37 million sites, 1.09 million domains, and 98.9k web-facing computers.

Whilst Apache's server software remains the market leader in most metrics, its share continues to erode in favour of competing products. In December 2018, Apache's flagship server product, httpd, saw a net decrease of 938k domains (-1.27% from last month), 28.6 million sites (-8.43%), and 848k active sites (-1.49%). Whilst another Apache product, known as Coyote, grew by 151k domains this month (+32.6%), the majority of the increase is attributable to parking pages, rather than sites with unique content. Apache's share of the top million sites has been falling steadily over the past few years, and is now down to 33.04%. Although it still holds the largest share, its margin is rapidly shrinking, and its dominance will likely be challenged by nginx in a year's time. Despite these losses, the number of web-facing computers running Apache software does, however, continue to grow, with an increase of 22.8k (+0.73%) this month.

nginx showed the fastest growth, having increased its domain count by 624k (+1.21%) this month. It also saw a 247k increase in active sites (+0.60%) — whereas Microsoft and Apache both saw losses — and has consistently seen large increases in its web-facing computer count. nginx has now reached 2.28 million web-facing computers, a significant 39.9% growth over the past 12 months compared to Apache's 4.98% and Microsoft's 1.93% over the same period. nginx saw two new releases in during November, fixing three CVEs, as well as introducing a few minor features and bug fixes.

Microsoft's server software holds the greatest market share when it comes to total number of sites. With over 688 million, it now has more sites than nginx (358 million) and Apache (314 million) combined, representing 41.53% of the market. This month Microsoft experienced an increase of 35.9 million sites and 246k domains. Much of the domain increase was seen at Amazon (+146k) and GoDaddy (+189k). Microsoft presently powers 9.33% of the world's top one million websites. Microsoft's top million share has seen some fluctuation over the year, with a slight decrease of 0.13pp this month, but only 0.30pp over 12 months.

Some of the large increase in overall domains this month can be seen at Cloudflare, which predominantly uses its own server software originally based on nginx. 8.20 million domains are currently served from servers identifying themselves as Cloudflare — 373k more than last month. At Google, usage of GSE (Google Servlet Engine) dropped massively from 897k domains down to 228k in favour of another google server, GHS, which is often used purely for redirects.

Total number of websites

Web server market share

DeveloperNovember 2018PercentDecember 2018PercentChange
Microsoft652,135,93639.47%688,039,05641.53%2.06
nginx358,695,27621.71%358,383,16921.63%-0.08
Apache341,675,49920.68%313,736,73918.94%-1.74
Google24,094,4311.46%23,810,3361.44%-0.02
Continue reading

Most Reliable Hosting Company Sites in November 2018

Rank Performance Graph OS Outage
hh:mm:ss
Failed
Req%
DNS Connect First
byte
Total
1 GoDaddy.com Inc Linux 0:00:00 0.000 0.448 0.006 0.020 0.021
2 EveryCity SmartOS 0:00:00 0.000 0.237 0.074 0.352 0.352
3 Hyve Managed Hosting Linux 0:00:00 0.000 0.179 0.076 0.153 0.153
4 Webair Linux 0:00:00 0.000 0.348 0.080 0.161 0.161
5 New York Internet (NYI) FreeBSD 0:00:00 0.005 0.554 0.064 0.128 0.128
6 Bigstep Linux 0:00:00 0.005 0.242 0.073 0.148 0.148
7 Swishmail FreeBSD 0:00:00 0.005 0.245 0.085 0.170 0.172
8 vXtream Ltd Linux 0:00:00 0.009 0.363 0.075 0.151 0.151
9 Netcetera Linux 0:00:00 0.009 0.206 0.088 0.180 0.180
10 ServerStack Linux 0:00:00 0.009 0.244 0.091 0.181 0.181

See full table

In November 2018, GoDaddy had the most reliable hosting company site, with zero failed requests and the fastest average connection time of 6ms. This is the second consecutive month and the fourth time in 2018 that GoDaddy has taken the top spot. GoDaddy provides a wide range of hosting and domain registration services, including its widely used website builder software, which provides customers with a simple way to create a web presence.

Also achieving zero failed requests this month were EveryCity, Hyve Managed Hosting and Webair, with the tie between them broken by their average connection times of 74ms, 76ms and 80ms respectively. EveryCity offers its own cloud hosting solutions, as well as managed third-party cloud hosting services, including AWS and Azure. Hyve offers fully managed colocation services in 35 worldwide locations and has appeared in the top ten for 20 of the past 24 months.

Linux is the most popular choice of operating system in the top 10 this month, used by seven of the ten sites. Among the remaining sites, FreeBSD appears twice and SmartOS once.

Netcraft measures and makes available the response times of around thirty leading hosting providers' sites. The performance measurements are made at fifteen minute intervals from separate points around the internet, and averages are calculated over the immediately preceding 24 hour period.

From a customer's point of view, the percentage of failed requests is more pertinent than outages on hosting companies' own sites, as this gives a pointer to reliability of routing, and this is why we choose to rank our table by fewest failed requests, rather than shortest periods of outage. In the event the number of failed requests are equal then sites are ranked by average connection times.

Information on the measurement process and current measurements is available.

November 2018 Web Server Survey

In the November 2018 survey we received responses from 1,652,185,816 sites, 226,752,928 unique domains, and 8,048,899 web-facing computers. This reflects a loss of 21.3 million sites, but a gain of 1.7 million domains and 121 thousand web-facing computers.

The largest contributor to the increase in domains this month came from sites running on Microsoft server software, with an additional 1.18 million domains. The growth was concentrated at GoDaddy on domains running IIS 7.5.x, which took up 1.16 million of the increase. In addition to newly detected IIS domains, and a movement of some domains from nginx, many existing domains running previously unknown IIS versions also reported as IIS 7.5.x this month, resulting in the total number of domains for this version almost doubling to reach 12.7 million. This makes IIS 7.5.x the most common version seen by the survey, 2.7 million domains ahead of the newer 8.5.x version in second place, which first saw release in 2013. Despite the large gain in domains, Microsoft experienced losses in most other metrics, including a significant loss of 3.14 million active sites. Many of the domains gained at GoDaddy are likely to be holding pages, contributing only a small increase to the number of IIS active sites hosted at the company.

nginx experienced the largest loss in domains this month of 560 thousand, largely caused by a drop of just over 1 million domains at GoDaddy. Nginx has been experiencing a steady rise in market share by domains for many years, however this has now remained relatively static for the last 6-months at 23%. nginx remains just under 3 percentage points behind Microsoft’s 26% market share. Meanwhile though, nginx has continued to show strong growth in web-facing computers, with the largest increase this month of 51 thousand. Here its steady market share increases have shown no such pause, and it holds almost 28% of the market. Apache and Microsoft followed some way behind in web-facing computer gains, with increases of 13 thousand and 12 thousand, both experiencing small losses in market share as a result.

Total number of websites

Web server market share

DeveloperOctober 2018PercentNovember 2018PercentChange
Microsoft656,395,38839.22%652,135,93639.47%0.25
nginx330,074,97419.72%358,695,27621.71%1.99
Apache384,514,94422.98%341,675,49920.68%-2.30
Google23,620,5551.41%24,094,4311.46%0.05
Continue reading