A survey of security.txt

I was curious to know how many popular websites have already uploaded a security.txt file and how many followed the most recent draft. So, I wrote a small Go program to check the Alexa Top 1 million websites.

What is security.txt

The security.txt file is intended to make reporting security vulnerabilities to organizations straight-forward. On web servers, the file must be a text file located in the /.well-known/ directory that describes, at a minimum:

  1. How to contact the organization.
  2. Have an expiration date.

The file may also contain comments, links to responsible disclosure policies, PGP keys and acknowledgments to security researchers.

Here’s the security.txt file I currently use:

# ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
# ░░░░     ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
# ▒▒  ▒▒▒▒   ▒▒▒▒▒▒▒▒▒▒▒▒▒▒     ▒▒▒▒   ▒    ▒▒▒▒▒▒   ▒▒▒▒▒
# ▒  ▒▒▒▒▒▒▒▒▒▒▒▒▒▒   ▒▒▒▒▒▒▒▒   ▒▒▒   ▒▒▒▒▒▒▒▒▒   ▒▒   ▒▒
# ▓   ▓▓▓▓▓▓▓▓▓▓▓   ▓▓   ▓▓▓   ▓▓▓▓▓    ▓▓▓▓▓▓   ▓▓▓▓▓   ▓
# ▓   ▓▓▓      ▓   ▓▓▓▓   ▓▓▓▓▓   ▓▓▓▓▓▓   ▓▓▓   ▓▓▓▓▓▓   
# ▓▓   ▓▓▓▓  ▓▓▓▓   ▓▓   ▓▓▓▓▓▓▓▓  ▓▓▓▓▓▓▓   ▓▓   ▓▓▓▓   ▓
# ███      ████████   █████      ███    █   ██████    ████
# ████████████████████████████████████████████████████████
# SECURITY
#
Contact: mailto:security@go350.com
Acknowledgments: https://www.go350.com/thanks/
Canonical: https://www.go350.com/.well-known/security.txt
Preferred-Languages: en
Expires: Fri, 31 Dec 2021 23:59:59 -0500

Prior research

Some prior research had already checked for security.txt files on the first 1,000 most popular Alexa websites. The author wrote, “I decided to fetch data only for the Top 1000 as the Top 1M would have required days and days”.

I thought I could survey the whole list and share my code and results.

My approach

Using Go routines, I estimated that I could check 720 websites for security.txt files every minute (from my 1.5Mbit residential DSL line). That’s 12 websites per second.

There are a total of 666,771 websites on the Alexa list. I’m not sure why they say 1 million. Anyway, I should be able to check them all in about 15 hours. Less than a day.

$ wc -l top-1m-alexa.csv
666771 top-1m-alexa.csv

The survey took just over 15 hours to complete. I forgot to use time, so I ran ps in a loop and this was the last reported elapsed time before the program ended.

$ ps -p 165199 -o etime
  ELAPSED
  15:06:42

I store the survey results in a simple struct.

type Sdt struct {
	site     string   // "google.com"
	contacts []string // ["https://g.co/vulnz", "mailto:security@google.com"]
	expires  string   // "Thu, 31 Dec 2020 18:37:07 -0800"
}

Here are a few example results.

{"website" ["contacts"] "expires"}
{"github.com" ["https://hackerone.com/github"] ""}
{"google.com" ["https://g.co/vulnz" "mailto:security@google.com"] ""}
{"facebook.com" ["https://www.facebook.com/whitehat/report/"] ""}
{"linkedin.com" ["mailto:security@linkedin.com" "https://www.linkedin.com/help/linkedin/answer/62924"] ""}
{"cloudflare.com" ["https://hackerone.com/cloudflare" "mailto:security@cloudflare.com" "https://www.cloudflare.com/abuse/"] "sat, 20 mar 2021 13:24:05 -0700"}

The survey results

Of the 666,771 most popular websites on the Alexa list, I found 2,884 security.txt files that were content-type “text/plain” and returned a HTTP 200 status code. Not all of these were valid security.txt files, but most were.

Sixty of the files were PGP signed. Twenty seven had the ‘Expires:’ field that the latest draft requires. Some had comments indicating that they followed an older version of the draft:

$ grep draft err.txt
# Conforms to IETF 'draft-foudil-securitytxt-07'
# draft-foudil-securitytxt-06 says this file should be in .well-known

Some were empty files. Some returned my IP address. Some were unrelated HTML or JSON. Some contained a single word such as ‘ok’. One had only a Unix epoch time.

A few contained multiple paragraphs of text describing the organization’s security policies and ground rules for bug bounty programs. Some of those were formatted as comments while others were not.

Some, such as DreamHost, had ASCII art with a sense of humor:

ascii-art

Here are the full survey results and the code. Maybe others can check behind me to see if I made mistakes. I probably did.