ssdt - Survey security.txt files

A program to quickly survey security.txt files found on the Alexa Top 1 Million websites. The program takes about 15 hours to run over a 1.5Mbit residential DSL connection. It could go much faster over high-speed links.

$ ps -p 165199 -o etime
  ELAPSED
  15:06:42

Sample output

{"website" ["contacts"] "expires"}
{"github.com" ["https://hackerone.com/github"] ""}
{"google.com" ["https://g.co/vulnz" "mailto:[email protected]"] ""}
{"facebook.com" ["https://www.facebook.com/whitehat/report/"] ""}
{"linkedin.com" ["mailto:[email protected]" "https://www.linkedin.com/help/linkedin/answer/62924"] ""}
{"cloudflare.com" ["https://hackerone.com/cloudflare" "mailto:[email protected]" "https://www.cloudflare.com/abuse/"] "sat, 20 mar 2021 13:24:05 -0700"}

Build the program

$ make

Run the program

$ ./ssdt -hosts top-1m-alexa.csv 2> err.txt > out.txt

Remove invalid security.txt entries

$ grep -v "\[\]" out.txt

Count results

$ grep -v "\[\]" out.txt | wc -l

Notes

• You may need to adjust the nofile limit in /etc/security/limits.conf before running ssdt. Otherwise, you may exceed the open file limit.
• Read my blog post about why I wrote this program.