Skip to content

electroneum/discv4-crawl

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

82 Commits
 
 
 
 
 
 
 
 

Repository files navigation

discv4-crawl

Background

Etn-sc ships with an implementation of EIP-1459. This EIP defines a way to put devp2p node lists behind a DNS name. There are a couple of things worth knowing about this system:

EIP-1459 is intended to be a replacement for hard-coded bootstrap node lists that we maintain in Electroneum-SC clients.
This is a centralized system where all nodes configured with a certain name resolve subdomains of the name find bootstrap nodes.
The node list is signed with a key which will be hard-coded into the client (i.e. etn-sc) and which we should keep in a secure place.

To create suitable bootstrap node lists for all common networks, we have devised a scheme where software crawls the discovery DHT, then creates a list of all found nodes in JSON format. The crawler software can filter this list and has a built-in deployer that can install the DNS records.

You can read the DNS Discovery Setup Guide for more information about the discovery DHT crawler.

Description

This repository contains the scripts used to automatically generate the list of nodes that are published to the multiple DNS zones. The node list is also automatically pushed to this repository.

Running with docker

Environment variables

Name Default Description
CRAWL_GIT_REPO https://github.com/electroneum/discv4-dns-lists.git Git repository `used to clone and push the node list
CRAWL_GIT_BRANCH master Git branch used for the fetch and push
CRAWL_GIT_PUSH false When set to true, it will push the node lists to the git repository
CRAWL_GIT_USER crawler Git username. Will appear in the commit messages.
CRAWL_GIT_EMAIL crawler@localhost Git email address. Will appear in the `commit messages.
CRAWL_DNS_DOMAIN nodes.example.local DNS domain suffix used for the directory structure
CRAWL_TIMEOUT 30m The time spent crawling the discovery DHT
CRAWL_INTERVAL 300 Interval, in seconds, between multiple executions.
CRAWL_RUN_ONCE false Set to true if you only want to run the execution once.
CRAWL_DNS_SIGNING_KEY /secrets/key.json Path to the signing key. Won't sign if the file doesn't exist.
CRAWL_DNS_PUBLISH_ROUTE53 false Publish the TXT records to a DNS zone on AWS Route53
ROUTE53_ZONE_ID `` Route53 DNS zone identifier. This is the zone where the records will be published to.
AWS_ACCESS_KEY_ID `` AWS access key
AWS_SECRET_ACCESS_KEY `` AWS secret access key
CRAWL_DNS_PUBLISH_CLOUDFLARE false Publish the TXT records to a DNS zone on Cloudflare
CLOUDFLARE_API_TOKEN `` API token used for the Cloudflare API
CLOUDFLARE_ZONE_ID `` Cloudflare DNS zone identifier. This is the zone where the records will be published to.
INFLUXDB_METRICS_ENABLED false Set to true if you want to send metrics to InfluxDB
INFLUXDB_URL http://localhost:8086 Address of the InfluxDB API
INFLUXDB_DB metrics Database name
INFLUXDB_USER user Username for InfluxDB
INFLUXDB_PASSWORD password Password for InfluxDB
PROMETHEUS_METRICS_ENABLED true Enable prometheus metrics endpoint
PROMETHEUS_METRICS_LISTEN 0.0.0.0:9100 Server listening

Building the image

$ docker build -t discv4-crawl .

Run examples

Run the list generation and push the results to git via SSH:

$ docker run -it \
    -v "$HOME/.ssh/crawler:/root/.ssh" \  # Needed if you use git via SSH
    -v "$HOME/secrets/secret-signing-key.json:/secrets/key.json" \ # Only needed if you want to sign the node lists
    -e CRAWL_TIMEOUT=10m \ # Specify your custom timeout
    -e [email protected]:electroneum/discv4-dns-lists.git \ # Use SSH instead of HTTPS
    -e CRAWL_GIT_PUSH=true \ # Specify that we want to push the changes
    electroneum/discv4-crawl

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Shell 84.8%
  • Dockerfile 15.2%