scrub

Glassnode API scrapping utility

setup local development:

• dev deps

pip3 install virtualenv wheel setuptools twine

• clone repo
• go to working dir

cd scrub

• setup pyton venv

virtualenv venv

• initialise venv

source venv/bin/activate

• install deps & build dev version in venv

python setup.py develop

• test run cli command

scrub

glassnode api key

put api key into envvar

export GLASSNODE_API_KEY=<your_api_key>

scrub commands

> scrub --help

Usage: scrub [OPTIONS] COMMAND [ARGS]...

Options:
  -g, --apikey TEXT  glassnode api key  [required]
  -o, --outdir TEXT  folder where data is dumped  [default: ./data; required]
  --help             Show this message and exit.

Commands:
  endpoints
  scrape

scrub endpoints

Generates endpoints to be fetched by scrub scrape

> scrub endpoints --help

Usage: scrub endpoints [OPTIONS]

Options:
  -a, --assets TEXT    digital asset code: BTC, ETH, ...  [default: *; required]
  -t, --tiers INTEGER  tier: 1, 2, 3  [default: 0; required]
  -p, --path TEXT      filter by path specified as regular expression [default: *; required]
  --help               Show this message and exit.

scrub scrape

fetch data specified in data/endpoints.json and store them into csv files into data folder

> scrub scrape --help

Usage: scrub scrape [OPTIONS]

Options:
  -n, --num_workers INTEGER  how many scrapping workers  [default: 18; required]
  -b, --batch_size INTEGER   size of scraping batch  [default: 8; required]
  --help                     Show this message and exit.