Skip to content

Latest commit

 

History

History
79 lines (66 loc) · 1.64 KB

README.md

File metadata and controls

79 lines (66 loc) · 1.64 KB
Raw

scrub

Glassnode API scrapping utility

setup local development:

  • dev deps
pip3 install virtualenv wheel setuptools twine
  • clone repo
  • go to working dir
cd scrub
  • setup pyton venv
virtualenv venv
  • initialise venv
source venv/bin/activate
  • install deps & build dev version in venv
python setup.py develop
  • test run cli command
scrub

glassnode api key

put api key into envvar

export GLASSNODE_API_KEY=<your_api_key>

scrub commands

> scrub --help

Usage: scrub [OPTIONS] COMMAND [ARGS]...

Options:
  -g, --apikey TEXT  glassnode api key  [required]
  -o, --outdir TEXT  folder where data is dumped  [default: ./data; required]
  --help             Show this message and exit.

Commands:
  endpoints
  scrape

scrub endpoints

Generates endpoints to be fetched by scrub scrape

> scrub endpoints --help

Usage: scrub endpoints [OPTIONS]

Options:
  -a, --assets TEXT    digital asset code: BTC, ETH, ...  [default: *; required]
  -t, --tiers INTEGER  tier: 1, 2, 3  [default: 0; required]
  -p, --path TEXT      filter by path specified as regular expression [default: *; required]
  --help               Show this message and exit.

scrub scrape

fetch data specified in data/endpoints.json and store them into csv files into data folder

> scrub scrape --help

Usage: scrub scrape [OPTIONS]

Options:
  -n, --num_workers INTEGER  how many scrapping workers  [default: 18; required]
  -b, --batch_size INTEGER   size of scraping batch  [default: 8; required]
  --help                     Show this message and exit.