Skip to content
This repository has been archived by the owner on Dec 20, 2024. It is now read-only.
/ dune-sync Public archive

Components for syncing off-chain data with Dune Community Sources

Notifications You must be signed in to change notification settings

cowprotocol/dune-sync

Folders and files

NameName
Last commit message
Last commit date

Latest commit

47d19c1 · Apr 15, 2024
Mar 5, 2024
Jan 29, 2024
Apr 15, 2024
Apr 15, 2024
Aug 23, 2023
Nov 21, 2022
Nov 21, 2022
Jun 20, 2023
Mar 20, 2023
Feb 22, 2023
May 1, 2023
Jun 9, 2023
Jun 20, 2023

Repository files navigation

dune-sync

Components for syncing off-chain data with Dune Community Sources

Local Development

  1. Clone Repo git clone [email protected]:cowprotocol/dune-sync.git
  2. Several Makefile Commands make XXX Key make commands are; install, check, test

Docker

Build

docker build -t local_dune_sync .

You must provide valid environment variables as specified in .env.sample

Run Local

docker run -v ${PWD}/data:/app/data --env-file .env local_dune_sync

Run Remote

You will need to attach a volume and have an env file configuration. This example

  • mounts $PWD/data
  • and assumes .env file is in $PWD
docker run -v ${PWD}/data:/app/data --env-file .env ghcr.io/cowprotocol/dune-sync:latest

Breaking Changes

Whenever the schema changes, we must coordinate with Dune that the data must be dropped and the table rebuilt. For this we have provided a script scripts/empty_bucket.py which can be called to delete all data from their buckets and our backup volume. This should only be run whilst in coordination with their team about the changes. They will "stop the stream", drop the table on their side and restart the stream. In the event that a hard reset is performed without the proper coordination, it is likely that duplicate records will appear in their production environment (i.e. the interface).

So, the process is:

  • Contact a Dune Team Member (@dsalv)
  • Mention that we need to rebuild table XYZ (because of a schema change)
  • Once they are aware/prepared run
docker run -v ${PWD}/data:/app/data \
    --env-file .env \
    ghcr.io/cowprotocol/dune-sync \
    --sync-table SYNC_TABLE

This will empty the buckets and repopulate with the appropriate changes.