spellbook

Welcome to your Spellbook. Cast a magical incantation to tame the blockchain.

Dune contributors using the project

Use the pipfile to create a pipenv.

pipenv install

If the env is created successfully, enter the env shell. (If you run into Python version errors, try installing the matching version or editing the python version in the Pipfile but don't commit your change).

pipenv shell

Set up the dbt spellbook project. (You must run this step from the spellbook directory) Run dbt init and select Databricks, then enter . or other placeholders for the host, HTTP path, and token. This will not connect to the database but you have access to some dbt actions.

When you are prompted to choose a target, please enter wizard so we know you are an external contributor.

dbt init

Then, run the following commands:

cd spellbook/
dbt compile

dbt compile will compile the JINJA and SQL templated SQL into plain SQL which can be executed in the Dune UI. We are thinking about better solutions to make more dbt actions available directly but also have to consider security.

setup.mov

How to use dbt to create abstractions

There's a couple new concepts to consider when making abstractions in dbt. The most common ones wizards will encounter are refs, sources, freshness, and tests.

In the body of each query, tables are referred to either as refs, ex {{ ref('1inch_ethereum') }} or sources, ex {{ source('ethereum', 'traces') }}. Refs refer to other dbt models and they should refer to the file name like 1inch_ethereum.sql, even if the model itself is aliased. Sources refer to "raw" data or tables/views not generated by dbt. Using refs and sources allows us to automatically build dependency trees.

Sources and models are defined in schema.yml files where tests and other attributes are defined.

Best practice is to add tests unique and non_null tests to the primary key for every new model. Similarly, a freshness check should be added to every new source (although we will try not to re-test freshness if the source is used elsewhere).

Adding descriptions to tables and columns will help people find and use your tables.

models:
  - name: 1inch_ethereum
    description: "Trades on 1inch, a DEX aggregator"
    columns:
      - name: tx_hash
        description: "Table primary key: a transaction hash (tx_hash) is a unique identifier for a transaction."
        tests:
          - unique
          - not_null
 
  sources:
  - name: ethereum
    freshness:
      warn_after: { count: 12, period: hour }
      error_after: { count: 24, period: hour }
    tables:
      - name: traces
        loaded_at_field: block_time

See links to more docs on dbt below.

Generating and serving documentation:

To generate documentation and view it as a website, run the following commands:

• dbt docs generate
• dbt docs serve You must have set up dbt with dbt init but you don't need database credentials to run these commands.

See dbt docs documentation for more information on how to contribute to documentation.

As a preview, you can do things like:

• Write simple one or many line descriptions of models or columns.
• Write longer descriptions as code blocks using markdown.
• Link to other models in your descriptions.
• Add images / project logos from the repo into descriptions.
• Use HTML in your description.

Troubleshooting

If you fail to run dbt compile with Could not find profile named 'spellbook' as the error message, check ~/.dbt/profiles.yml and make sure there is a profile named spellbook. When you run dbt init to initiate a project, a profile gets created. Inside spellbook you cannot initiate a project called the same name, so you need to run dbt init spellbook outside the project so it creates the profile, or create one with a different name and then manually edit the profiles.yml file.

DBT Resources:

• Learn more about dbt in the docs
• Check out Discourse for commonly asked questions and answers
• Join the chat on Slack for live discussions and support
• Find dbt events near you
• Check out the blog for the latest news on dbt's development and best practices