Skip to content

Latest commit

 

History

History
164 lines (118 loc) · 4.33 KB

README.md

File metadata and controls

164 lines (118 loc) · 4.33 KB
Raw

tap-snowflake

tap-snowflake is a Singer tap for Snowflake.

Built with the Meltano Tap SDK for Singer Taps.

Installation

pipx install git+https://github.com/MeltanoLabs/tap-snowflake.git

Configuration

Accepted Config Options

A full list of supported settings and capabilities for this tap is available by running:

tap-snowflake --about

tables configuration parameter.

An array of the table names that you want to sync. The table names should be fully qualified, including schema and table name.

NOTES:

  • This limits discovery to the tables specified for performance reasons. Do not specify tables if you intend to discover the entire available catalog.
  • Syntax to specify tables differs slightly from select (schema_name.table_name vs schema_name-table_name.*)

💡 When adding more elements to select ensure the table is specified in tables if using tables.

Example:

  ...
  tables:
    - schema_name1.table1
    - schema_name1.table2

Configure using environment variables

This Singer tap will automatically import any environment variables within the working directory's .env if the --config=ENV is provided, such that config values will be considered if a matching environment variable is set either in the terminal context or in the .env file.

Source Authentication and Authorization

This tap supports two authentication methods:

Key Pair Authentication

Key Pair authentication can be enabled by specifing either the private_key of the private_key_path, and optionally the private_key_passphrase if required.

Enabling Batch Messaging

This tap is built using the Meltano SDK and therefore supports a BATCH message type, in addition to the RECORD messages of the Singer spec. This can be enabled either by adding the following to your config.json:

{
  // ...
  "batch_config": {
    "encoding": {
      "format": "jsonl",
      "compression": "gzip"
    },
    "storage": {
      "root": "file://tests/core/resources",
      "prefix": "test-batch"
    }
  }
}

or its equivalent to your meltano.yml

config:
  plugins:
    extractors:
      - name: tap-snowflake
        config:
          batch_config:
            encoding:
              format: jsonl
              compression: gzip
            storage:
              root: "file://tests/core/resources"
              prefix: test-batch

Note: This variant of tap-snowflake does not yet support the INCREMENTAL replication strategy in BATCH mode. Follow here for updates.

Usage

You can easily run tap-snowflake by itself or in a pipeline using Meltano.

Executing the Tap Directly

tap-snowflake --version
tap-snowflake --help
tap-snowflake --config CONFIG --discover > ./catalog.json

Developer Resources

Initialize your Development Environment

pipx install poetry
poetry install

Create and Run Tests

Create tests within the tap_snowflake/tests subfolder and then run:

poetry run pytest

You can also test the tap-snowflake CLI interface directly using poetry run:

poetry run tap-snowflake --help

Testing with Meltano

Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.

Your project comes with a custom meltano.yml project file already created. Open the meltano.yml and follow any "TODO" items listed in the file.

Next, install Meltano (if you haven't already) and any needed plugins:

# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-snowflake
meltano install

Now you can test and orchestrate using Meltano:

# Test invocation:
meltano invoke tap-snowflake --version
# OR run a test `elt` pipeline:
meltano elt tap-snowflake target-jsonl

SDK Dev Guide

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.