0.2.0a32

What's Changed in Docs

• moving to new docs structure by @TyDunn in #245
• adds Agolia DocSearch to the dlt docs 🚀 by @TyDunn in #248
• Zendesk pipeline docs by @AmanGuptAnalytics in #222
• Added Hubspot setup guide by @AmanGuptAnalytics in #250
• moving create a pipeline to use weatherapi and duckdb by @TyDunn in #255
• first version of exploring the data docs page by @TyDunn in #257
• adds schema general usage and schema adjusting walkthrough to docs by @rudolfix in #243
• filling in deploying section by @TyDunn in #262
• Examples for customisations by @adrianbr in #247

What's Changed

• Typed pipeline state by @steinitzu in #239
• allows incremental to be passed to resource.apply_hints() method
• adds state property to sources and resources to get actual value of source and resource scoped state
• Fix failing tests for Redshift and PostgreSQL by @burnash in #270
• add resource name to table schema by @steinitzu in #265
• resets the resource scoped state when doing replace on resource
• you can add Incremental as a transform step, instead of injecting

Full Changelog: 0.2.0a30...0.2.0a32

0.2.0a30

What's Changed

This release includes two important features

• merge write disposition: load data incrementally by merging with merge keys and/or deduplicate/upsert with primary keys
• incremental loading with last value and dlt state available when declaring resources
  We consider those features still in alpha. Try them out and report bugs! Preliminary documentation is here: https://dlthub.com/docs/customization/incremental-loading

This release includes improved support for resources that use dynamic hint to dispatch data to several database tables and other bug fixes.

What's Changed in docs

• Strapi setup guide by @TyDunn in #212
• add edit this page button on all docs pages by @TyDunn in #226
• adding alerting content from workshop by @TyDunn in #233
• adding monitoring content from workshop by @TyDunn in #229
• adding the chess pipeline documentation by @rahuljo in #237
• adds deduplication of staging dataset during merge by @rudolfix in #240

New Contributors

• @rahuljo made their first contribution in #237

Full Changelog: 0.2.0a29...0.2.0a30

0.2.0.a29

What's Changed

• Allow changing write_disposition in the resource without dropping dataset by @burnash in #205
• Add a suffix to the default dataset name by @burnash in #207
• improves and adds several dlt pipeline commands: info, trace, load-package, failed-jobs and sync (https://dlthub.com/docs/command-line-interface#dlt-pipeline)
• extends LoadInfo to include the schema changes applied to destination and a list of loaded package infos (https://dlthub.com/docs/running-in-production/running#inspect-save-and-alert-on-schema-changes)
• extends load info with raise_on_failed_jobs and has_failed_jobs to make handling failed jobs easier
• LoadInfo and pipeline.last_trace can be directly loaded into destination to store more metadata on each load (https://dlthub.com/docs/running-in-production/running#inspect-and-save-the-load-info-and-trace)
• adds retry strategy for tenacity to retry load pipeline step (or any other per request) (https://dlthub.com/docs/running-in-production/running#handle-exceptions-failed-jobs-and-retry-the-pipeline)
• raise_on_failed_jobs config option aborts the load package on first failed jobs (https://dlthub.com/docs/running-in-production/running#failed-jobs)

What's Changed in docs

• Fix typos and wording in docs/concepts/state by @burnash in #200
• Fix a broken link in README.md by @burnash in #203
• replacing team@ with community@ by @TyDunn in #211
• GitHub and Google Sheets setup guides by @AmanGuptAnalytics in #195
• "run a pipeline" troubleshooting & walkthrough https://dlthub.com/docs/walkthroughs/run-a-pipeline
• "run a pipeline in production": https://dlthub.com/docs/running-in-production/running
• dlt pipeline command: https://dlthub.com/docs/command-line-interface#dlt-pipeline

Full Changelog: 0.2.0a28...0.2.0a29

0.2.0a28

What's Changed

• transform functions may be added to the resources ie. map, filters and generators. https://dlthub.com/docs/concepts/resource#filter-transform-and-pivot-data
• resources can be added to instantiated sources ie. to enrich data https://dlthub.com/docs/concepts/resource#feeding-data-from-one-resource-into-another and https://dlthub.com/docs/concepts/source#add-more-resources-to-existing-source

Docs

• improve explanations by @adrianbr in #181
• docs: init from other sources by @adrianbr in #182
• operation duck blog post by @TyDunn in #185
• added setup_guide_pipedrive to pipelines folder by @AmanGuptAnalytics in #183
• Docs orchestrators by @adrianbr in #166
• fix typo: add a space after gen(10) by @burnash in #196
• adds transform docs by @rudolfix in #192

New Contributors

• @AmanGuptAnalytics made their first contribution in #183
• @burnash made their first contribution in #196

Full Changelog: 0.2.0a26...0.2.0a28

0.2.0a26

What's Changed

• adds anonymous telemetry https://dlthub.com/docs/reference/telemetry
• adds pipeline and exception tracing with sentry https://dlthub.com/docs/reference/tracing

0.2.0a25

What's Changed

• getting started: bigquery --> duckdb by @TyDunn in #146
• dbt docs by @adrianbr in #164
• incremental pipeline docs by @adrianbr in #165
• dlt init working with pipelines repo by @rudolfix in #168

New dlt init command.

With this release you can use dlt init to add existing pipelines to your project. Please see updated documentation
The pipelines currently come from pipelines contrib repo. You can use any other repo or a local folder that follows the same structure (dlt init ... --location <url>)

0.2.0a23

What's Changed

• fixes multiple unnecessary alter tables to add columns by @rudolfix in #163

Full Changelog: 0.2.0a22...0.2.0a23

0.2.0a22

dlt library changes

• DATE type is supported on all destinations
• table and column names will be shortened to fit into particular destination
• duckdb database will be created in the current working directory by default #148
• fixes connection problem on duckdb 0.7.1
• allows to configure naming conventions or adopt naming conventions preferred by a destination
• streamlit app generated by dlt pipeline ... show does not display deprecation warnings

PRs

• iterating on blog posts by @TyDunn in #144
• improves naming conventions by @rudolfix in #157
• Add support for date type by @steinitzu in #160
• fixes duckdb database default location by @rudolfix in #159

0.2.0.a21

What's Changed

• improves naming convention protocol by @rudolfix in #143

This is mostly a 🐛 bugfix release: names of flattened columns were incorrectly normalized
It also introduces improved interface to naming convention used by the schemas in pipelines (try Schema.naming)

Full Changelog: 0.2.0a20...0.2.0a21

0.2.0a20

What's Changed

• groups all config and secret values used by all sources from a single python module in a single section (https://github.com/dlt-hub/dlt/blob/devel/docs/technical/secrets_and_config.md#the-sources-section)
• allows to provide alternatives of credentials for sources with multiple auth methods (https://github.com/dlt-hub/dlt/blob/devel/docs/technical/secrets_and_config.md#working-with-alternatives-of-credentials-union-types)
• gives write access to schemas in the source decorated function to allow to generate/modify settings and tables (https://github.com/dlt-hub/dlt/blob/devel/docs/technical/working_with_schemas.md#schema-is-modified-in-the-source-function-body)
• all generators are closed if there's an exception in the pipe during extraction

Full Changelog: 0.2.0a19...0.2.0a20