Releases: dagster-io/dagster
0.7.2
Docs
- New docs site at docs.dagster.io.
- dagster.readthedocs.io is currently stale due to availability issues.
New
- Improvements to S3 Resource. (Thanks @dwallace0723!)
- Better error messages in Dagit.
- Better font/styling support in Dagit.
- Changed
OutputDefinition
to takeis_required
rather thanis_optional
argument. This is to
remain consistent with changes toField
in 0.7.1 and to avoid confusion
with python's typing and dagster's definition ofOptional
, which indicates None-ability,
rather than existence.is_optional
is deprecated and will be removed in a future version. - Added support for Flower in dagster-k8s.
- Added support for environment variable config in dagster-snowflake.
Bugfixes
- Improved performance in Dagit waterfall view.
- Fixed bug when executing solids downstream of a skipped solid.
- Improved navigation experience for pipelines in Dagit.
- Fixed for the dagster-aws CLI tool.
- Fixed issue starting Dagit without DAGSTER_HOME set on windows.
- Fixed pipeline subset execution in partition-based schedules.
0.7.1
Dagit
- Dagit now looks up an available port on which to run when the default port is
not available. (Thanks @rparrapy!)
dagster_pandas
- Hydration and materialization are now configurable on
dagster_pandas
dataframes.
dagster_aws
- The
s3_resource
no longer uses an unsigned session by default.
Bugfixes
- Type check messages are now displayed in Dagit.
- Failure metadata is now surfaced in Dagit.
- Dagit now correctly displays the execution time of steps that error.
- Error messages now appear correctly in console logging.
- GCS storage is now more robust to transient failures.
- Fixed an issue where some event logs could be duplicated in Dagit.
- Fixed an issue when reading config from an environment variable that wasn't set.
- Fixed an issue when loading a repository or pipeline from a file target on Windows.
- Fixed an issue where deleted runs could cause the scheduler page to crash in Dagit.
Documentation
- Expanded and improved docs and error messages.
Waiting To Exhale
🎆 🚢 🎆 Dagster 0.7.0: Waiting To Exhale 😤 😌 🍵
We are pleased to announce version 0.7.0 of Dagster, codenamed “Waiting To Exhale”. We set out to make Dagster a solution for production-grade pipelines on modern cloud infrastructure. In service of that goal, we needed to fill missing gaps and incorporate feedback from the community at large.
Our last release, 0.6.0, expanded Dagster from local developer experience to a hostable product, allowing for scheduling, execution, and monitoring of pipelines in the cloud.
This release goes further, supporting pipelines with 100s and 1000s of nodes, deployable to modern, scalable cloud infrastructure, with dramatically improved monitoring tools, as well as other features.
Given this, 0.7.0 introduces the following:
- Revamped, Scalable Dagit A completely redesigned Dagit with a more intuitive navigation structure, beautiful look-and-feel, and massive performance improvements to handle pipelines with hundreds or even thousands of nodes.
- Execution Viewer Executing and historical runs within Dagit uses a new live-updating, queryable waterfall viewer. See below for a preview of the new UI:
https://media.giphy.com/media/Rhx6ujovXlvuKaLCGY/giphy.gif
- A Dagster-K8s library which provides the ability to launch runs in ephemeral Kubernetes Pods, as well as an early helm chart for executing pipelines.
- A Dagster-Celery library designed to work with K8s that provides global resource management using dedicated queues, and distributed execution of dagster pipelines across a cluster.
- Streamlined scheduler configuration and new backfill APIs and tools to help manage your scheduled workflows in production.
- A Dagster-Pandas integration that provides useful APIs for dataframe validation, summary statistics emission, and auto-documentation in dagit so that you can better understand and control how data flows through your pipelines.
- Redesigned documentation, examples, and guides to help flesh out the core ideas behind the system.
Warning
There are a substantial number of breaking changes in the 0.7.0 release. These changes effect the scheduler system, config system, required resources, and the type system. We apologize for the thrash, and thank you for bearing with us!
For more info on changes check out the following resources:
Changelog: https://github.com/dagster-io/dagster/blob/master/CHANGES.md
0.7.0 migration guide: https://github.com/dagster-io/dagster/blob/master/070_MIGRATION.md
0.4.0
API Changes
- There is now a new top-level configuration section
storage
which controls whether or not
execution should store intermediate values and the history of pipeline runs on the filesystem,
on S3, or in memory. Thedagster
CLI now includes options to list and wipe pipeline run
history. Facilities are provided for user-defined types to override the default serialization
used for storage. - Similarily, there is a new configuration for
RunConfig
where the user can specify
intermediate value storage via an API. OutputDefinition
now contains an explicitis_optional
parameter and defaults to being
not optional.- New functionality in
dagster.check
:is_list
- New functionality in
dagster.seven
: py23-compatibleFileNotFoundError
,json.dump
,
json.dumps
. - Dagster default logging is now multiline for readability.
- The
Nothing
type now allows dependencies to be constructed between solids that do not have
data dependencies. - Many error messages have been improved.
throw_on_user_error
has been renamed toraise_on_error
in all APIs, public and private
GraphQL
- The GraphQL layer has been extracted out of Dagit into a separate dagster-graphql package.
startSubplanExecution
has been replaced byexecutePlan
.startPipelineExecution
now supports reexecution of pipeline subsets.
Dagit
- It is now possible to reexecute subsets of a pipeline run from Dagit.
- Dagit's
Execute
tab now opens runs in separate browser tabs and a newRuns
tab allows you to
browse and view historical runs. - Dagit no longer scaffolds configuration when creating new
Execute
tabs. This functionality will
be refined and revisited in the future. - Dagit's
Explore
tab is more performant on large DAGs. - The
dagit -q
command line flag has been deprecated in favor of a separate command-line
dagster-graphql
utility. - The execute button is now greyed out when Dagit is offline.
- The Dagit UI now includes more contextual cues to make the solid in focus and its connections
more salient. - Dagit no longer offers to open materializations on your machine. Clicking an on-disk
materialization now copies the path to your clipboard. - Pressing Ctrl-Enter now starts execution in Dagit's Execute tab.
- Dagit properly shows List and Nullable types in the DAG view.
Dagster-Airflow
- Dagster-Airflow includes functions to dynamically generate containerized (
DockerOperator
-based)
and uncontainerized (PythonOperator
-based) Airflow DAGs from Dagster pipelines and config.
Libraries
- Dagster integration code with AWS, Great Expectations, Pandas, Pyspark, Snowflake, and Spark
has been reorganized into a new top-level libraries directory. These modules are now
importable asdagster_aws
,dagster_ge
,dagster_pandas
,dagster_pyspark
,
dagster_snowflake
, anddagster_spark
. - Removed dagster-sqlalchemy and dagma
Examples
- Added the event-pipeline-demo, a realistic web event data pipeline using Spark and Scala.
- Added the Pyspark pagerank example, which demonstrates how to incrementally introduce dagster
into existing data processing workflows.
Documentation
- Docs have been expanded, reorganized, and reformatted.
0.2.8.post3
Hotfix to not put config values in error messages. Had to re-release because of packaging errors uploaded pypi (.pyc files or similar were included)
v.0.2.8.post0
Pushing an update because dagit 0.2.8 was getting out-of-date code.
v0.2.8
v.0.2.7
Version 0.2.7 Release Notes
Most notable improvements in this release are bunch of improvements to dagit, most notably hot reloading and the in-browser rendering of python error. Also the ability to scaffold configs from the command line is the first fruit of the rearchitecting of the config system.
-
Dagster improvements:
- Added scaffold_config command which generates the template of a yaml file needed to drive the execution of a particular pipeline
- Added the ability to automatically serialize intermediate inputs as they flow between solids. Consider this alpha quality. It is currently hard-coded to write out to /tmp/dagster/runs/<<run_id>>
-
Dagit improvements:
- Hot-Reloading and in-browser rendering of python errors.
- Scrolling and performance improvements
- Keyboard short cuts to navigate between solids using arrow keys
- In-app previews of notebooks for dagstermill solids
v0.2.6
Changes:
- 'run_id' value automatically included in ExecutionContext context
stack. This is a uuid. - Config system update:
This is a significant change in the config system. Now the top level environment objects (and all descendants) are now part of the dagster type system. Unique types are generated on a per-pipeline basis. This unlocks a few things:
- The entirety of yaml config files are now type-checked in the same fashion as the user-defined config.
- One can now pass dictionaries to execute_pipeline that mimic the yaml files exactly. You no longer have to use the dagster.config APIs (although those still work)
- The entire config system is queryable via graphql (and therefore shows up in dagit). This adds some noise to the type browser (we can mitigate that soon), but this will enable the building of a config-editor is fully aware of the dagster type system.
- This has one breaking change. The yaml file's format has changed slightly.
Previously:
context:
name: context_name
config: some_config_value
Now:
context:
context_name:
config: some_config_value
BREAKING CHANGE: Config format change. See above.