All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, adheres to Semantic Versioning, and is generated by Changie.
- add missing self arg in gcs job
- ensure all Job classes do not inherit from NamedTuple since attempt tracking per payload is mutable
- relax python version to allow 3.11
- added a MAX_JOBS_QUEUED variable to regulate workers's global jobs queue
- fix jobs class attempts variable incrementing
- ensure sinks wait for workers to finish jobs before killing them
- each worker gets its own offset in the storage write implementation since they may process multiple tables
- New schema_resolver_version config option. Defaults to 1 which means no change for existing users. Users can opt into a version 2 which is much more resilient and will fallback to JSON columns when unable to resolve a particular property. This option is only used when denormalized=True
- Establish message passing pattern from workers to main thread for process introspectability
- Bubble up errors from workers and reraise in main thread with traceback info
- Option to ignore failures in workers via fail_fast config option. This option is set to True and you must explicitly set fail fast to false if you want to process to ignore row insertion errors
- Manage offset in sink class instead of on workers so the stream payload always has the correct offset
- Use changie for changelog management