Skip to content

Releases: bpolaszek/bentools-etl

Major design refactoring

04 Apr 15:49
Compare
Choose a tag to compare
Pre-release

3.0 alpha is out! This is a complete redesign of the library, so don't upgrade from 2.5 if you don't intend to refactor your whole app, since almost everything has changed. This is now a lot simpler:

  • No more useless extractors: if your input data is already an iterable, you don't need one.
  • Removed ContextElement object to carry data during the ETL lifecycle - no need for that wrapper anymore.
  • Transformers now MUST return a \Generator even if they just transform a single value. Because transformers can now transform 1 thing into multiple others.
  • As a consequence, a \Generator is expected to be injected into all loaders.
  • Redesigned the built-in event dispatcher, in favor of a PSR-14 compliant one.
  • Events are now read-only, they can't modify data anymore. They still allow skipping the current item / stopping the process, and ignore exceptions at the item level. Way less difficult to maintain. Implement your own extractor / transformer and/or loader instead, if you want a fine control of what goes in and what goes out.
  • Dropped support for asynchronous ETL processing.
  • More decoupling.
  • New ETL builder with its recipes.
  • Newcomers can get started way more easier since there are less concepts to understand.

Transformer workflow (multiple transformers)

08 Jun 08:56
Compare
Choose a tag to compare

v2.5

07 Jun 14:26
Compare
Choose a tag to compare
Step and Stack transformers

Exceptions handling + CSV Iterators

05 May 15:22
Compare
Choose a tag to compare

Exception Handling

The extract, transform, or load part of the ETL process may sometimes throw exceptions.
Before throwing it, we should be able to hook on the exception, handle it, and decide wether or not to continue the process.

CSV Iterators

You can now use CsvFileIterator or CsvStringIterator to iterate over your CSVs.
Also check KeysAwareCsvIterator to automatically assign the keys of each CSV row (you can turn an indexed array into an associative one). Check out the Iterators documentation.