2.10.9

Features

EFS Volume support for @batch

This release adds support for mounting existing EFS volumes with @batch

Usage

You can specify one or multiple volumes to be mounted with the batch decorators efs_volumes= attribute. The volumes will be mounted under /mnt/volume-name by default, but you can also specify a custom mount point separated by a semicolon with the volume.

Examples

@batch(efs_volumes="fs-001")
@step
...
@batch(efs_volumes=["fs-002", "fs-003"])
@step
...
@batch(efs_volumes="fs-003:/mnt/custom-mountpoint"])
@step

Support custom mount point for host volumes with @batch

This release also adds support for custom mount points for host volumes with @batch.
Example

@batch(host_volumes="/home:/mnt/host-homedir")
@step

Improvements

@conda and @pypi changes

@pypi can now handle dependencies with special characters in versions

Fixes a @conda bug where environments were not appending to PATH during runtime if the decorator was applied implicitly to the step. This bug affected dependencies that install a binary, causing them to not be found during runtime due to the environment missing from the PATH.

Decorator lifecycle improvement

Previously the lifecycle methods task_pre_step and task_decorate were being called consecutively on a per-decorator basis. This release changes the lifecycle so that task_pre_step is called on all decorators before any task_decorate is called.

Better checking of datastore dependencies

This release improves the way that datastore dependencies are checked, in order to avoid an issue where previously run metadata could be published to the metadata service, but the execution fails to start due to missing dependencies for pushing data to the datastore.

What's Changed

• [@card] Realtime cards (1/N) by @valayDave in #1550
• fix rendering of zero-length DataFrames in default card by @amerberg in #1652
• fix: pypi decorator handling packages with special characters by @saikonen in #1643
• fix: conda environment counts as disabled with implicit decorator by @saikonen in #1668
• fix: decorator init order by @saikonen in #1669
• don't create run ids if dependencies for data store are missing i.e. fail fast by @madhur-ob in #1666
• Bump follow-redirects from 1.15.2 to 1.15.4 in /metaflow/plugins/cards/ui by @dependabot in #1675
• [@card] Realtime cards (2/N) by @valayDave in #1551
• AWS Batch decorator: EFS volumes support (and custom mount point) by @enricomarchesin in #1650
• add linting for EFS volume mount by @madhur-ob in #1679
• Bump version to 2.10.9 by @saikonen in #1680

New Contributors

• @madhur-ob made their first contribution in #1666
• @enricomarchesin made their first contribution in #1650

Full Changelog: 2.10.8...2.10.9