pipedag compatibility with pydiverse.transform >=0.2 #237
Conversation
implement NotImplementedError when pydiyverse.transform 0.2.0 is used
implement NotImplementedError when pydiyverse.transform 0.2.0 is used
…and new pdt; remove support for py 3.9 when using new pdt
…res; remove support for py 3.9 when using new pdt; drop support for osx for the time being (leads to pixi errors due to pypi dependencies); pavel: i don't care anymore
…res; remove support for py 3.9 when using new pdt; drop support for osx for the time being (leads to pixi errors due to pypi dependencies); pavel: i don't care anymore
prefect = ">=2.13.5, <3.0.0"
There was a problem hiding this comment.
lgtm, please add some pointers to the PR description.
I'd appreciate a second opinion from others, e.g., @nicolasmueller
| @@ -58,7 +58,6 @@ jobs: | |||
| - py312all | |||
| - py311all | |||
| - py310all | |||
| - py39all | |||
There was a problem hiding this comment.
could you please add to the PR description why we decided to drop Python 3.9 from the tests?
| @@ -37,7 +37,7 @@ kazoo = ">=2.8.0" | |||
| dask = ">=2022.1.0" | |||
|
|
|||
| [feature.prefect.dependencies] | |||
| prefect = ">=2.13.5" | |||
| prefect = ">=2.13.5, <3.0.0" | |||
There was a problem hiding this comment.
and could you please also add a note that/why we pinned the prefect versions?
| # ibm_db is not on conda-forge for macOS | ||
| # https://github.com/ibmdb/db2drivers/issues/3 | ||
| ibm-db = ">=3.1.4" | ||
| platforms = ["linux-64", "win-64"] # TODO "osx-arm64" |
There was a problem hiding this comment.
...and that/why we removed ibm-on-osx support?
There was a problem hiding this comment.
@pavelzw maybe that's also something you could elaborate on. I'm wondering if we should maybe unbundle this particular change. wdyt?
There was a problem hiding this comment.
The issue was that ibm-db on macos didn't have a conda-forge build back then (conda-forge/ibm_db-feedstock#77) and just got one recently in conda-forge/ibm_db-feedstock#79. In a future PR (#231) i will re-add macos support for ibm-db with the new package.
This lead to us using the ibm-db build on pypi for macos (in combination with telling people to install gcc next to clang) and thus activating pypi-dependencies in pixi.
This introduces some issues, namely in some cases (haven't found a minimal reproducer for this issue here, though) where pixi thinks the lockfile is out of date (which leads to pixi install --locked failing). xref prefix-dev/pixi#1295, prefix-dev/pixi#1417
I'm wondering if we should maybe unbundle this particular change
this would require a change of the lockfile which will likely lead to ci breakages. if it's not high effort to fix these breakages (or introduce corresponding pinnings in pixi.toml), we could do it 🤷🏻
This PR makes pydiverse.pipedag compatible with pydiverse.transform >0.2.0 while still maintaining support for pydiverse.transform <0.2.0.
However pydiverse.pipedag still doesn't support pydiverse.transform 0.2.0, as it significasntly differs from every other verison in files and functions
Changes:
Checklist
docs/source/changelog.mdentrydocs/source/docs/source/examples.md