Lock files don't mesh with dynamic package version info

[hynek]

Since tox-uv just shipped lock files, I took a stab at moving attrs to a fully-locked dev environment.

Here's the experiment PR: python-attrs/attrs#1349

A problem I've run into is that packages often use dynamic packaging data that is based on git metadata (setuptools-scm, hatch-vcs, etc). As such, attrs's package version changes after every commit, which allows us to upload to test PyPI and continually verify our package. See, for example, https://test.pypi.org/project/attrs/24.2.1.dev19/

Unfortunately the current project gets locked like this:

[[package]]
name = "attrs"
version = "24.2.1.dev20"
source = { editable = "." }

Which means it's outdated after each commit, because every commit increments the number behind dev.

Is there a way around this that I've missed? If not, could there be built one? 😇 AFAICT, this is currently the biggest blocker for FOSS use from uv's side. I'm also not sure what the point of that version lock is in the first place for the current, editable project? But that's a topic for another day. :)

---

❯ uv --version
uv 0.4.12 (2545bca69 2024-09-18)

[sbidoul]

@hynek have you seen tool.uv.cache-keys ?

[hynek]

I have while searching this bug tracker, but I couldn't find a way for it to affect uv.lock? Changing the value to cache-keys = [{ git = true }, { file = "pyproject.toml" }] at least doesn't do anything when running uv.lock.

From the documentation (and its name 🤓) I've gathered it's mostly about how caching, not about locking? And TBH it sounds to me as if adding git as a git key, I'm actually adding cache busting.

[charliermarsh]

I think you could pass --upgrade-package attrs to ensure that attrs is rebuilt and that the lockfile is updated. Or even set it in your configuration:

[tool.uv]
upgrade-package = ["attrs"]

But there's sort of an infinite loop here, right? Since every time you commit the lockfile, it will be outdated. So you then re-lock, and commit again, etc. I don't know how to solve this holistically. We could omit versions for editables, maybe? Or even, write "dynamic" or something for the version, for packages that have dynamic versions?

[hynek]

But there's sort of an infinite loop here, right? Since every time you commit the lockfile, it will be outdated. So you then re-lock, and commit again, etc. We could omit versions for editables, maybe? Or even, write "dynamic" or something for the version, for packages that have dynamic versions?

Yeah that's what I meant in my last sentence, but I'm always assuming Chesterton's Fence. :) Setting it to dynamic seems the best way since it maps nicely to project.dynamic. Just dunno if it would be a problem that there's a case where there's an invalid version "number" in the field which would speak for omitting.

[jkeifer]

I too have this problem. I generally subscribe to the philosophy that I use a VCS for versioning, thus what is in the VSC repo shouldn't track it's own version because that's what I'm using the VCS for. Said another way, at best a version tracked within repo is imperfect for exactly the reasons described above about how you can't update a version from a commit without making another commit, i.e., a new version.

I'd love to understand the rationale behind storing this version in the lock file. Can anyone elaborate on that?

[burgholzer]

We are also facing a similar problem over at cda-tum/mqt-core#706 where we use renovate to keep the uv lock file up to date, which would be pretty convenient in principle.
However, currently it triggers a never ending chain of renovate update PRs because every update to the lock file adds a commit, which changes the setuptools-scm version, which causes another update PR to be triggered. (see cda-tum/mqt-core#703, cda-tum/mqt-core#704, cda-tum/mqt-core#705, cda-tum/mqt-core#706)

[LordFckHelmchen]

This problem is not only for dynamic versioning. We have a release-please workflow, which will create a PR for the next release. This will update the pyproject.toml version correctly but of course does not know about the uv.lock version.
If there would be an option to omit the project version from the uv.lock it would be great, as I currently also don't understand why it needs to be kept in the lock-file itself.

[david-waterworth]

As I mentioned in ttps://github.com//issues/7994 I usepython-semantic-release which behaves the same. The way I work around the infinite build loop is to configure my Jenkins script to skip commits that contains the python-semantic-release message - these commits include the changed pyproject.toml and __version__.py file, along with a generated CHANGELOG.md

As a workaround, I also added uv lock at the start of the build script and git add uv.lock at the end - this ensures that python-semantic-release commit includes the correctly versioned uv.lock file (I'm not sure yet if this causes other problems, i.e. updating other packages - ideally there would be a way of only updating the package version in uv.lock and nothing else.

[alejandrodnm]

I've just encountered this. We are also using release-please and uv in an internal project, after a release then the surprise is that the uv.lock was outdated.

Ours plans were to add uv to https://github.com/timescale/pgai but since we are already using release-please, we are hesitant due to this issue.

[RazerM]

While the setuptools-scm case seems difficult to solve, I think another common reason¹ a dynamic version is used is when the version is statically defined but not in pyproject.toml. Typically as a __version__ attribute like in this example:

1. Create example project

   uv init --lib mpypkg && cd mpypkg

2. Configure pyproject.toml to use a dynamic version

   cat <<EOF > pyproject.toml
   [build-system]
   requires = ["setuptools"]
   build-backend = "setuptools.build_meta"
   
   [project]
   name = "mpypkg"
   requires-python = ">=3.12"
   dynamic = ["version"]
   
   [tool.uv]
   cache-keys = [{ file = "pyproject.toml" }, { file = "src/mpypkg/__init__.py" }]
   
   [tool.setuptools.dynamic]
   version = { attr = "mpypkg.__version__" }
   
   [tool.setuptools]
   package-dir = { "" = "src" }
   
   [tool.setuptools.packages.find]
   where = ["src"]
   EOF

3. Add __version__ attribute to __init__.py

   cat <<EOF >> src/mpypkg/__init__.py
   __version__ = "1.0"
   EOF

4. Run uv lock

5. Change version number

   sed -i.bak 's/__version__ = "1\.0"/__version__ = "1.1"/' src/mpypkg/__init__.py

6. Run uv lock

What I expected:

 version = 1
 requires-python = ">=3.12"
 
 [[package]]
 name = "mpypkg"
-version = "1.0"
+version = "1.1"
 source = { editable = "." }

What happened: uv.lock still has version = "1.0"

I've also tried uv lock --no-cache --refresh to no avail.

The version doesn't change until I do an unrelated uv add or something. This seems like a cache invalidation problem more than anything else, which I expected cache-keys to solve.

---

❯ uv --version
uv 0.4.30 (61ed2a236 2024-11-04)

Footnotes

1. sometimes for backward compatibility reasons or in my case I can't move version into pyproject.toml until some other tooling I use catches up. ↩

[charliermarsh]

We can fix this, but I still strongly recommend against setting the version dynamically like that.

[charliermarsh]

I've fixed the issue @RazerM pointed out in #8867. I guess I'll leave this issue open though since it's more about the inherent incompatibility between lockfiles and VCS-based versioning.

[cpascual]

Hi @charliermarsh, you mention "inherent incompatibility of lock files and VCS-based versioning" , but you also mentioned:

We could omit versions for editables, maybe? Or even, write "dynamic" or something for the version, for packages that have dynamic versions?

To me the above workaround seems good enough, specially if it is controlled by an opt-in configuration (e.g. omit_lock_version=["mypackage",]).

Is there any fundamental problem with this approach?

[cpascual]

my use case is that I use setuptools-scm and the presence of commit-dependent version numbers in uv.lock is a frustrating cause of git merge conflicts preventing what otherwise would be an automated merge.

[ceejatec]

We can fix this, but I still strongly recommend against setting the version dynamically like that.

@charliermarsh Out of curiosity, why?

[Coruscant11]

I just discovered this issue with the recent release 0.5.0. Using --locked when syncing the project makes everything fails. I have a dynamic version based on git tags, and when we release we tag our repository. So the lock file can be outdated without any code change.

I think excluding editable dependencies in the lock file could be a good workaround. Or we can also write "dynamic" as a version. By the way I will check what tools like poetry for example are doing in this use case.
@charliermarsh Is there any news on this? In the meantime, we will pin our version to the 0.4.30.

By the way I would suggest to list it in the breaking changes of the release

Thanks !!!

Edit: Forgot to mention, I was wondering if you already have some implementation ideas, I totally volunteer to work on this if needed. I will try a draft pull-request on my side for fun

[Stealthii]

We can fix this, but I still strongly recommend against setting the version dynamically like that.

If references or opinion articles exist around this topic, they are few in number. I think a few on this issue would be interested in wider discussion around this. PEP 621 introduced toml metadata, and as of writing the official specification still allows version to be set dynamically, with no recommendation to the contrary.

It's a pretty standard paradigm (see hatch-vcs, setuptools-scm, poetry-dynamic-versioning) for single project repos where versioning, point release branches, development builds, etc. are all managed and controlled via VCS change control.

Whilst alternatives exist for separating static versions (such as 0.5.1 from a generated identifier ref like f399a5271 in uv 0.5.1 (f399a5271 2024-11-08)), there are some compelling reasons to use dynamic (such as dunamai) formed versions:

• Unique reference of all release, post, dev, dirty distributions of Python packages
• All tarball/wheel builds being uniquely available in an internal PyPI repository for further testing or triage
• Rules around versioning and releases can be set by project owners irrespective of development language
• No room for user error (such as forgetting to update a static version of 0.5.1 for the git tag v0.5.1 and publishing builds)
• Exposing the version in test logs, headers, etc. automatically shows the exact build used, aiding debugging

---

My expectation for the project package that has a dynamic version (source = { editable = "." } specifically) would be simply not defining version in uv.lock for the local package (or providing a dynamic reference). This by nature isn't a lockable reference (especially when set by VCS signatures) whereas other uses of uv lock such as multi-project mono repos or static versions uphold the current behaviour.

It's worth noting that although version is defined dynamically in source control, builds of the package provide the version statically:

❯ ls -1 dist/*
dist/mypackage-0.5.0.dev5+g5d056c1-py3-none-any.whl
dist/mypackage-0.5.0.dev5+g5d056c1.tar.gz
❯ unzip -p dist/*.whl '*.dist-info/METADATA' | grep -E '^Version: '
Version: 0.5.0.dev5+g5d056c1
❯ tar -Oxf dist/*.tar.gz '*.egg-info/PKG-INFO' | grep -E '^Version: '
Version: 0.5.0.dev5+g5d056c1

[charliermarsh]

If references or opinion articles exist around this topic, they are few in number.

This is just my personal opinion! Dynamic metadata makes a number of things more complicated than they need to be. I understand the use-case for scm versioning, but in my opinion using dynamic metadata to read a version from __init__.py is a really heavy hammer with the only benefit being that you get to avoid writing out a constant twice -- I was responding to that specific case, where I don't think the tradeoffs are very good.

[charliermarsh]

...simply not defining version in uv.lock for the local package (or providing a dynamic reference)

Unfortunately it's not super simple -- it breaks the fundamental assumption that packages have versions, so we have to take that into account everywhere.

[Coruscant11]

Unfortunately it's not super simple -- it breaks the fundamental assumption that packages have versions, so we have to take that into account everywhere.

Yes this is what I saw this week-end when I was trying to contribute on this 🥲
But anyway, I had the opportunity to deep dive in the code base so it's fine.
We will have to trust you on this 😄 (No need to worry at all then)

Thanks!

[ceejatec]

it breaks the fundamental assumption that packages have versions

Not exactly, I don't think. The package always DOES have a version. It's just that the version may change outside of uv's immediate notice. Perhaps it would be fine if uv lock and uv sync and similar commands re-assessed the version if it's marked dynamic in the pyproject.toml. That wouldn't handle cases where the dynamic version contains the git commit SHA in some form, but I suspect that's pretty much an intractable problem.

That said, I feel like a cleaner solution would be to omit the main package from uv.lock entirely. However I'm definitely not familiar with the implementation here, so I don't know what unforeseen consequences that might have.

[hynek]

If references or opinion articles exist around this topic, they are few in number.

This is just my personal opinion! Dynamic metadata makes a number of things more complicated than they need to be. I understand the use-case for scm versioning, but in my opinion using dynamic metadata to read a version from __init__.py is a really heavy hammer with the only benefit being that you get to avoid writing out a constant twice -- I was responding to that specific case, where I don't think the tradeoffs are very good.

By the way, this can be inverted and made more standard-compliant while doing so by putting something like this into your __init__.py:

def __getattr__(name: str) -> str:
    if name != "__version__":
        msg = f"module {__name__} has no attribute {name}"
        raise AttributeError(msg)

    from importlib.metadata import version

    return version("YOUR-PACKAGE")

You get static metadata with no duplication.

[ninoseki]

I have the same pain point and would like to share how many projects in the wild do Git based versioning for your information.

• hatch-vcs: 2.2k files
• poetry-dynamic-versioning: 1.6k files

[roteiro]

I have the same pain point and would like to share how many projects in the wild do Git based versioning for your information.

• hatch-vcs: 2.2k files
• poetry-dynamic-versioning: 1.6k files

add to that:

• versioningit 420 files

[jamesbraza]

I think #8867 released in uv==0.5.0 leads uv to correctly respect dynamic metadata, which now breaks setuptools-scm + pre-commit users using the uv-lock hook: astral-sh/uv-pre-commit#25.

This is textbook Hyrum's Law 😆

---

To share on the SCM versioning debate, I use setuptools-scm as it's a nice way for an automated (no human error) workflow of GitHub tagged release to PyPI publish with matching version.

[charliermarsh]

Yeah, it seems like fundamentally the only way to make this work with setuptools-scm is to omit the version or otherwise mark it as dynamic. I don't think it's impossible, but it's not, like, an hour of work. I'm also fairly worried about how we would integrate this if PEP 751 gets approved -- the PEP doesn't make any carveouts for dynamic versions, and there hasn't been any discussion of it in the PEP thread. If we add a feature like this here, and it doesn't make it into the PEP, we'll have to remove it later.