Python images broke - can we get a changelog?

[black-snow]

We recently upgraded our base images from, for example,

python:3.11-bookworm@sha256:f7543d9969bdc112dd9819ca642e14433fdacfe857f170f6b803392fc7e451ad

to

python:3.11-bookworm@sha256:157a371e60389919fe4a72dff71ce86eaa5234f59114c23b0b346d0d02c74d39

and that broke all our Python images. I'm trying to understand what exactly changed (and then how to fix it). I saw in the issues that setuptools and wheel were drop as well as something with ensurepip?

What I've noticed in the image itself so far is that the name of the python binary seems to have changed - my symlink to /usr/local/bin/python3.11 is dead, there's only /usr/local/bin/python.

[pataquets]

I'm getting build errors for an already-deployed, successfully building commitd, since yesterday.
Base image is pinned to :3.12-slim.
Googling the error mostly pop setuptools issues, but nothing about that on my Dockerfile, AFAIK (at least no change on that). So, @black-snow's guess looks reasonable to me (within my knowledge).

The error pops when building Django staticfiles, no code nor deps changed:

#12 [stage-0 7/9] RUN   DEFAULT_FROM_EMAIL=dummy   FEEDBACK_EMAIL=dummy   EMAIL_URL=consolemail://   DATABASE_URL=sqlite://:memory:   SECRET_KEY=dummy   STATICFILES_STORAGE=django.contrib.staticfiles.storage.StaticFilesStorage   python manage.py collectstatic --no-input --verbosity 2
#12 0.426 Traceback (most recent call last):
#12 0.426   File "/app/manage.py", line 15, in <module>
#12 0.426     execute_from_command_line(sys.argv)
#12 0.426   File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
#12 0.427     utility.execute()
#12 0.427   File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 416, in execute
#12 0.427     django.setup()
#12 0.427   File "/usr/local/lib/python3.12/site-packages/django/__init__.py", line 24, in setup
#12 0.427     apps.populate(settings.INSTALLED_APPS)
#12 0.427   File "/usr/local/lib/python3.12/site-packages/django/apps/registry.py", line 91, in populate
#12 0.427     app_config = AppConfig.create(entry)
#12 0.427                  ^^^^^^^^^^^^^^^^^^^^^^^
#12 0.427   File "/usr/local/lib/python3.12/site-packages/django/apps/config.py", line 193, in create
#12 0.427     import_module(entry)
#12 0.427   File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
#12 0.427     return _bootstrap._gcd_import(name[level:], package, level)
#12 0.427            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 0.427   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
#12 0.427   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
#12 0.427   File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
#12 0.427   File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
#12 0.427   File "<frozen importlib._bootstrap_external>", line 995, in exec_module
#12 0.427   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
#12 0.427   File "/usr/local/lib/python3.12/site-packages/rest_framework_simplejwt/__init__.py", line 1, in <module>
#12 0.427     from pkg_resources import DistributionNotFound, get_distribution
#12 0.427 ModuleNotFoundError: No module named 'pkg_resources'
#12 ERROR: process "/bin/sh -c DEFAULT_FROM_EMAIL=dummy   FEEDBACK_EMAIL=dummy   EMAIL_URL=consolemail://   DATABASE_URL=sqlite://:memory:   SECRET_KEY=dummy   STATICFILES_STORAGE=django.contrib.staticfiles.storage.StaticFilesStorage   python manage.py collectstatic --no-input --verbosity 2" did not complete successfully: exit code: 1
------

[pataquets]

In addition, same commit builds successfully with :3.9-slim.
Trying from :3.12-slim (which equals to :3.12.6-slim as of now) downwards, worked for me with :3.12.4-slim.
Fair enough.

[black-snow]

Regarding my particular issue I'm not quite sure where it stems from as there's a bunch of moving pieces involved (python base image, minimal debian base, copying things around, poetry, venvs, ...) - but for now I'll just fix it with
RUN ln -s /usr/local/bin/python /usr/local/bin/python3.11.

I'm still curious about what exactly changed. Could we perhaps get a changelog we could check before bumping the digest? It's rather difficult to manually diff what's changed in between versions ;)

[LaurentGoderre]

This looks very similar to #967

[tianon]

See also #957 (and other linked/related issues/PRs) 👀

[edmorley]

Hi

These are two completely separate issues (in general it's best to open separate issues rather than re-using an existing open issue when the symptoms aren't the same):

• @black-snow your issue is presumably due to the subtle change in shebang lines described in Shebangs using different Python binary with latest Docker image in 3.10 #967, that was a side-effect of Switch from get-pip.py to ensurepip #955, and changed in the images of all supported Python versions. Most apps will work with both the old and new shebang lines, but if you are performing unusual manual symlinking/moving files around (like the use-case in Shebangs using different Python binary with latest Docker image in 3.10 #967 was), then you may need to adjust your symlinks/manual steps. The path you mentioned (/usr/local/bin/python3.11) isn't in the images published from this repo, so it must be something you are creating manually.
• @pataquets your issue is due to setuptools no longer being installed in the Python 3.12+ images as of Stop installing setuptools in python 3.12+ #954. This removal will only affect broken packages that don't correctly specify their dependencies. It seems you are using an old version of djangorestframework-simplejwt that doesn't include the fix from jazzband/djangorestframework-simplejwt@19cf38e. Upgrade to a newer djangorestframework-simplejwt version or else manually install the setuptools package in the meantime.

[klausmyrseth]

Setuptools was distributes with 3.12.1-4 but adruptedly stopped in 3.12.5 ... Is this something that permanent or is it just something that was done in the middle of a version chain by accident?

[edmorley]

@klausmyrseth This was answered/linked to already above. Please read:

• Stop installing setuptools in python 3.12+ #954

[klausmyrseth]

ye but still quite interesting to do that on a minor verison update if you read my message it was a try to not be to harsh about the version usage and the fact that this is a breaking change for many. and to be completely sure as the update is HUGE (for many) and if someone suddenly brings it back that would break the builds again.

[edmorley]

The reason I linked to that PR, is that your message asked if this change was done by accident - and the PR title and description make it clear that it wasn't by accident. It sounds that you are instead meaning "why was this change made deliberately" (which is a very reasonable question to ask, but it's still a different question), in which case there's more detail on the reasoning why in:
#964 (comment)

[black-snow]

Thanks @edmorley !

Breaking changes suck and they are rather hard to avoid when the tag doesn't reflect a SemVer of the image but the underlying Python version (and who would want a 3.11-bookworm-1.17?).

So the remaining open question for me is if we could start providing a changelog. I pin all my digests and updates are deliberate, but it's veeeery cumbersome to find out what changed between two digests. I'll rename the issue to reflect that.

[edmorley]

I agree that normally a changelog is a must for a published project. However, I'm not sure a changelog would be very easy to implement in a useful way for this repository, due to it not being a typical type of package with versioned releases. How would you list the changes, when there isn't a version? By date? Also, the rate of commits on this repo is generally very low, so skim reading https://github.com/docker-library/python/commits/master/ should typically be sufficient to figure out exactly what changed in the last week or so - and any changes outside of version bumps are so rare the changelog will be redundant 99% of the time.

In general if you ever have a problem with any open source project, I'd recommend checking the GitHub releases section, followed by the commit history, then searching recently updated issues and PRs. In this case a search for the word "symlink" would have found the issue filed 5 days ago, describing exactly your issue:
https://github.com/docker-library/python/issues?q=is%3Aissue+symlink

[klausmyrseth]

What about bringing the links and names to the pull requests into a changelog that is easy accessible.

For me that would help me to see what's going on. For my sake I was very aware of the change python wise) but it can be hard to see what's going on with the docker. We had some code refering distutils in the old libs so I knew what to look for when the services refused to start suddently, but the first time I met the error it was hard to find the reason.

Also our thought process to remove the change when it was in v3.12.1 we thought you would not remove it until 13 as the version association brings that association to most users (like me)

[black-snow]

@edmorley I disagree with a bunch of that ;)

Ofc I found similar issues, but do I want to rely on that someone else has already run into a similar problem and filed an issue for me to find before I bump the digest? Certainly not.

Do I want to identify all diff commits and read them in order to see if anything relevant might have changed? Nope.

I get that its somewhat difficult and many docker images are in a rather bad shape regarding this, yet writing a simple CHANGELOG.md when substantial things change seems easy enough to me.