Use more recent Python packages

[dougbu]

we have a large number of fixed Python package versions in our runtime_python_requirements.txt file as well as a few compatibility (~=) and bound (>= or >=...<=) ranges. the impact of this is we're falling behind Python package deprecations. it's also inconsistent

for example, discovered our docker package version (v3.7.2) is from 2019 while working on #4307. fix for that issue intentionally left the package version unchanged when using Python 3.6 and 3.7 b/c that PR was focused on Azure Linux 3 (which now uses v7.1.0). we should circle back and move to docker v6.1.3 for down-level Python versions. we shouldn't be using such an old docker package

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

updated some Python packages to use newer versions and to consistently apply [chosen approach]

for example, today we'll use docker package to v5.0.3 with Python 3.6 or 3.7 and v7.1.0 with newer Python versions. this update will also allow both versions to update as new v5.x or v7.x versions are released. were previously using the very old v3.7.2 package everywhere

[dougbu]

turns out Azure Linux 3 did not drive a docker package update. this issue now tracks a general version bump for that package

[dougbu]

generalizing this issue b/c our frozen Python package versions are a general issue. there are a few ways forward:

1. remove upper bounds for all Python packages when used w/ 3.12 or greater and use compatibility ranges in other "bands"

• this has the upside of letting the queues move forward as reboot and downside of potentially hitting breaking changes in production

2. run short jobs using different Python versions to determine current versions then use pip freeze to determine the requirements for each. use the generated requirements files in both helix.base.yaml and runtime_python_requirements.txt

• this has the upside of confirming all version changes b/4 rolling out and downside of being a fairly large change

3. choose some mix of the above e.g. using compatibility ranges for each "band" of Python versions

I lean toward (3) but would appreciate other thoughts

[dougbu]

@dotnet/dnceng I would appreciate comments on approaches just above