drain-swamp

Python build backend with build plugins and dependency lock switch

* Python 3.9 through 3.13, PyPy

new in 2.0.x

approach for requirements organize by venvs rather than folders; new implementation for pipenv-unlock commands;

new in 1.8.x

add dependency drain-swamp-snippet; use msftcangoblowm/drain-swamp-action; add support for .shared.in; command pipenv-unlock fix;

What swamp?

Code in Makefile or python scripts should be reduced or removed entirely, in favor of packaged, unittested, and well documented code.

These files are the favorite target for those placing obfuscated code triggering malware.

Lets call this hiding place, the swamp

Boilerplate

Authors and maintainers deal with many packages. Boilerplate code is copy+paste into multiple packages.

• an eye sore
• completely untested code
• small variations leak in
• code quality and feature improvements are less likely to happen

drain-swamp started out to reduce this

Generated files

For Python packages, which contain generated files, UX is improved by generating those files during the build process.

These batteries are included:

• setuptools-scm version file

• dependency lock switch

  .lock, .unlock, and .lnk files

Micro services use message queues for inter-process communications. The messages use protobuf message protocol which produces a generated file. That must be part of the build process.

Would be a good fit for a build plugin

version file

The build plugin for interacting with version file, it's about having flexibility on which version str to use.

Different circumstances calls for different version str

• CI tagged release workflow needs tag version from the version file
• CI on push workflows need the current version
• python -m build provide the new version str

This flexibility allows to test building a package before git push or a tagged release.

Get scm (source control management) version

scm-version get

0.5.2.dev0+g2988c13.d20240724

Get from version file

drain-swamp tag

0.5.2

Write a semantic version str to version file. drain-swamp pretag to check/fix semantic version str

scm-version write "0.5.2post0.dev1"

Features

Updating docs

Before a commit, update the date and version str in several locations

updates

• Sphinx docs/conf.py
• CHANGES.rst
• NOTICE.txt

This Sphinx conf.py contains a snippet. The entire contents of the snippet is replaced. This technique is now a separate package, drain-swamp-snippet

Dependency lock switch

Authors disappear or die. Unfunded projects quickly become abandonware. Packages with locked dependencies do not age well.

Lets check the license. Hmmm Apache2.0 abandonware, that's a great reason to turn the dependency lock off.

pipenv-unlock is a light switch to turn on/off dependency locking.

On your repo, set a CI variable and that is the switch.

When the repo is inactive, turn off the switch and make a release without dependency locking.

How it works

A snippet in pyproject.toml containing both dependencies and optional-dependencies. There is additional meta data as well.

Refresh both .unlock and .lock files. During build time, .lnk shortcut is created.

Create dependency files with the .in extension. These include the dependencies and lines with -r and -c to include other dependency files.

Then

Create both lock and unlock dependency files

pipenv-unlock lock
pipenv-unlock unlock

Update the pyproject.toml snippet and refreshes symlinks (.lnk)

pipenv-unlock refresh --set-lock "off"
pipenv-unlock refresh --set-lock "on"

set lock state values

State	Possible values
lock	"1", "true", "t", "yes", "y", "on"
unlock	"0", "false", "f", "no", "n", "off"

build config settings

The Python packages build process occurs within a subprocess. The hottest trending topic is how to pass config settings to this subprocess?

Right before python -m build, depending on context, use whichever method is most appropriate.

custom build backend

This would only work for a custom build backend. Will see it's use only in drain-swamp howto.txt

python -m build -C--kind="0.5.1a4.dev6" -C--set-lock="0"

Unless authoring a custom build backend, can safely ignore.

cli

Use bash-workaround

tox

Similiar to cli. During pre_command, the TOML file and environment variable DS_CONFIG_SETTINGS are created.

tox test -- drain-swamp-tox-test

tox -- drain-swamp-tox

github workflows

drain-swamp-action creates the TOML file and environment variable, DS_CONFIG_SETTINGS.

Immediately after this gh action, there is fair bit of:

upload and download artifacts, between step communication, and maybe between jobs communication.

• matrix size == 1 drain-swamp-release-yml

There is one job. Communication is only between steps. e.g. ubuntu-latest-3.10

• matrix size > 1 drain-swamp-quality-yml

There are several jobs. A parent job occurs once. Constraining artifact upload to only occur once.

See also gh workflows folder -- drain-swamp-gh-workflows