Update package metadata

[ofek]

• The code follows the Code Style Guidelines
• Tests have been added for the bug fix or new feature
• Docs have been added for the bug fix or new feature

Background

Hello there! This was requested in in-toto/in-toto#490 (comment)

The Python packaging ecosystem has standardized on the interface for build backends (PEP 517/PEP 660) and the format for metadata declaration (PEP 621/PEP 631). As a result, the use of setup.py files is now heavily discouraged.

So, I'm spending my free time updating important projects so that they are modernized and set an example for others 😄

Summary of changes

This implements PEP 621, obviating the need for setup.cfg and MANIFEST.in. Support has not landed in setuptools, so builds will now use hatchling. It is quite stable and actively maintained, the only reason the version is 0.x is because 1.0.0 will drop support for Python 2. It's also on the major distribution channels such as conda-forge, Debian, Fedora, etc.

I've done this for such projects as pipx, all Datadog Agent integrations, etc.

Notes

• The empty setup.py remains for Dependabot

[jku]

before you spend more time on implementation details, let's discuss if this is something we want. I mentioned this in the in-toto issue:

We ... would likely need good reasons to move to a less known build tool: I'm not trying to belittle hatchling, I'm just saying we try to be careful about pipeline security. Getting rid of MANIFEST.in might not be a good enough reason to make a major build tool change.

You provided a comprehensive comment that explains really well why you are building your project: I am convinced new build tools are a good idea. I'm not convinced there are enough advantages for python-tuf to migrate:

• our build is already declarative so improvements in that area seem really minor (do correct if this is wrong)
• hatchling is seemingly a two months old project and has one developer

I want to support cool python infrastructure work but I also take build pipeline choices seriously.

[ofek]

our build is already declarative so improvements in that area seem really minor

Declarative yes, but not in the standard format, and therefore no support for new standards like licensing.

hatchling is seemingly a two months old project

Since we use it at Datadog for shipping business critical things, our CI would be the first to yell at me if something goes awry 😄

We also were the first to use tuf + in-toto at scale, which I assume has contributed to their current stability.

and has one developer

As mentioned in the comment you linked, setuptools is almost exclusively maintained by one person and the codebase is an order of magnitude larger.

---

I rebased; would you mind triggering the CI with that button below again?

[coveralls]

Pull Request Test Coverage Report for Build 2004983729

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 98.311%

---

Totals
Change from base Build 1999224414:	0.0%
Covered Lines:	1178
Relevant Lines:	1194

---

💛 - Coveralls

[joshuagl]

Thanks for the proposal @ofek.

For context: our most recent round of changes to package metadata were to bring it inline with what is recommended in the Python Packaging User Guide, specifically Packaging and distributing projects. That tutorial does suggest considering alternatives, but does not motivate why one should. I appreciate your related remarks here and in the linked in-toto issue comparing alternatives. We did briefly look at flit in the past, but ultimately didn't feel the need to switch away from setuptools yet. One reason to consider alternatives is our desire to have verifiably reproducible build artefacts (#1269).

Aside from discussion of switching packaging tool, which I'll defer to more active maintainers on, there are some good unrelated fixes in this PR (tox isolated builds, gitignore fixes, redundant comment about version in tuf/init.py). @ofek would you mind if I pull those out into separate patches and credit you with Co-authored-by: in the commit message? See: https://github.com/joshuagl/tuf/commits/joshuagl/build-nits

[ofek]

would you mind if I pull those out into separate patches

Sure!

One reason to consider alternatives is our desire to have verifiably reproducible build artefacts (#1269).

Hatchling does that by default https://ofek.dev/hatch/latest/config/build/#reproducible-builds, including everything mentioned in that linked issue like file ordering

[jku]

Hatchling does that by default https://ofek.dev/hatch/latest/config/build/#reproducible-builds, including everything mentioned in that linked issue like file ordering

Now this is interesting and a reason to consider this, thanks.

[joshuagl]

Hatchling does that by default https://ofek.dev/hatch/latest/config/build/#reproducible-builds, including everything mentioned in that linked issue like file ordering

A quick glance at the source code tells me that includes producing reproducible sdist. Nice!

[ofek]

I updated the branch.

A quick glance at the source code tells me that includes producing reproducible sdist. Nice!

Yup you can test here by doing:

$ docker run --rm -v $PWD:/home/proj -w /home/proj python bash -c "pip install -qq build;python -m build -o /tmp/dist;python -c \"import hashlib,os,pathlib;print(os.linesep.join(hashlib.sha256(p.read_bytes()).hexdigest() for p in pathlib.Path('/tmp/dist').iterdir()))\""
...
a467c36feab72b9f1464b446eb6ec87bc2eeff40861c6c088c30f812900f7e6e
3e548ba8c530d6ff2b04737a24ae42073361927560e30c95aa4ae3f7d3bc4ab0
$ docker run --rm -v $PWD:/home/proj -w /home/proj python:alpine ash -c "pip install -qq build;python -m build -o /tmp/dist;python -c \"import hashlib,os,pathlib;print(os.linesep.join(hashlib.sha256(p.read_bytes()).hexdigest() for p in pathlib.Path('/tmp/dist').iterdir()))\""
...
a467c36feab72b9f1464b446eb6ec87bc2eeff40861c6c088c30f812900f7e6e
3e548ba8c530d6ff2b04737a24ae42073361927560e30c95aa4ae3f7d3bc4ab0

[lukpueh]

Thanks for your efforts, @ofek! Hatch/hatchling looks really cool and getting reproducible builds for free is quite a compelling argument.

I admit that I share the reservations @jku and @joshuagl have expressed, wrt preferring a more "established" tool. But this patch is simple enough to revert (or migrate to another PEP 621 compatible build system) if needed, which makes the bus factor of your project less worrisome. And the integrations you mentioned along with the lightweightness of the hatchling builders, make me confident that issues are detected and probably also fixed swiftly.

So I lean towards giving this a try.

[jku]

Agreed, potentially reproducible builds is a game changer, something I'm willing to take a bit of risk for.

Merging this doesn't get us quite to reproducible builds (doesn't pin the hatch/hatchling version we use for official builds for a start) but getting that right does not need to happen in this PR.

[jku]

Looks good overall: left a question and a few specific comments in source.

• Are the readthedocs.yaml changes unrelated?
• If you could rebase (so we don't end up with merge commits in this branch), that would be nice

[jku]

Actually one more question: since we have the setup.py workaround (for dependabot), how does it even work?

from setuptools import setup
setup()

...but we no longer depend on setuptools?

Or is it so that we know that nothing in the build/install pipeline ever runs setup.py -- it just needs to exist for dependabot?

[ofek]

Addressed and rebased!

Are the readthedocs.yaml changes unrelated?

Yes because by default it does python setup.py install

since we have the setup.py workaround (for dependabot), how does it even work? [..] it just needs to exist for dependabot?

Yes it's just for dependabot for now dependabot/dependabot-core#3290

[jku]

I can't think of anything else to add:

• I suppose we could make the dummy setup.py error out (to make sure no-one is running it by accident)... but that does not have to be in this PR
• thinking the reproducible build aspect through (e.g. do we want to pin the build tools) is important but again not required for this PR

Thanks for pushing the issue here ofek, I'm looking forward to trying out build reproducibility.

Approving. I wouldn't mind if another maintainer wants to have a look at the build results before merging.