tests: Add fail_on_changes to toxgen

[sentrivana]

Add fail_on_changes to toxgen. The idea is that the script will now have two modes:

• Normal mode (when fail_on_changes is False) that is used to actually generate the tox.ini file. This will be run in a cron job in CI and create a PR with the updated test setup.
• The newly added fail-on-changes mode (when fail_on_changes is True) that is used to detect manual changes to one of the affected files without updating the rest (e.g. making a manual change to tox.ini without updating the tox.jinja template). This will be run in CI similar to the fail_on_changes check of split-tox-gh-actions.

The problem with detecting manual changes is that if we just reran the script on each PR, chances are it would pull in new releases that are not part of the tox.ini on master, making the file look different from what was committed as if it had unrelated manual changes.

To counteract this, we now store the timestamp when the file was last generated in tox.ini. We use this in fail-on-changes mode to filter out releases that popped up after the file was last generated. This way, the package versions should be the same and if there is anything different in tox.ini, it's likely to be the manual changes that we want to detect.

Closes #4051

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
22963	1	22962	5844

View the top 1 failed test(s) by shortest run time

tests.integrations.stdlib.test_httplib test_http_timeout

Stack Traces | 0.114s run time

.../integrations/stdlib/test_httplib.py:419: in test_http_timeout
    assert len(transaction["spans"]) == 1
E   assert 0 == 1
E    +  where 0 = len([])

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[szokeasaurusrex]

Looks good, but please address comments before merging

[szokeasaurusrex]

[nit – feel free to ignore]

Suggested change

	if older_than is not None:
	if datetime.fromisoformat(meta["upload_time_iso_8601"]) > older_than:
	continue
	if older_than is not None and datetime.fromisoformat(meta["upload_time_iso_8601"]) > older_than:
	continue

[szokeasaurusrex]

Some suggestions to improve readability and clarity:

Suggested change

	Please make sure to not make manual changes to `tox.ini`. The file
	is generated from a template in `scripts/populate_tox/tox.jinja`
	by the `scripts/populate_tox/populate_tox.py` script. Any changes
	should be made to the template or the script directly and the
	resulting `tox.ini` file should be generated with:
	Please do not make manual changes to `tox.ini`. The
	`scripts/populate_tox/populate_tox.py` script generates `tox.ini`
	using the `scripts/populate_tox/tox.jinja` template.
	Instead, modify the template or the script as needed, then run the
	following commands to regenerate `tox.ini`:

[sentrivana]

Reworded this to also include #4072 (comment)

[szokeasaurusrex]

Can this error also occur if the jinja template and/or populate_tox.py script are changed, but we forget to regenerate the tox.ini file afterwards? If yes, please mention this possibility, as well