Add python (et al.) to used variants even if only referenced via special selectors

[mbargull]

Description

Package-dependent special selectors like py and similar are hard-coded to be always available no matter if python etc. are actually recognized as used variant variables.

This is problematic since the values of those selectors are not well-defined in case the associated package is not pinned down as variant.
E.g., for a list of variants with different python versions, the py selector can be any of those versions and is in fact non-deterministically chosen in the current implementation.

We have instances in the ecosystem where a multi-output recipe does not have a dependency on python but some skip: true # [py ...] selector is used, as in

package:
  name: top-level
  version: 1.0
build:
  skip: true  # [py<39]
outputs:
  - name: sub-package
    requirements:
      host:
        - python

; e.g., see conda-forge/conda-smithy#1782 and the linked recipes in there.

There are multiple ways to argue about this:

1. The recipe could be seen as deficient since it uses py without python at the top level.
2. The py (etc.) selectors should only be defined if the corresponding packages are used.

Re. 1.: Yes, the recipe could be amended to have skip: true # [python and py<39] or move the skip line to each output.
Regardless, having an implicit non-deterministic behavior is something we should avoid in any case.

Re. 2.: If the selectors would not be defined, the rendering could gracefully fail with an error.
Changing the code to do this could be difficult since the rendering of the current recipe format is rather convoluted.
Plus, there could be recipes that expect those selectors to always be defined, i.e., it might be more disruptive regarding backwards compatibility to remove them.

The alternative, which is implemented here, is to pull in the packages into the "used variant variables" whenever such selectors are used.
This is also a breaking change (due to potentially introducing more variants), but less disruptive as the selectors' removals but possibly (e.g., for the top-level skip: true # [py<39] example above) even what the user expected in the first place.

N.B.: Personally, I'd consider those selectors as unnecessary tech-debt and hope the ecosystem can move off of their usage with the new recipe format which is currently worked on in the rattler-build project.

Checklist - did you ...

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation?

[mbargull]

• Add / update outdated documentation?

I didn't really see a place where selector<->variant interactions are addressed.
If anyone has suggestions for a doc addition, feel free to push to this branch.

[github-actions]

Hi there, thank you for your contribution!

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this pull request to remain open please:

1. Rebase and verify the changes still work
2. Leave a comment with the current status

NOTE: If this pull request was closed prematurely, please leave a comment.

Thanks!

[github-actions]

Hi there, thank you for your contribution!

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this pull request to remain open please:

1. Rebase and verify the changes still work
2. Leave a comment with the current status

NOTE: If this pull request was closed prematurely, please leave a comment.

Thanks!

[beckermr]

Not stale.