Spack binary buildcache not fetching correct keys

[heatherkellyucl]

When we have the manylinux entry in our mirrors and are creating a new site, this happens:

[ccspapp@build01 hpc-spack]$ spacksites/spacksites create hk-stack

...

# SPACKSITES: Now calling:  /lustre/scratch/ccspapp/spack/0.22/hpc-spack/spacksites/process-env-scripts/spack-deps-spack-env-run.sh /shared/u
cl/apps/spack/0.22/hk-stack/spack/share/spack/setup-env.sh spack gpg create ARCHPCSolutions [email protected]
# SPACKSITES: have set SPACK_DISABLE_LOCAL_CONFIG=1
# SPACKSITES: have set HPC_SPACK_ROOT=/lustre/scratch/ccspapp/spack/0.22/hpc-spack
# SPACKSITES: Have set spacks's external compiler and python dependencies - now calling spack gpg create ARCHPCSolutions [email protected]
gpg: keyring `/lustre/shared/ucl/apps/spack/0.22/hk-stack/spack/opt/spack/gpg/secring.gpg' created
gpg: keyring `/lustre/shared/ucl/apps/spack/0.22/hk-stack/spack/opt/spack/gpg/pubring.gpg' created
gpg: skipping control `%no-protection' ()
gpg: /lustre/shared/ucl/apps/spack/0.22/hk-stack/spack/opt/spack/gpg/trustdb.gpg: trustdb created
gpg: key B59A0FE1 marked as ultimately trusted
# SPACKSITES: Now calling:  /lustre/scratch/ccspapp/spack/0.22/hpc-spack/spacksites/process-env-scripts/spack-deps-spack-env-run.sh /shared/ucl/apps/spack/0.22/hk-stack/spack/share/spack/setup-env.sh spack buildcache keys --install --trust
# SPACKSITES: have set SPACK_DISABLE_LOCAL_CONFIG=1
# SPACKSITES: have set HPC_SPACK_ROOT=/lustre/scratch/ccspapp/spack/0.22/hpc-spack
# SPACKSITES: Have set spacks's external compiler and python dependencies - now calling spack buildcache keys --install --trust
==> Fetching https://binaries.spack.io/develop/developer-tools-manylinux2014/build_cache/_pgp/0ACDCFDA91DB974A68C3DDC2F85815B32355CB19.pub
gpg: key 2355CB19: no valid user IDs
gpg: this may be caused by a missing self-signature
gpg: Total number processed: 1
gpg:           w/o user IDs: 1
==> Error: Command exited with status 2:
'/bin/gpg2' '--batch' '--import' '/lustre/shared/ucl/apps/spack/0.22/hk-stack/build_stage/ccspapp/build_cache/0ACDCFDA91DB974A68C3DDC2F85815B32355CB19.pub'

(Continues to create site, but then fails during some installs because it tries to get them from manylinux and cannot).

This lists the public key fingerprints:

https://binaries.spack.io/develop/developer-tools-manylinux2014/build_cache/_pgp/index.json

{"keys":{"0ACDCFDA91DB974A68C3DDC2F85815B32355CB19":{},"2C8DD3224EF3573A42BD221FA8E0CA3C1C2ADA2F":{},"78F3726939CA1B94893B66E8BC86F6FB94429164":{}}}

They can be got from eg https://binaries.spack.io/develop/developer-tools-manylinux2014/build_cache/_pgp/0ACDCFDA91DB974A68C3DDC2F85815B32355CB19.pub

spack buildcache keys --install --trust doesn't work.

[heatherkellyucl]

The 0ACDCFDA91DB974A68C3DDC2F85815B32355CB19.pub which is the e4s-uo-spack-01 key always fails because it has no valid user ID (name). The other two keys can be trusted.

spack gpg trust 2C8DD3224EF3573A42BD221FA8E0CA3C1C2ADA2F.pub
spack gpg trust 78F3726939CA1B94893B66E8BC86F6FB94429164.pub

[heatherkellyucl]

Now it is successfully fetching developer tools from binaries.spack.io

[owainkenwayucl]

On Kathleen and Myriad ccspapp has a folder in ~ called gpg_keys_for_spack which has the keys in it.