Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

spack in interior container fails when sourcing setup script or when running any spack command #17

Open
frankwillmore opened this issue Mar 4, 2020 · 21 comments
Open

spack in interior container fails when sourcing setup script or when running any spack command #17

frankwillmore opened this issue Mar 4, 2020 · 21 comments

Comments

@frankwillmore
Copy link
Member

When running build pipeline prototype using demo configuration, spack fails and generates a python traceback when sourcing setup-env.sh or running e.g. spack --version. The spack.yaml is never processed.

Also, /opt/spack/bin/spack is not included in the PATH when the container is run.

I am starting from the .gitlab-ci.yml at:
https://github.com/scottwittenburg/sc2019-tutorial-pipeline/blob/master/.gitlab-ci.yml

The pipeline fails even with both the version in /opt/spack or with a freshly cloned copy of v0.14.0.

. /opt/spack/share/spack/setup-env.sh
/opt/spack/bin/spack --version

$ . /opt/spack/share/spack/setup-env.sh
Traceback (most recent call last):
  File "/opt/spack/lib/spack/spack/environment.py", line 396, in validate
    spack.schema.Validator(spack.schema.env.schema).validate(data)
  File "/opt/spack/lib/spack/external/jsonschema/validators.py", line 114, in validate
    raise error
jsonschema.exceptions.ValidationError: Additional properties are not allowed ('final-stage-rebuild-index' was unexpected)

Failed validating 'additionalProperties' in schema['patternProperties']['^env|spack$']['properties']['gitlab-ci']:
    {'additionalProperties': False,
     'patternProperties': {'bootstrap': {'items': {'anyOf': [{'type': 'string'},
                                                             {'additionalProperties': False,
                                                              'properties': {'compiler-agnostic': {'default': False,
                                                                                                   'type': 'boolean'},
                                                                             'name': {'type': 'string'}},
                                                              'required': ['name'],
                                                              'type': 'object'}]},
                                         'type': 'array'},
                           'mappings': {'items': {'additionalProperties': False,
                                                  'properties': {'match': {'items': {'type': 'string'},
                                                                           'type': 'array'},
                                                                 'runner-attributes': {'additionalProperties': True,
                                                                                       'properties': {'image': {'oneOf': [{'type': 'string'},
                                                                                                                          {'properties': {'entrypoint': {'items': {'type': 'string'},
                                                                                                                                                         'type': 'array'},
                                                                                                                                          'name': {'type': 'string'}},
                                                                                                                           'type': 'object'}]},
                                                                                                      'tags': {'default': [],
                                                                                                               'items': {'type': 'string'},
                                                                                                               'type': 'array'},
                                                                                                      'variables': {'default': {},
                                                                                                                    'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}},
                                                                                                                    'type': 'object'}},
                                                                                       'required': ['tags'],
                                                                                       'type': 'object'}},
                                                  'required': ['match',
                                                               'runner-attributes'],
                                                  'type': 'object'},
                                        'type': 'array'}},
     'required': ['mappings'],
     'type': 'object'}

On instance['ispack']['gitlab-ci']:
    CommentedMap([('mappings',
                   [CommentedMap([('match', ['trilinos os=ubuntu18.04']),
                                  ('runner-attributes',
                                   CommentedMap([('image',
                                                  CommentedMap([('name',
                                                                 'scottwittenburg/spack-tutorial-builder'),
                                                                ('entrypoint',
                                                                 [''])])),
                                                 ('tags',
                                                  ['spack-kube',
                                                   'r5.2xlarge']),
                                                 ('variables',
                                                  CommentedMap())]))]),
                    CommentedMap([('match', ['os=ubuntu18.04']),
                                  ('runner-attributes',
                                   CommentedMap([('image',
                                                  CommentedMap([('name',
                                                                 'scottwittenburg/spack-tutorial-builder'),
                                                                ('entrypoint',
                                                                 [''])])),
                                                 ('tags', ['spack-kube']),
                                                 ('variables',
                                                  CommentedMap())]))])]),
                  ('final-stage-rebuild-index',
                   CommentedMap([('tags', ['spack-kube']),
                                 ('image',
                                  CommentedMap([('name',
                                                 'scottwittenburg/spack-tutorial-builder'),
                                                ('entrypoint', [''])]))]))])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/opt/spack/lib/spack/spack/main.py", line 651, in main
    env = ev.find_environment(args)
  File "/opt/spack/lib/spack/spack/environment.py", line 267, in find_environment
    return Environment(env)
  File "/opt/spack/lib/spack/spack/environment.py", line 551, in __init__
    self._read_manifest(f)
  File "/opt/spack/lib/spack/spack/environment.py", line 578, in _read_manifest
    self.yaml = _read_yaml(f)
  File "/opt/spack/lib/spack/spack/environment.py", line 406, in _read_yaml
    validate(data, filename)
  File "/opt/spack/lib/spack/spack/environment.py", line 399, in validate
    e, data, filename, e.instance.lc.line + 1)
spack.config.ConfigFormatError: /builds/root/spack-env/spack.yaml:44: Additional properties are not allowed ('final-stage-rebuild-index' was unexpected)
$ echo "sourced setup script"
sourced setup script
$ spack --version
Traceback (most recent call last):
  File "/opt/spack/lib/spack/spack/environment.py", line 396, in validate
    spack.schema.Validator(spack.schema.env.schema).validate(data)
  File "/opt/spack/lib/spack/external/jsonschema/validators.py", line 114, in validate
    raise error
jsonschema.exceptions.ValidationError: Additional properties are not allowed ('final-stage-rebuild-index' was unexpected)

Failed validating 'additionalProperties' in schema['patternProperties']['^env|spack$']['properties']['gitlab-ci']:
    {'additionalProperties': False,
     'patternProperties': {'bootstrap': {'items': {'anyOf': [{'type': 'string'},
                                                             {'additionalProperties': False,
                                                              'properties': {'compiler-agnostic': {'default': False,
                                                                                                   'type': 'boolean'},
                                                                             'name': {'type': 'string'}},
                                                              'required': ['name'],
                                                              'type': 'object'}]},
                                         'type': 'array'},
                           'mappings': {'items': {'additionalProperties': False,
                                                  'properties': {'match': {'items': {'type': 'string'},
                                                                           'type': 'array'},
                                                                 'runner-attributes': {'additionalProperties': True,
                                                                                       'properties': {'image': {'oneOf': [{'type': 'string'},
                                                                                                                          {'properties': {'entrypoint': {'items': {'type': 'string'},
                                                                                                                                                         'type': 'array'},
                                                                                                                                          'name': {'type': 'string'}},
                                                                                                                           'type': 'object'}]},
                                                                                                      'tags': {'default': [],
                                                                                                               'items': {'type': 'string'},
                                                                                                               'type': 'array'},
                                                                                                      'variables': {'default': {},
                                                                                                                    'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}},
                                                                                                                    'type': 'object'}},
                                                                                       'required': ['tags'],
                                                                                       'type': 'object'}},
                                                  'required': ['match',
                                                               'runner-attributes'],
                                                  'type': 'object'},
                                        'type': 'array'}},
     'required': ['mappings'],
     'type': 'object'}

On instance['ispack']['gitlab-ci']:
    CommentedMap([('mappings',
                   [CommentedMap([('match', ['trilinos os=ubuntu18.04']),
                                  ('runner-attributes',
                                   CommentedMap([('image',
                                                  CommentedMap([('name',
                                                                 'scottwittenburg/spack-tutorial-builder'),
                                                                ('entrypoint',
                                                                 [''])])),
                                                 ('tags',
                                                  ['spack-kube',
                                                   'r5.2xlarge']),
                                                 ('variables',
                                                  CommentedMap())]))]),
                    CommentedMap([('match', ['os=ubuntu18.04']),
                                  ('runner-attributes',
                                   CommentedMap([('image',
                                                  CommentedMap([('name',
                                                                 'scottwittenburg/spack-tutorial-builder'),
                                                                ('entrypoint',
                                                                 [''])])),
                                                 ('tags', ['spack-kube']),
                                                 ('variables',
                                                  CommentedMap())]))])]),
                  ('final-stage-rebuild-index',
                   CommentedMap([('tags', ['spack-kube']),
                                 ('image',
                                  CommentedMap([('name',
                                                 'scottwittenburg/spack-tutorial-builder'),
                                                ('entrypoint', [''])]))]))])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/opt/spack/lib/spack/spack/main.py", line 651, in main
    env = ev.find_environment(args)
  File "/opt/spack/lib/spack/spack/environment.py", line 267, in find_environment
    return Environment(env)
  File "/opt/spack/lib/spack/spack/environment.py", line 551, in __init__
    self._read_manifest(f)
  File "/opt/spack/lib/spack/spack/environment.py", line 578, in _read_manifest
    self.yaml = _read_yaml(f)
  File "/opt/spack/lib/spack/spack/environment.py", line 406, in _read_yaml
    validate(data, filename)
  File "/opt/spack/lib/spack/spack/environment.py", line 399, in validate
    e, data, filename, e.instance.lc.line + 1)
spack.config.ConfigFormatError: /builds/root/spack-env/spack.yaml:44: Additional properties are not allowed ('final-stage-rebuild-index' was unexpected)```
@tgamblin
Copy link
Member

tgamblin commented Mar 5, 2020

@scottwittenburg @opadron

@scottwittenburg
Copy link
Collaborator

scottwittenburg commented Mar 5, 2020 •
edited
Loading

Hi @frankwillmore, thanks for giving it a try. I'm curious if this is happening when running the spack-infrastructure setup or in your own instance of gitlab?

At any rate, this .gitlab-ci.yml hasn't been merged yet, but that repo will be a better starting point.

The errors you pasted make me wonder if that really is spack 0.14, as final-stage-rebuild-index should be allowed in your spack.yaml in that version of spack. Did you set SPACK_REPO and SPACK_REF environment variables in your CI settings?

@frankwillmore
Copy link
Member Author

@scottwittenburg This is with gitlab/gitlab-ee:12.1.0-ee.0 as specified in the docker-compose, I didn't change anything there.

I edited the .gitlab-ci.yml to clone and check out different versions of spack (including the 0.14.0 tag), but wasn't able to run spack --version or source the setup-env.sh for any of them.

I
was able to docker run -it scottwittenburg/spack-tutorial-builder on the outer instance of docker and run spack inside a shell there, but not in the dind. This spack reported itself as v0.13.0.

I'm running docker desktop 2.2.0.0 on MacOS 10.15.3.

@opadron
Copy link
Member

opadron commented Mar 5, 2020

Also, /opt/spack/bin/spack is not included in the PATH when the container is run.

This is because the docker container has to run with no entrypoint, due to how Gitlab runner works with docker image.

The version of Spack under /opt doesn't work with your chosen .gitlab-ci.yml because the latter uses newer features in Spack and I don't think @scottwittenburg has recently rebuilt his worker image. (Is that right, Scott?)

You should be able to clone and use a newer version, though. All you'd need to do is set SPACK_REPO and SPACK_REF variables in your Gitlab project, or simply edit the yml file and hardcode their values.

Failing the above, can you post your docker-compose.yaml and .gitlab-ci.yml files? The answer is likely in those files.

@scottwittenburg
Copy link
Collaborator

scottwittenburg commented Mar 5, 2020 •
edited
Loading

Thanks @opadron, you're right that's an old image and probably shouldn't be used anymore. The branch on the repo I linked above (spack-tutorial-container) has a better and more recent example .gitlab-ci.yml, and should be used instead of the sc2019-tutorial-pipeline repo (there we use spack/ubuntu-bionic instead). Also I did mention the need for setting SPACK_REPO and SPACK_REF in the gitlab ci environment variables, but didn't hear back from @frankwillmore whether that's how he tried to use a newer spack.

I'm currently trying out the workflow in this repo to see if it's still working the way it is documented in gitlab-docker/README.md.

@scottwittenburg
Copy link
Collaborator

Oh, and I just remembered that spack 0.14.0 included the distributed builds PR, an otherwise really great feature which broke the automated pipelines (because it broke spack install --only [dependencies|package], something the pipelines rely on).

@scottwittenburg
Copy link
Collaborator

So I just ran through the README, starting from a clean slate, and found that with a couple exceptions, the workflow is mostly working fine. I created #18 to reflect the issues I found. Here are the two files which worked for me:

The .gitlab-ci.yml:

pipeline-job:
    image:
      name: spack/ubuntu-bionic
      entrypoint: ['']
    tags:
      - spack-k8s
    before_script:
      - git clone ${SPACK_REPO} --branch ${SPACK_REF}
      - . "./spack/share/spack/setup-env.sh"
    script:
      - spack ci start
        --spack-repo ${SPACK_REPO}
        --spack-ref ${SPACK_REF}
        --downstream-repo "${DOWNSTREAM_CI_REPO}"
        --branch-name "${CI_COMMIT_REF_NAME}"
        --commit-sha "${CI_COMMIT_SHA}"
    after_script:
      - rm -rf "./spack"

And the spack.yaml (just an example, obviously):

spack:
  definitions:
  - gcc_system_packages:
    - matrix:
      - - zlib
        - [email protected]
        - [email protected] cppflags=-O3
        - tcl
        - tcl ^[email protected] cppflags=-O3
        - lmod
      - ['%[email protected]']
  specs:
  - $gcc_system_packages

  mirrors: { "mirror": "s3://spack-public/mirror" }

  gitlab-ci:
    mappings:
    - match: [os=ubuntu18.04]
      runner-attributes:
        image:
          name: spack/ubuntu-bionic
          entrypoint: ['']
        tags: [spack-k8s]
        variables: {}
    final-stage-rebuild-index:
      tags: [spack-k8s]
      image:
        name: spack/ubuntu-bionic
        entrypoint: ['']
    enable-debug-messages: True

  cdash:
    build-group: Couple small-ish packages
    url: http://cdash
    project: spack
    site: Docker-Compose Testing Framework

  view: false
  config: {}
  modules:
    enable: []
  packages:
    all:
      target: [x86_64]
  repos: []
  upstreams: {}
  concretization: separately

And with that, I was able to run a pipeline reporting to CDash, uploading binaries to the minio S3 bucket, etc... And for the custom spack I set SPACK_REPO and SPACK_REF in the CI environment variables as follows:

SPACK_REPO=https://github.com/spack/spack.git
SPACK_REF=develop

@scottwittenburg
Copy link
Collaborator

Actually, I wonder if I should update the README to actually specify the content of those two files, rather than assuming some other repo has up-to-date examples?

@frankwillmore
Copy link
Member Author

Below is what I get while using the docker-compose.yml from commit 813df65 (latest), and the gitlab-ci.yml that Scott just posted. Now I see the below stacktrace:

Traceback (most recent call last):
  File "/builds/root/spack-env/spack/lib/spack/spack/config.py", line 119, in first_existing
    return next(k for k in keys if k in dictionary)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builds/root/spack-env/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/builds/root/spack-env/spack/lib/spack/spack/main.py", line 702, in main
    env = ev.find_environment(args)
  File "/builds/root/spack-env/spack/lib/spack/spack/environment.py", line 268, in find_environment
    return Environment(env)
  File "/builds/root/spack-env/spack/lib/spack/spack/environment.py", line 582, in __init__
    self._read()
  File "/builds/root/spack-env/spack/lib/spack/spack/environment.py", line 611, in _read
    self._read_manifest(f)
  File "/builds/root/spack-env/spack/lib/spack/spack/environment.py", line 640, in _read_manifest
    for item in config_dict(self.yaml).get('definitions', []):
  File "/builds/root/spack-env/spack/lib/spack/spack/environment.py", line 368, in config_dict
    key = spack.config.first_existing(yaml_data, env_schema_keys)
  File "/builds/root/spack-env/spack/lib/spack/spack/config.py", line 121, in first_existing
    raise KeyError("None of %s is in dict!" % keys)
TypeError: not all arguments converted during string formatting
$ spack ci start --spack-repo ${SPACK_REPO} --spack-ref ${SPACK_REF} --downstream-repo "${DOWNSTREAM_CI_REPO}" --branch-name "${CI_COMMIT_REF_NAME}" --commit-sha "${CI_COMMIT_SHA}"
Traceback (most recent call last):
  File "/builds/root/spack-env/spack/lib/spack/spack/config.py", line 119, in first_existing
    return next(k for k in keys if k in dictionary)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builds/root/spack-env/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/builds/root/spack-env/spack/lib/spack/spack/main.py", line 702, in main
    env = ev.find_environment(args)
  File "/builds/root/spack-env/spack/lib/spack/spack/environment.py", line 268, in find_environment
    return Environment(env)
  File "/builds/root/spack-env/spack/lib/spack/spack/environment.py", line 582, in __init__
    self._read()
  File "/builds/root/spack-env/spack/lib/spack/spack/environment.py", line 611, in _read
    self._read_manifest(f)
  File "/builds/root/spack-env/spack/lib/spack/spack/environment.py", line 640, in _read_manifest
    for item in config_dict(self.yaml).get('definitions', []):
  File "/builds/root/spack-env/spack/lib/spack/spack/environment.py", line 368, in config_dict
    key = spack.config.first_existing(yaml_data, env_schema_keys)
  File "/builds/root/spack-env/spack/lib/spack/spack/config.py", line 121, in first_existing
    raise KeyError("None of %s is in dict!" % keys)
TypeError: not all arguments converted during string formatting```

@scottwittenburg
Copy link
Collaborator

Are you following the instructions in gitlab-docker/README.md? If so, can you paste the values of SPACK_REPO and SPACK_REF you added to your gitlab CI environment variables?

@scottwittenburg
Copy link
Collaborator

Also regarding the stack traces you're pasting: are you getting those from the gitlab job output terminal?

@frankwillmore
Copy link
Member Author

Progress!

Yes, I've been following the instructions in gitlab-docker/README.md and only deviated in an effort to troubleshoot after seeing failures. I've since torn it all down and started fresh using the .gitlab-ci.yml and spack.yaml above. I would second inlining these with the directions so as to maintain a working snapshot.

I have been and am currently setting the values of SPACK_REPO and SPACK_REF to https://github.com/spack/spack.git and v0.14.0, respectively. And yes, I am cutting and pasting from the gitlab job output terminal.

Currently it appears to be installing spack and sourcing the setup-env.sh correctly. The job now appears to fail reading a URL (see below). Also, I notice that there are two variables CI_COMMIT_REF_NAME and CI_COMMIT_SHA referenced in the gitlab-ci.yml but I don't have values for them or otherwise see being set.

Cloning into 'spack'...
Note: checking out '97d46dc36f2512e5d93f640499100c8bf5ce0f8f'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

$ . "./spack/share/spack/setup-env.sh"
$ spack ci start --spack-repo ${SPACK_REPO} --spack-ref ${SPACK_REF} --downstream-repo "${DOWNSTREAM_CI_REPO}" --branch-name "${CI_COMMIT_REF_NAME}" --commit-sha "${CI_COMMIT_SHA}"
Initialized empty Git repository in /builds/root/spack-env/.git/
warning: adding embedded git repository: spack
hint: You've added another git repository inside your current repository.
hint: Clones of the outer repository will not contain the contents of
hint: the embedded repository and will not know how to obtain it.
hint: If you meant to add a submodule, use:
hint: 
hint: 	git submodule add <url> spack
hint: 
hint: If you added this path by mistake, you can remove it from the
hint: index with:
hint: 
hint: 	git rm --cached spack
hint: 
hint: See "git help submodule" for more information.
rm 'spack'
[master (root-commit) 8345107] Auto-generated commit testing master (04559d833869c975dff103a05e9bf26cf52ef846)
 4 files changed, 453 insertions(+)
 create mode 100644 .gitlab-ci.yml
 create mode 100755 .spack-env/transaction_lock
 create mode 100644 README.md
 create mode 100644 spack.yaml
fatal: unable to access 'http://root:gitlab-secret@gitlab:10080/root/spack-env.git
/': Illegal characters found in URL
==> Warning: Unable to populate buildgroup without CDash credentials
==> Error: Command exited with status 128:
'/usr/bin/git' 'push' '--force' 'downstream' 'master:multi-ci-master'
==> Stages for phase "specs"
==>   Staging summary:
==>     stage 0 (10 jobs):
==>       libbsd/w7unr45 -> [email protected]%[email protected]  arch=linux-ubuntu18.04-x86_64
==>       libiconv/zvmmgjb -> [email protected]%[email protected]  arch=linux-ubuntu18.04-x86_64
==>       libsigsegv/3khohgm -> [email protected]%[email protected]  arch=linux-ubuntu18.04-x86_64
==>       pcre2/ikxxwcl -> [email protected]%[email protected]  arch=linux-ubuntu18.04-x86_64
==>       pkgconf/eifxmps -> [email protected]%[email protected]  arch=linux-ubuntu18.04-x86_64
==>       unzip/fxlf2hu -> [email protected]%[email protected]  arch=linux-ubuntu18.04-x86_64
==>       xz/ur2jffe -> [email protected]%[email protected]  arch=linux-ubuntu18.04-x86_64```

@scottwittenburg
Copy link
Collaborator

scottwittenburg commented Mar 6, 2020 •
edited
Loading

Progress!

Yes, I've been following the instructions in gitlab-docker/README.md and only deviated in an effort to troubleshoot after seeing failures. I've since torn it all down and started fresh using the .gitlab-ci.yml and spack.yaml above. I would second inlining these with the directions so as to maintain a working snapshot.

I'm still not clear what was causing the initial problems you reported. I mean, it seems to have been the wrong version of spack compared to what you intended (0.14.0), it just not clear to me why that was the case. To the extent possible, sharing the .gitlab-ci.yml and spack.yaml which led to that situation could be useful.

Also, in spack 0.14.0, the cli command arguments --only dependencies and --only package were broken (that will likely result spack install failures when you get to that point) but recent versions of develop have been working. For now you might want to find a version of spack that works for you and keep it on a branch on your fork or somewhere.

I have been and am currently setting the values of SPACK_REPO and SPACK_REF to https://github.com/spack/spack.git and v0.14.0, respectively. And yes, I am cutting and pasting from the gitlab job output terminal.

Thanks for clarifying.

Currently it appears to be installing spack and sourcing the setup-env.sh correctly. The job now appears to fail reading a URL (see below). Also, I notice that there are two variables CI_COMMIT_REF_NAME and CI_COMMIT_SHA referenced in the gitlab-ci.yml but I don't have values for them or otherwise see being set.

You can echo CI_COMMIT_REF_NAME and CI_COMMIT_SHA from your gitlab job (script section) if you want to make sure gitlab is setting them for you. They're used in the auto-generated commit message (going away soon thanks to a feature we've eagerly anticipated !) and in picking part of the name of the branch to push to.

Cloning into 'spack'...
Note: checking out '97d46dc36f2512e5d93f640499100c8bf5ce0f8f'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

$ . "./spack/share/spack/setup-env.sh"
$ spack ci start --spack-repo ${SPACK_REPO} --spack-ref ${SPACK_REF} --downstream-repo "${DOWNSTREAM_CI_REPO}" --branch-name "${CI_COMMIT_REF_NAME}" --commit-sha "${CI_COMMIT_SHA}"
Initialized empty Git repository in /builds/root/spack-env/.git/
warning: adding embedded git repository: spack
hint: You've added another git repository inside your current repository.
hint: Clones of the outer repository will not contain the contents of
hint: the embedded repository and will not know how to obtain it.
hint: If you meant to add a submodule, use:
hint: 
hint: 	git submodule add <url> spack
hint: 
hint: If you added this path by mistake, you can remove it from the
hint: index with:
hint: 
hint: 	git rm --cached spack
hint: 
hint: See "git help submodule" for more information.
rm 'spack'
[master (root-commit) 8345107] Auto-generated commit testing master (04559d833869c975dff103a05e9bf26cf52ef846)
 4 files changed, 453 insertions(+)
 create mode 100644 .gitlab-ci.yml
 create mode 100755 .spack-env/transaction_lock
 create mode 100644 README.md
 create mode 100644 spack.yaml
fatal: unable to access 'http://root:gitlab-secret@gitlab:10080/root/spack-env.git
/': Illegal characters found in URL
==> Warning: Unable to populate buildgroup without CDash credentials
==> Error: Command exited with status 128:
'/usr/bin/git' 'push' '--force' 'downstream' 'master:multi-ci-master'

This is where we're trying to push the generated commit to a different branch on the DOWNSTREAM_CI_REPO you see in the fatal:unable to access message a few lines above. Maybe your DOWNSTREAM_CI_REPO doesn't match the actual project repository in some way (name, organization/user)?

==> Stages for phase "specs"
==> Staging summary:
==> stage 0 (10 jobs):
==> libbsd/w7unr45 -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64
==> libiconv/zvmmgjb -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64
==> libsigsegv/3khohgm -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64
==> pcre2/ikxxwcl -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64
==> pkgconf/eifxmps -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64
==> unzip/fxlf2hu -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64
==> xz/ur2jffe -> [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64```

The stuff above makes it look like the jobs were staged at least, we just need to figure out why it can't be pushed.

@frankwillmore
Copy link
Member Author

Okay, I had a \n at the end of SPACK_REPO, shame on me. Tore down again and I'm now able to get as far as spack generating and starting the CI stages, but one of the jobs fails in stage 0, and later stages are skipped. Also, CDash is not getting any results.

  on bc2e52909e12 bN4msTi1
Using Docker executor with image spack/ubuntu-bionic ...
Pulling docker image spack/ubuntu-bionic ...
Using docker image sha256:402136596f97f39574776d733fd58fe2cd490d6b815c10a5ee6a69f247766fe5 for spack/ubuntu-bionic ...
Running on runner-bN4msTi1-project-1-concurrent-0 via bc2e52909e12...
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/root/spack-env/.git/
Checking out 0e575ddb as multi-ci-master...
Removing jobs_scratch_dir/
Removing spack.yaml_BACKUP

Skipping Git submodules setup
$ git clone "https://github.com/spack/spack.git" --branch "develop" --depth 1 --single-branch
Cloning into 'spack'...
$ pushd ./spack && git rev-parse HEAD && popd
/builds/root/spack-env/spack /builds/root/spack-env
fd69994c680773e28ae473e2be8664355dbf448b
/builds/root/spack-env
$ . "./spack/share/spack/setup-env.sh"
$ spack -d ci rebuild
Running after script...
$ rm -rf "./spack"
Uploading artifacts...
jobs_scratch_dir: found 6 matching files           
WARNING: cdash_report: no matching files           
Uploading artifacts to coordinator... ok            id=79 responseStatus=201 Created token=E_8QYEk4
ERROR: Job failed: exit code 1

@scottwittenburg
Copy link
Collaborator

Thanks for sticking with it. I see the spack you're getting has the sha fd69994c68, but I don't see that reachable from today's develop. Just curious what it might be, maybe a commit on your fork?

Anyway, then one way to see what actually went wrong (because CDash isn't yet getting results in your case) you can browse to the artifacts and view the pipeline.log file. It should contain more information (possibly including details as to why you don't see any results in CDash). If you could upload that file here as an attachment, that would be great. Thanks!

@frankwillmore
Copy link
Member Author

frankwillmore commented Mar 9, 2020 •
edited
Loading 

ktsai7 added cmake_args (#15411)
Latest commit
fd69994
1 hour ago

It just landed. Where can I find pipeline.log?

@scottwittenburg
Copy link
Collaborator

job_page_artifacts_circled

If you click on "Browse" button, you can navigate through to jobs_scratch_dir/logs and there you should see pipeline_log.txt. I had the name wrong I guess, sorry. You can also download all the artifacts from the job as a zip then look at them locally.

@frankwillmore
Copy link
Member Author

OK, found it. I hadn't started minio and the log seemed to complain that it couldn't connect to it so I started it and created 'spack-public' bucket and now it complains it can't find the object. Also, still nothing showing in CDash.

==> [2020-03-09-21:46:21.779793, 41] Imported gpg from built-in commands
==> [2020-03-09-21:46:21.782531, 41] Imported gpg from built-in commands
==> [2020-03-09-21:46:21.784271, 41] Imported compiler from built-in commands
==> [2020-03-09-21:46:21.785746, 41] Imported compiler from built-in commands
==> [2020-03-09-21:46:21.785871, 41] Imported ci from built-in commands
==> [2020-03-09-21:46:21.787248, 41] Imported ci from built-in commands
==> [2020-03-09-21:46:21.787633, 41] cdash_base_url = http://cdash
==> [2020-03-09-21:46:21.787667, 41] cdash_project = spack
==> [2020-03-09-21:46:21.787689, 41] cdash_project_enc = spack
==> [2020-03-09-21:46:21.787710, 41] cdash_build_name = [email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 (Couple small-ish packages)
==> [2020-03-09-21:46:21.787733, 41] cdash_site = Docker-Compose Testing Framework
==> [2020-03-09-21:46:21.787753, 41] related_builds = 
==> [2020-03-09-21:46:21.787773, 41] job_spec_buildgroup = Couple small-ish packages
==> [2020-03-09-21:46:21.787796, 41] ci_artifact_dir = /builds/root/spack-env
==> [2020-03-09-21:46:21.787871, 41] root_spec = eJztXFuTo6oWft+/ot/2k6kEUNL9V06dmgJERAURwduvP6SvGmfKTGpPurNPd1VXRRJYuNa37mhrOHv6K3qoVJ0+/fUQ/jpuW1nrp4e/jzv49/MQsSx/+fLhwVTEZbVVTw+V1H44G/1Rt08Pnnrt/OG426PXrx2xgrunh+GY/EheBlmtjKy4fVtXE8WfHgRjr9fv28A7tNv/9faT1hAWfke9rJzUz8OG2PCFC79/W4tlFRFhI//579uAMauhYTgfys4HqnQ9Ukn6MZByw3XKNZP8nXjlydvHh4ectPnTQ59CUWtfM5UPU1+xDMqGGEeL0YimbPz7z91o+Mfk6Pk209l1YHn5QSYK/1ngYTu2jqtzogD0mclRjVJtMbaiLkTZMaDKSnFTiuxiotbrJU1Tt3I4J4dLPLRx4UuDJtGMU6wgwlSMJJ4AZAO8gpxj1TkVB4XBGNZt2hvHyoqXgX8ixghMtEGAXsPJOdEXKtDSpNNc9ToDqOnzJnWOdlr7snJTzF7k9bzij5cJupENdR0zzHSiRkBmcdfjCRGZMWvr/qRgb7B4B3a8g7v4W79+V78087b9uHyTmVBSTgRx0hcdzUcvBSJT7SafjkobTa5BhuUk7Efzc1qIxrzuWiJ5xycVY019xnJcEy0z1KvLVWtGy+tJmnNC2VBlIPckrZXQQ0wJz3IoIXetZJTF5peElnjuOldb5qocO5qRtq56LlpuPIMpl9L1KzxvGqxoKYYPn5HsDvfpM9pRnaQRtpKRquVvm+A2DNNgeKx/G/tU9AfLzWqdnQOFy2xQpk19Q9tO2BK0Q2w0ZmPMLSHaXoDIl4US2tnBl7YtMPS5LyhmjpZJzaYi7x2oVlDZ1Ltoued3GR12yQ7eJVRuBoDfkO1cJJsTojPLNg/59vepvv8fLullXZlwB1FBbdsg3I0VZSPMx44jZpCCqlsBYtNfRXP3Mzfm32i4REMvctNzgWxOiH4V338I5/AD/9j/AN8C+m11FdKdq2pR6IYbZYxNoB11gvyUIVcPgJStoKi9SFVvkvm9rGtTkKKeu0Ek0jSlxpw0+diCnhLM4xGsELeZDUYfbHkXLNiB+FWy9wawkDu6chnLfSrmiHf1zwI3w8caTWMZEtgQZaVkLPEEcJNAndTxlF2WSpwWV6Rc5SmjqzXMq6LLGhhTmhY5sp52xLYEpT0tL1o8+LZVGh5nMTYUpWAkOmC68V4MCVNjgZtOwOKyhc8SID4YstJLVdV6mmxqOagTX7GmKdo8yxJoYgc1zq8hFGDo+LAiNZEkT5pGQ+NU1euRyE4mBa3HfgBB3lcldQEWMki9W9HqlBIF1ZmdaB0kBGTRDKwwpexwA8lVZYwTrVSDVWZQAIvJRHJXOAfScQjcrHQsBpvVMlijK0m5ul6hYvC6Ql6hALa6wF0ZUoi2k5xMDWjjroovIqXQ+bLeJagiiTWO9XULfV+B0sSq4jEGeZXLi5atg0q27WrLfODEnsQuvOpVXvVmEtwnEowqlTWdruGOYQGt54RkOQw9q4qGZtAmmLA4LhKbctaWCQagvooQX2smgiDE+LyICeoLMtYuS8NlikotCk1SfA2d6ZQBn9GpQScbPFopSZ/ovjGE4MqZcWpKUuv8CgS/liqC7vVWqKxpMLWw7UEJTdVhm6suYLZcObZNzx2d2d6PCArsksfv8Om3Xdkf0tBl7fePYntZHSsl8qkmQ5VXZhS2H2Gc9wYYlwU6qnQryG367OidR7NqB9odjncJtuCWWf4h/ugBHjEmNEbH7HiMH8NHAAEmAMA953t0DKHpPhg4ftgngGf8eMjYI+CPB4oz+MjS/f59ofAFTQ5JjAik5ECOacJSfGR8f8Qcs8cEJgkDj3vKwprhw2O2fzyCY4ofIT3GkB1fF2qlaLnoPqdIt+DOP8Sbf4gzP+1Ohb2/sutMu2CZ17lQfU5FN/C8ctgyracmG3g1dujXteYNi36RcZir1+aEaHUXc5t++E6JL6lZXCTuuVg2J0Qzoz1vcu13h/sUiCFaLYxKmxPL08WQy0+l1PbrNAhESlet4ETBTvR5pkKwO1igQI5HEDtrqHFk9JcF5j9RbdTjKU5FXTs2NjHCDcOaABiiXO6Ra9eec9ODR7P9z3xnQMGdYugzoXC79uVruaGwpGHGjkPbcwk1l04Cb7pmbGyrqFrBYROX0VlpYw6J5BsSX6wMdcOYvasLBghGPDBaTGU/llxl1TRkXZyybJxWUNsshUWzQtdMvMnxTgugKbG9fC55LIqgom2JkcuxE2qeuxKrMbAc1CJ3zoAvVFa9YVnnllWQsrGktkWSpymJuwKVQmMydUZgZAs30hW8Nwuy0ZJVc0O6O6R3ifCXthjj1n2h8OuP2kDHX2Oq26Kx8EVTdHHDi95NxONJUdZkDnPLWhLX8QqNm+oXzfY/g2JI3u7Tp9fGSSUnvsChkWwre7hxzncRQOaC3JwQzds2877h7vEuBRmYRttPktEvyjWn/Zwpeo+9tihuxCle66oql8pNvG1iEFMh46sV/aJu2xwfmxOixR28S2a/O+zvNK66scpeJOrF0c2tCdGyATqTCbjXSg2d5CkqnSvty4Gs+dDpeMP8OjDba9k6K7VYxb6DqpYLhjtaXA/T1zESL7d/ZiOcsqLrce0lndBQViFybIchZD4KpEncXxUCvzHmjFSWEJpPcMpIgunYKNR7EuuYsNElvlZXNelveMr7JNozMnL0iXZWwFQ1rCFdJ3jVZflUWDP23F1RC39GzBkVb0GRZdwTKCYR106rRmQ6S/kgXDGw6hoq61TZizhu6k55jUQrYpM1U7DXKSxQYGMyrg9/bx6EiOaIm4Vv+919Nrg+MzD7mT6nMsu8k9UK/V3p8Why3Ls0HiEGaQkNdrmIK+faEegLAPOqRkDkLaVJKXMgnS5Zw0CXSEukKgC0aIWJTWsSne/6oyUDd/i7I3NN5Hejo0O/ga05JDYnRGf3MDu6ujsk35C4IPq7SNgL2701IVo68VnK9ninKZsZXX6a+mVqkrc89PfnHge6VcByy0JWJmLeCUSHARUjDOCq6oKipKpLlBWsylbqtBnZRu9cmnW8wQ7dpSbd2LhdhKbFwZCtCdFHLD93NvD7RMhXtmK/kXLN0bA5IVoeiZ65uhB/kG9EXIGIj4LJGSrg4HpXAI/QgBSSuDYj64hxjA1pP/bFtai46PT6ovWwNSH6yX3MamCPu8N3XfLLG4uL4LY4vLY1IVo+1DAzFmiXfAPiS5wRf+2mau7jou+Vr0GiijaBCWwS6g2bijLJ108IbD6WEs0f2ZgFDvsdvNOXGSgfvqSj45/ZbbzokZeFR9+aEP3kDTTvbILwnU/3Jq5///uKXg9h5Y3SWWknw5lMOeh8xTw2SNBEsjJZv8lg86VC0cdLgt4FcNwld1qI/lQc3DIRv+g1Tov689aE/wESkO8D
==> [2020-03-09-21:46:21.787907, 41] remote_mirror_url = s3://spack-public/mirror
==> [2020-03-09-21:46:21.787929, 41] job_spec_pkg_name = pcre2
==> [2020-03-09-21:46:21.787949, 41] compiler_action = NONE
==> [2020-03-09-21:46:21.788257, 41] Current working directory: /builds/root/spack-env, Contents:
==> [2020-03-09-21:46:21.788313, 41]   .gitlab-ci.yml
==> [2020-03-09-21:46:21.788339, 41]   spack
==> [2020-03-09-21:46:21.788361, 41]   .git
==> [2020-03-09-21:46:21.788381, 41]   README.md
==> [2020-03-09-21:46:21.788401, 41]   .spack-env
==> [2020-03-09-21:46:21.788421, 41]   jobs_scratch_dir
==> [2020-03-09-21:46:21.788441, 41]   spack.yaml
==> [2020-03-09-21:46:21.788555, 41] job concrete spec path: /builds/root/spack-env/jobs_scratch_dir/specs/pcre2.yaml
==> [2020-03-09-21:46:21.788585, 41] ci.import_signing_key() will attempt to import a key
==> [2020-03-09-21:46:21.806525, 41] spack gpg list:
==> [2020-03-09-21:46:21.806685, 41] gpg: keybox '/builds/root/spack-env/spack/opt/spack/gpg/pubring.kbx' created
gpg: /builds/root/spack-env/spack/opt/spack/gpg/trustdb.gpg: trustdb created

==> [2020-03-09-21:46:21.827461, 41] spack gpg trust /tmp/tmplz5udno1/signing_key
==> [2020-03-09-21:46:21.827614, 41] gpg: key 0D6A53AC813F980B: public key "Spack Build Pipeline (Demo Key) <[email protected]>" imported
gpg: key 0D6A53AC813F980B: "Spack Build Pipeline (Demo Key) <[email protected]>" not changed
gpg: key 0D6A53AC813F980B: secret key imported
gpg: Total number processed: 2
gpg:               imported: 1
gpg:              unchanged: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1

==> [2020-03-09-21:46:21.848430, 41] spack gpg list --trusted
==> [2020-03-09-21:46:21.848570, 41] /builds/root/spack-env/spack/opt/spack/gpg/pubring.kbx
------------------------------------------------------
pub   rsa2048 2020-03-06 [SCEA]
      81397AAD7D0FDBBB792738E30D6A53AC813F980B
uid           [ unknown] Spack Build Pipeline (Demo Key) <[email protected]>
sub   rsa2048 2020-03-06 [SEA]


==> [2020-03-09-21:46:21.848621, 41] spack gpg list --signing
==> [2020-03-09-21:46:21.848659, 41] /builds/root/spack-env/spack/opt/spack/gpg/pubring.kbx
------------------------------------------------------
pub   rsa2048 2020-03-06 [SCEA]
      81397AAD7D0FDBBB792738E30D6A53AC813F980B
uid           [ unknown] Spack Build Pipeline (Demo Key) <[email protected]>
sub   rsa2048 2020-03-06 [SEA]

/builds/root/spack-env/spack/opt/spack/gpg/pubring.kbx
------------------------------------------------------
sec   rsa2048 2020-03-06 [SCEA]
      81397AAD7D0FDBBB792738E30D6A53AC813F980B
uid           [ unknown] Spack Build Pipeline (Demo Key) <[email protected]>
ssb   rsa2048 2020-03-06 [SEA]


==> [2020-03-09-21:46:21.848724, 41] No compiler action to be taken
==> [2020-03-09-21:46:21.967306, 41] Reading config file /builds/root/spack-env/spack/etc/spack/defaults/repos.yaml
==> [2020-03-09-21:46:21.969980, 41] Reading config file /builds/root/spack-env/spack.yaml
==> [2020-03-09-21:46:22.035754, 41] Here is the concrete spec: [email protected]%[email protected]+multibyte arch=linux-ubuntu18.04-x86_64
==> [2020-03-09-21:46:22.037793, 41] Done writing concrete spec
==> [2020-03-09-21:46:22.037945, 41] Wrote spec file, read it back.  Contents:
==> [2020-03-09-21:46:22.038002, 41] spec:
- pcre2:
    version: '10.31'
    arch:
      platform: linux
      platform_os: ubuntu18.04
      target: x86_64
    compiler:
      name: gcc
      version: 7.4.0
    namespace: builtin
    parameters:
      multibyte: true
      cflags: []
      cppflags: []
      cxxflags: []
      fflags: []
      ldflags: []
      ldlibs: []
    hash: ikxxwcljqbf3r67ac55j6rdecsk6722o
    build_hash: ikxxwcljqbf3r67ac55j6rdecsk6722o

==> [2020-03-09-21:46:22.136023, 41] Checking pcre2-10.31, dag_hash = ikxxwcljqbf3r67ac55j6rdecsk6722o, full_hash = j6q3mqrfbrf3whyqjf6aqwt5jvvfplth
==> [2020-03-09-21:46:22.136396, 41] [email protected]%[email protected]+multibyte arch=linux-ubuntu18.04-x86_64

==> [2020-03-09-21:46:26.631257, 41] Error: Unable to determine whether [email protected]%[email protected]+multibyte arch=linux-ubuntu18.04-x86_64/ikxxwcl needs rebuilding, caught exception attempting to read from s3://spack-public/mirror/build_cache/linux-ubuntu18.04-x86_64-gcc-7.4.0-pcre2-10.31-ikxxwcljqbf3r67ac55j6rdecsk6722o.spec.yaml.
==> [2020-03-09-21:46:26.631417, 41] SpackWebError: Download failed: <urlopen error An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.>
==> [2020-03-09-21:46:26.631550, 41] Warning: Package ([email protected]%[email protected]+multibyte arch=linux-ubuntu18.04-x86_64/ikxxwcl) will be rebuilt
==> [2020-03-09-21:46:26.634046, 41] '/builds/root/spack-env/spack/bin/spack' 'mirror' 'list'
mirror    s3://spack-public/mirror
==> [2020-03-09-21:46:26.910316, 41] listing spack mirrors:
==> [2020-03-09-21:46:26.910608, 41] None
==> [2020-03-09-21:46:26.910667, 41] Registering build with CDash
==> [2020-03-09-21:46:26.910755, 41] Registering cdash build to http://cdash/api/v1/addBuild.php, payload:
==> [2020-03-09-21:46:26.910849, 41] {'project': 'spack', 'site': 'Docker-Compose Testing Framework', 'name': '[email protected]%[email protected] arch=linux-ubuntu18.04-x86_64 (Couple small-ish packages)', 'stamp': '20200309-2146-Couple small-ish packages'}
Traceback (most recent call last):
  File "/builds/root/spack-env/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/builds/root/spack-env/spack/lib/spack/spack/main.py", line 763, in main
    return _invoke_command(command, parser, args, unknown)
  File "/builds/root/spack-env/spack/lib/spack/spack/main.py", line 488, in _invoke_command
    return_val = command(parser, args)
  File "/builds/root/spack-env/spack/lib/spack/spack/cmd/ci.py", line 482, in ci
    args.func(args)
  File "/builds/root/spack-env/spack/lib/spack/spack/cmd/ci.py", line 395, in ci_rebuild
    cdash_site, job_spec_buildgroup)
  File "/builds/root/spack-env/spack/lib/spack/spack/ci.py", line 854, in register_cdash_build
    response = opener.open(request)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized

@scottwittenburg
Copy link
Collaborator

There's something in the README.md (in the CDash section) about unchecking "Authenticate Submissions". Can you double-check that?

The error you mentioned about not being able to find the object is just printing out the error that lead to not being able to find that particular spec.yaml on the remote mirror, and the result of it is just that we'll assume that spec needs to be rebuilt.

@frankwillmore
Copy link
Member Author

I had to go back and check 'public dashboard', authenticate submissions was unchecked. CDash shows only the job that doesn't build (pcre2-10.31).

How do I view what what built? There's no longer directions for checking builds via the nginx portal. I started the nginx container but it just shows an empty directory.

Also, still reporting failure connecting to minio:

==> [2020-03-09-22:37:34.672498, 43] '/usr/bin/gpg2' '--list-secret-keys' '--with-colons' '--fingerprint' '--fingerprint'
==> [2020-03-09-22:37:34.686516, 43] '/usr/bin/gpg2' '--detach-sign' '--armor' '--default-key' '81397AAD7D0FDBBB792738E30D6A53AC813F980B' '--output' '/tmp/tmpy4n6x61b/build_cache/linux-ubuntu18.04-x86_64-gcc-7.4.0-pcre2-10.31-ikxxwcljqbf3r67ac55j6rdecsk6722o.spec.yaml.asc' '/tmp/tmpy4n6x61b/build_cache/linux-ubuntu18.04-x86_64-gcc-7.4.0-pcre2-10.31-ikxxwcljqbf3r67ac55j6rdecsk6722o.spec.yaml'
gpg: using "81397AAD7D0FDBBB792738E30D6A53AC813F980B" as default secret key for signing
==> [2020-03-09-22:37:28.254419, 43] Buildcache files will be output to s3://spack-public/mirror/build_cache
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/boto3/s3/transfer.py", line 279, in upload_file
    future.result()
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builds/root/spack-env/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/builds/root/spack-env/spack/lib/spack/spack/main.py", line 763, in main
    return _invoke_command(command, parser, args, unknown)
  File "/builds/root/spack-env/spack/lib/spack/spack/main.py", line 488, in _invoke_command
    return_val = command(parser, args)
  File "/builds/root/spack-env/spack/lib/spack/spack/cmd/ci.py", line 482, in ci
    args.func(args)
  File "/builds/root/spack-env/spack/lib/spack/spack/cmd/ci.py", line 448, in ci_rebuild
    remote_mirror_url, cdash_build_id)
  File "/builds/root/spack-env/spack/lib/spack/spack/ci.py", line 952, in push_mirror_contents
    False)
  File "/builds/root/spack-env/spack/lib/spack/spack/cmd/buildcache.py", line 399, in _createtarball
    not no_rebuild_index)
  File "/builds/root/spack-env/spack/lib/spack/spack/binary_distribution.py", line 432, in build_tarball
    spackfile_path, remote_spackfile_path, keep_original=False)
  File "/builds/root/spack-env/spack/lib/spack/spack/util/web.py", line 221, in push_to_url
    remote_path, ExtraArgs=extra_args)
  File "/usr/local/lib/python3.6/dist-packages/boto3/s3/inject.py", line 131, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "/usr/local/lib/python3.6/dist-packages/boto3/s3/transfer.py", line 287, in upload_file
    filename, '/'.join([bucket, key]), e))
boto3.exceptions.S3UploadFailedError: Failed to upload /tmp/tmpy4n6x61b/build_cache/linux-ubuntu18.04-x86_64/gcc-7.4.0/pcre2-10.31/linux-ubuntu18.04-x86_64-gcc-7.4.0-pcre2-10.31-ikxxwcljqbf3r67ac55j6rdecsk6722o.spack to spack-public/mirror/build_cache/linux-ubuntu18.04-x86_64/gcc-7.4.0/pcre2-10.31/linux-ubuntu18.04-x86_64-gcc-7.4.0-pcre2-10.31-ikxxwcljqbf3r67ac55j6rdecsk6722o.spack: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

It seems to only be affecting pcre2-10.31. I edited and re-ran my-vars.sh and restarted minio and gitlab with the values for minio but nothing changed. All of the stage one jobs finish except for pcre2-10.31.

@scottwittenburg
Copy link
Collaborator

I had to go back and check 'public dashboard', authenticate submissions was unchecked.

Ah right, I forgot which option it was. But at least the instructions mentioned it.

CDash shows only the job that doesn't build (pcre2-10.31).

CDash should show you all the jobs. Could some jobs have finished successfully before CDash reporting got enabled or started working? Generally speaking, if the pipeline runs and a spec is already up to date in the mirror, it doesn't need to be rebuilt, and in that case nothing will get reported to CDash.

How do I view what what built? There's no longer directions for checking builds via the nginx portal. I started the nginx container but it just shows an empty directory.

Nothing is getting pushed into the file system mirror anymore, so the nginx container won't help.
Browse to the minio server (see the README.md part where you added a bucket). From there you should be able to click on your bucket, then follow links to the buildcache.

Also, still reporting failure connecting to minio:

==> [2020-03-09-22:37:34.672498, 43] '/usr/bin/gpg2' '--list-secret-keys' '--with-colons' '--fingerprint' '--fingerprint'
==> [2020-03-09-22:37:34.686516, 43] '/usr/bin/gpg2' '--detach-sign' '--armor' '--default-key' '81397AAD7D0FDBBB792738E30D6A53AC813F980B' '--output' '/tmp/tmpy4n6x61b/build_cache/linux-ubuntu18.04-x86_64-gcc-7.4.0-pcre2-10.31-ikxxwcljqbf3r67ac55j6rdecsk6722o.spec.yaml.asc' '/tmp/tmpy4n6x61b/build_cache/linux-ubuntu18.04-x86_64-gcc-7.4.0-pcre2-10.31-ikxxwcljqbf3r67ac55j6rdecsk6722o.spec.yaml'
gpg: using "81397AAD7D0FDBBB792738E30D6A53AC813F980B" as default secret key for signing
==> [2020-03-09-22:37:28.254419, 43] Buildcache files will be output to s3://spack-public/mirror/build_cache
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/boto3/s3/transfer.py", line 279, in upload_file
    future.result()
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builds/root/spack-env/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/builds/root/spack-env/spack/lib/spack/spack/main.py", line 763, in main
    return _invoke_command(command, parser, args, unknown)
  File "/builds/root/spack-env/spack/lib/spack/spack/main.py", line 488, in _invoke_command
    return_val = command(parser, args)
  File "/builds/root/spack-env/spack/lib/spack/spack/cmd/ci.py", line 482, in ci
    args.func(args)
  File "/builds/root/spack-env/spack/lib/spack/spack/cmd/ci.py", line 448, in ci_rebuild
    remote_mirror_url, cdash_build_id)
  File "/builds/root/spack-env/spack/lib/spack/spack/ci.py", line 952, in push_mirror_contents
    False)
  File "/builds/root/spack-env/spack/lib/spack/spack/cmd/buildcache.py", line 399, in _createtarball
    not no_rebuild_index)
  File "/builds/root/spack-env/spack/lib/spack/spack/binary_distribution.py", line 432, in build_tarball
    spackfile_path, remote_spackfile_path, keep_original=False)
  File "/builds/root/spack-env/spack/lib/spack/spack/util/web.py", line 221, in push_to_url
    remote_path, ExtraArgs=extra_args)
  File "/usr/local/lib/python3.6/dist-packages/boto3/s3/inject.py", line 131, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "/usr/local/lib/python3.6/dist-packages/boto3/s3/transfer.py", line 287, in upload_file
    filename, '/'.join([bucket, key]), e))
boto3.exceptions.S3UploadFailedError: Failed to upload /tmp/tmpy4n6x61b/build_cache/linux-ubuntu18.04-x86_64/gcc-7.4.0/pcre2-10.31/linux-ubuntu18.04-x86_64-gcc-7.4.0-pcre2-10.31-ikxxwcljqbf3r67ac55j6rdecsk6722o.spack to spack-public/mirror/build_cache/linux-ubuntu18.04-x86_64/gcc-7.4.0/pcre2-10.31/linux-ubuntu18.04-x86_64-gcc-7.4.0-pcre2-10.31-ikxxwcljqbf3r67ac55j6rdecsk6722o.spack: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

It seems here maybe the AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY aren't set correctly in your gitlab CI variables? Those are the S3 authentication credentials the pipeline will need to communicate with S3. Also, because this minio instance is not hosted by AWS, you also need to set S3_ENDPOINT_URL. See the README.md for the details and the values you need to set each one to. Unless you changed the defaults in the my-vars.sh file, the defaults for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY should be minio and minio123 respectively. Unless you used different values for your bucket name, you should set the S3_ENDPOINT_URL gitlab CI env var to "http://minio:10080", I think. Again, see the README.md to be sure.

It seems to only be affecting pcre2-10.31. I edited and re-ran my-vars.sh and restarted minio and gitlab with the values for minio but nothing changed. All of the stage one jobs finish except for pcre2-10.31.

If you want to trigger everything to build again from scratch (and hopefully see all the results in CDash now that it's configured correctly), you can just delete everything from your binary mirror and run the pipeline again. There's an option in the minio interface to remove stuff, just remove the entire build_cache directory. Then click on "Pipelines" in the gitlab interface and click the "Run Pipeline" button near the upper right. On the page you see there, choose to run a pipeline against your master branch. At least that's the branch you were pushing your spack.yaml to earlier, so I'm assuming that hasn't changed. Because no specs will then be found up to date in the mirror, everything should be rebuilt from source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tgamblin @frankwillmore @opadron @scottwittenburg