Error starting recorded run

[craig-willis]

I don't have a repeatable test case, but I have seen this intermittently during testing. It seems that the container has been removed before we hit the wait? Maybe remove=False and container.remove() would fix it?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/girder_worker/task.py", line 148, in __call__
    results = super(Task, self).__call__(*_t_args, **_t_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/gwvolman/gwvolman/tasks.py", line 723, in recorded_run
    tag = image_builder.get_tag()
  File "/gwvolman/gwvolman/build_utils.py", line 126, in get_tag
    ret, output_digest = self.run_r2d(
  File "/gwvolman/gwvolman/build_utils.py", line 207, in run_r2d
    ret = container.wait()
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 512, in wait
    return self.client.api.wait(self.id, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 1304, in wait
    return self._result(res, True)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.28/containers/726c5d9b0849a007f87cad7d4ee33de62e7b1f3463e8d54f2aed2af2f278e690/wait: Not Found ("No such container: 726c5d9b0849a007f87cad7d4ee33de62e7b1f3463e8d54f2aed2af2f278e690")

[craig-willis]

Separate but possibly related issue:

Mounted wt_dms data
Mounted wt_run workspace
Writing the environment to /host/var/lib/docker/volumes/6230a1f50c6c5c6b59785d23_cawillis_3eki1r/_data/workspace/environment.json
Using repo2docker wholetale/repo2docker_wholetale:v1.1
Starting recorded run
Running Tale with command: sh run.sh
Running image: registry.wholetale.org/tale/e2b93af91417e3c348ebaa1b6160caf9:93aec7cd4b1c6ac428bc428c374770f3
ImageNotFound: 404 Client Error for http+docker://localhost/v1.28/containers/create: Not Found ("No such image: registry.wholetale.org/tale/e2b93af91417e3c348ebaa1b6160caf9:93aec7cd4b1c6ac428bc428c374770f3")
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/girder_worker/task.py", line 148, in __call__
    results = super(Task, self).__call__(*_t_args, **_t_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/gwvolman/gwvolman/tasks.py", line 753, in recorded_run
    _recorded_run(image_builder.dh.cli, mountpoint, container_config, tag, entrypoint)
  File "/gwvolman/gwvolman/utils.py", line 334, in _recorded_run
    container = cli.containers.create(
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 878, in create
    resp = self.client.api.create_container(**create_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 428, in create_container
    return self.create_container_from_config(config, name)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 439, in create_container_from_config
    return self._result(res, True)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)

Seems to be that the cli.containers.create is called but the image hasn't been pulled.