Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

r2d-action broken? #20

Open
sgibson91 opened this issue Mar 8, 2022 · 4 comments · Fixed by #21
Open

r2d-action broken? #20

sgibson91 opened this issue Mar 8, 2022 · 4 comments · Fixed by #21
Labels
type: bug Something isn't working properly

Comments

@sgibson91
Copy link
Member

Referencing PR #19

  • The image build succeeded in the PR
  • But it fails on push to main

I haven't tracked down the actual error message yet since the logs from the r2d-action are very long and verbose.

Is there a known issue regarding not being able to push this image to quay.io from CI/CD?

cc: @yuvipanda @damianavila
@sgibson91 sgibson91 added the type: bug Something isn't working properly label Mar 8, 2022
@sgibson91
Copy link
Member Author

Ok, I found some logs by using the raw logs instead of the endless scrolling in the GitHub Actions UI

2022-03-08T12:07:08.7444889Z Traceback (most recent call last):
2022-03-08T12:07:08.7446166Z   File "/usr/lib/python3.9/site-packages/urllib3/response.py", line 438, in _error_catcher
2022-03-08T12:07:08.7469113Z     yield
2022-03-08T12:07:08.7469935Z   File "/usr/lib/python3.9/site-packages/urllib3/response.py", line 519, in read
2022-03-08T12:07:08.7474051Z     data = self._fp.read(amt) if not fp_closed else b""
2022-03-08T12:07:08.7474426Z   File "/usr/lib/python3.9/http/client.py", line 462, in read
2022-03-08T12:07:08.7499803Z     n = self.readinto(b)
2022-03-08T12:07:08.7500550Z   File "/usr/lib/python3.9/http/client.py", line 496, in readinto
2022-03-08T12:07:08.7500870Z     return self._readinto_chunked(b)
2022-03-08T12:07:08.7501183Z   File "/usr/lib/python3.9/http/client.py", line 591, in _readinto_chunked
2022-03-08T12:07:08.7501488Z     chunk_left = self._get_chunk_left()
2022-03-08T12:07:08.7501791Z   File "/usr/lib/python3.9/http/client.py", line 559, in _get_chunk_left
2022-03-08T12:07:08.7505064Z     chunk_left = self._read_next_chunk_size()
2022-03-08T12:07:08.7505433Z   File "/usr/lib/python3.9/http/client.py", line 519, in _read_next_chunk_size
2022-03-08T12:07:08.7505735Z     line = self.fp.readline(_MAXLINE + 1)
2022-03-08T12:07:08.7506032Z   File "/usr/lib/python3.9/socket.py", line 704, in readinto
2022-03-08T12:07:08.7510496Z     return self._sock.recv_into(b)
2022-03-08T12:07:08.7511385Z socket.timeout: timed out
2022-03-08T12:07:08.7511555Z 
2022-03-08T12:07:08.7511710Z During handling of the above exception, another exception occurred:
2022-03-08T12:07:08.7511913Z 
2022-03-08T12:07:08.7512026Z Traceback (most recent call last):
2022-03-08T12:07:08.7512523Z   File "/usr/bin/jupyter-repo2docker", line 8, in <module>
2022-03-08T12:07:08.7518867Z     sys.exit(main())
2022-03-08T12:07:08.7519368Z   File "/usr/lib/python3.9/site-packages/repo2docker/__main__.py", line 418, in main
2022-03-08T12:07:08.7525546Z     r2d.start()
2022-03-08T12:07:08.7526020Z   File "/usr/lib/python3.9/site-packages/repo2docker/app.py", line 834, in start
2022-03-08T12:07:08.7533417Z     self.push_image()
2022-03-08T12:07:08.7533890Z   File "/usr/lib/python3.9/site-packages/repo2docker/app.py", line 533, in push_image
2022-03-08T12:07:08.7534252Z     for chunk in client.push(self.output_image_spec):
2022-03-08T12:07:08.7534721Z   File "/usr/lib/python3.9/site-packages/docker/api/client.py", line 349, in _stream_helper
2022-03-08T12:07:08.7548670Z     data = reader.read(1)
2022-03-08T12:07:08.7549617Z   File "/usr/lib/python3.9/site-packages/urllib3/response.py", line 541, in read
2022-03-08T12:07:08.7550221Z     raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
2022-03-08T12:07:08.7561654Z   File "/usr/lib/python3.9/contextlib.py", line 137, in __exit__
2022-03-08T12:07:08.7562069Z     self.gen.throw(typ, value, traceback)
2022-03-08T12:07:08.7562847Z   File "/usr/lib/python3.9/site-packages/urllib3/response.py", line 443, in _error_catcher
2022-03-08T12:07:08.7563433Z     raise ReadTimeoutError(self._pool, None, "Read timed out.")
2022-03-08T12:07:08.7564649Z urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

@damianavila
Copy link
Contributor

damianavila commented Mar 8, 2022 •
edited
Loading

I faced some timeouts using r2d in the past and this is why I ended up pushing with docker on this one: #9.

I think we should be able to pass a timeout value now: jupyterhub/repo2docker#711, but I am not sure if this was released and propagated to the action.

@yuvipanda, since you implemented the fix for the above r2d issue, do you have any more context here?

yuvipanda added a commit that referenced this issue Mar 9, 2022
@yuvipanda
Push built image manually
9f9025c 
Temporary, until we can set the push timeout in r2d properly

Fixes #20
@yuvipanda yuvipanda mentioned this issue Mar 9, 2022
Merged
@yuvipanda yuvipanda reopened this Mar 9, 2022
yuvipanda added a commit that referenced this issue Mar 9, 2022
@yuvipanda
Use r2d action from a PR
7f22d30 
Try out action from jupyterhub/repo2docker-action#84
to see if that fixes our pushing problem

Ref #20
@yuvipanda yuvipanda mentioned this issue Mar 9, 2022
Merged
yuvipanda added a commit that referenced this issue Mar 9, 2022
@yuvipanda
Login to quay.io before pushing
1858778 
Ref #20
@yuvipanda yuvipanda mentioned this issue Mar 9, 2022
Merged
@yuvipanda
Copy link
Member

@sgibson91 So #21, #22, #23 have fixed this, and I've submitted jupyterhub/repo2docker-action#84 upstream as well. This is still temporary though, will need to finish up the 'real' fix which would be to pass a larger timeout to the repo2docker action itself.

@sgibson91
Copy link
Member Author

Thank you for your hard work on this @yuvipanda!

yuvipanda added a commit to yuvipanda/repo2docker-action that referenced this issue Mar 11, 2022
@yuvipanda
Use docker push to push image
42a88f1 
docker push has a couple of advantages over --push in
repo2docker:

- It times out sometimes, causing issues where the image just
  fails to push after building. docker push doesn't have this
  issue - see 2i2c-org/utoronto-image#20,
  where we had to manually run a docker push.
- docker push provides more useful output than --push

This commit also removes some code duplication - we now build
the image regardless, and use the flag just to differentiate pushing.
yuvipanda added a commit to yuvipanda/repo2docker-action that referenced this issue Mar 11, 2022
@yuvipanda
Use docker push to push image
cce9044 
docker push has a couple of advantages over --push in
repo2docker:

- It times out sometimes, causing issues where the image just
  fails to push after building. docker push doesn't have this
  issue - see 2i2c-org/utoronto-image#20,
  where we had to manually run a docker push.
- docker push provides more useful output than --push

This commit also removes some code duplication - we now build
the image regardless, and use the flag just to differentiate pushing.
An unused bash function was also deleted.
@yuvipanda yuvipanda mentioned this issue Mar 11, 2022
Merged
@rgaiacs rgaiacs mentioned this issue Aug 28, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type: bug Something isn't working properly
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Push built image manually 2i2c-org/utoronto-image
3 participants
@yuvipanda @damianavila @sgibson91