Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

figure out dependency caching #110

Closed
maxheld83 opened this issue May 1, 2019 · 4 comments
Closed

figure out dependency caching #110

maxheld83 opened this issue May 1, 2019 · 4 comments
Labels
byod bring your own dockerfile; design where every project gets a custom docker BUILD docker anything docker related github Related to / requiring support from github (actions) pkg-dev stuff to help other package developers

Comments

@maxheld83
Copy link
Owner

maxheld83 commented May 1, 2019

currently, we've always specified R-dependencies (packages) by hand in the byod-Dockerfile.
It's arguably good to get rid of DESCRIPTIONs for Rmd projects, where it was always a little awkward and necessary only because of Travis.

But for pkg dev, we must install the dependencies from DESCRIPTION (not Dockerfile).

For some packages, installing all the dependencies can be pretty expensive, so the question is how to make this fast.

Right now, I see two options to do this:

  1. Every package uses the same docker image as per provide good docker image for pkg devel #106.
    We then invoke remotes:: before, say, the build action, to get all the necessary dependencies.
    This takes a while, so we need caching (travis had this out-of-the-box).
    We can either:
  • Wait for GitHub to support this natively migrate to native caching #48
  • In the meantime, cache somewhere else (probably a bad idea because of network ingress issues etc.).
  1. We actually build a new docker image for every package, and bake the current dependencies into it, by running remotes:: inside the Dockerfile.
    This then again requires the somewhat awkward first step of building a custom docker image for every pkg repo, from which follows an even more awkward way to load these images downstream see find a more elegant way to identify docker images from past docker build . actions #1 (haha! this is an old one!).
    The upside is that I think this gives us caching out of the box, b/c GitHub seems to natively cache docker build actions already. (Need to double check this as per cache byod images #3) (another early issue!)
@maxheld83 maxheld83 added github Related to / requiring support from github (actions) docker anything docker related pkg-dev stuff to help other package developers labels May 1, 2019
@maxheld83
Copy link
Owner Author

@maxheld83
Copy link
Owner Author

my preliminary guess would be that it's best to just wait until github implements this natively.

I'd be good to have some guidance from the github actions team whether they're actually planning on doing this.

@maxheld83
Copy link
Owner Author

this also seems to confirm that the current github actions position on caching is to cache docker images (as I suspected): https://jasonet.co/posts/use-github-actions-for-ci/

@maxheld83 maxheld83 added the byod bring your own dockerfile; design where every project gets a custom docker BUILD label May 9, 2019
@maxheld83
Copy link
Owner Author

This is ok for now; we're going with 1 in the above and just waiting for native caching. Shouldn't be too bad in the meantime though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
byod bring your own dockerfile; design where every project gets a custom docker BUILD docker anything docker related github Related to / requiring support from github (actions) pkg-dev stuff to help other package developers
Projects
None yet
Development

No branches or pull requests

1 participant