keeping built/installed historical versions between atime CI runs #6528
Comments
|
FWIW we can probably install these versions more minimally, e.g.
Just tested this out: R CMD BUILD data.table
cd /tmp
tar -xzf ~/github/data.table_1.16.99.tar.gz
cd data.table
rm -rf NEWS.md build inst man tests vignettes
mkdir inst # for ./configure
R CMD build .
R CMD INSTALL data.table_1.16.99.tar.gzThe |
|
Great ideas! Also, I think we can apply tags to each test case and then filter the test cases that are to be run based on the files changed. (e.g.: if a PR is dealing with modifications to One potential challenge/drawback with this could be in isolating functions appropriately, considering their interactions with each other under the hood. Otherwise this would also help in reducing the horizontal scaling of the overall plot, or to mitigate the 'too many tests' visual issue that we are running into (we are no longer in the comfort zone of checking test cases without clicking the image) |
|
recent atime runs did not install historical versions for most test cases (a bug), and we saw that the time it takes during that step of installation is greatly reduced (Installing different package versions), so that is evidence that keeping build/installed historical versions would actually result in speedups. |
now that we have 10+ performance tests, the amount of time taken by performance testing CI is getting larger, 8min last I checked. I believe most of this time is taken installing different historical/reference versions of data.table, which do not change between runs of the performance testing CI jobs.
in other words, each CI job needs to compile and install a bunch of different git versions of data.table, and some of these versions are different in each run (master, merge-base, HEAD, CRAN), but many are the same (historical versions).
https://github.com/actions/upload-artifact could be used at the end of our CI job, to upload compiled/installed historical versions of data.table, then https://github.com/actions/download-artifact could be used at the begining of each CI job to download previously compiled/installed historical versions of data.table.
There is some trade-off between storage space and Ci run time. I don't think it is currently worth our time to actually implement this, because https://github.com/organizations/Rdatatable/settings/billing/summary says we are nowhere close to our quota (0 usage in all categories, quota resets in 18 days). But one day we may need to consider implementing this.
https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#included-storage-and-minutes says the free plan includes 500 MB, and 2000 minutes per month. I estimate ~10MB per installed historical version of data.table, which means we could get speedups for free up to 50 historical versions. Below we see file sizes of 13 installed historical versions of data.table on my local computer, total ~100MB, so average per version a bit less than ~10MB.
The text was updated successfully, but these errors were encountered: