Skip to content

Commit

Permalink
Fixup
Browse files Browse the repository at this point in the history
  • Loading branch information
mwestphal committed Sep 4, 2024
1 parent 810d595 commit 0e17b0a
Show file tree
Hide file tree
Showing 3 changed files with 36 additions and 46 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,6 @@ jobs:
cache_postfix: 'ci-cache'
target_directory: 'output_dir'

- name: Checkout output directory
- name: Check output has expected size
shell: bash
run: ls output_dir/
run: [ `cat output_dir/testing/data/f3d.vtp | wc -c` == 4104 ]
50 changes: 20 additions & 30 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,22 @@
A producer/consumer github action to cache and recover LFS data.

Its main use is to avoid cloning LFS data in CI to avoid
haing to pay for LFS bandwidth because of CI needs.
It needs cmake to be available on the host.
having to pay for LFS bandwidth because of CI needs.

The action can be used as a consumer or a producer, and needs
a SHA to recover LFS data with.
It expects cmake to be available on the host.

The action can be used as a consumer or a producer, and must
provide the repository containing the LFS data to recover from.

It is possible to provide a specific SHA to produce.
If not provided, the last commit modyfing the LFS file will be produced.

Is has the following inputs:

- `type`: should be `producer` or `consumer`, default producer
- `repository`: the git repository to recover LFS data from, required
- `lfs_sha`: The git sha to recover LFS data from, required
- `cache_index`: An index used in the cache name, default is 0
- `type`: should be `producer` or `consumer`, default to `producer`
- `repository`: the git repository to produce LFS data from, default: ${{ github.repository }}
- `lfs_sha`: The git sha to recover LFS data from, optional
- `cache_postfix`: An postfix added to the cache name, to support multiple caches, default to `cache`
- `target_directory`: A target directory to copy LFS data to

## Logic
Expand All @@ -34,8 +38,9 @@ Finally, Producer/Consumer will copy the LFS data only using cmake to the `targe

## Usage

In an first job, recover the LFS sha to recover and use the `producer` action, output the LFS sha
In a second job depending on the first, recover the LFS sha from first job and use the `consumer` action.
In an first job, use the `producer` action, which output the LFS sha that will be produced
In a second job, usually a matrix joib, depending on the first,
recover the LFS sha from first job and use the `consumer` action.

```
jobs:
Expand All @@ -50,35 +55,22 @@ jobs:
lfs_sha: ${{ steps.lfs_sha_recover.outputs.lfs_sha }}
steps:
# Checkout WITHOUT LFS as the data itself is not needed at this point
# Checkout your repository WITHOUT LFS
- name: Checkout
uses: actions/checkout@v3
with:
path: 'source'
fetch-depth: 0
lfs: false
# Recover the last time LFS data was changed on the repository
# TODO: Update the list of directory you want to watch for changes
- name: Set LFS env var
working-directory: ${{github.workspace}}/source
id: lfs_sha_recover
shell: bash
run: echo "lfs_sha=$(git log -n 1 --pretty=format:%H -- path/to/lfs/data path/to/lfs/data/again)" >> $GITHUB_OUTPUT
# Use producer action to recover the LFS data and upload it as cache/artifacts
- name: Cache LFS Data
uses: f3d-app/lfs-data-cache-action:latest
with:
workflow_label: 'producer'
repository: 'your/repo'
lfs_sha: ${{ steps.lfs_sha_recover.outputs.lfs_sha }}
target_directory: 'source'
uses: f3d-app/lfs-data-cache-action:v1
recover_lfs:
needs: cache_lfs
# Checkout WITHOUT LFS as the data itself is not needed at this point
# Checkout your repository WITHOUT LFS
- name: Checkout
uses: actions/checkout@v3
with:
Expand All @@ -87,10 +79,8 @@ jobs:
lfs: false
- name: Recover LFS Data
uses: f3d-app/lfs-data-cache-action:latest
uses: f3d-app/lfs-data-cache-action:v1
with:
workflow_label: 'consumer'
repository: 'your/repo'
lfs_sha: ${{ inputs.lfs_sha}}
target_directory: 'source'
lfs_sha: ${{ needs.cache_lfs.outputs.lfs_sha}}
```
28 changes: 14 additions & 14 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ inputs:
default: 'producer'
repository:
description: 'Repository to recover LFS data from'
required: true
required: false
default: ${{ github.repository }}
lfs_sha:
description: 'LFS sha to recover. If not set, target directory will be used to recover it.'
required: false
Expand All @@ -33,38 +34,37 @@ runs:
- name: Create a working directory
working-directory: ${{github.workspace}}
shell: bash
run: mkdir lfs-data-cache
run: mkdir lfs_data_cache

- name: Checkout repository without LFS data
if: inputs.lfs_sha == ''
uses: actions/checkout@v4
with:
repository: ${{ inputs.repository }}
path: 'lfs-data-cache/lfs_source'
path: 'lfs_data_cache/lfs_source'
fetch-depth: 0
lfs: false

- name: Set LFS sha env var without inputs
- name: Set LFS sha env var from repository
if: inputs.lfs_sha == ''
working-directory: ${{github.workspace}}/lfs-data-cache/lfs_source
working-directory: ${{github.workspace}}/lfs_data_cache/lfs_source
shell: bash
run: echo "lfs_data_cache_sha=$(git log -n 1 --pretty=format:%H -- `git-lfs ls-files -n`)" >> $GITHUB_ENV

- name: Set LFS sha env var with inputs
- name: Set LFS sha env var from inputs
if: inputs.lfs_sha != ''
working-directory: ${{github.workspace}}/lfs-data-cache/lfs_source
shell: bash
run: echo "lfs_data_cache_sha=${{inputs.lfs_sha}}" >> $GITHUB_ENV

- name: Cache LFS data
id: cache-lfs
uses: actions/cache@v4
with:
path: 'lfs-data-cache/lfs_data'
path: 'lfs_data_cache/lfs_data'
key: lfs-data-${{env.lfs_data_cache_sha}}-${{inputs.cache_postfix}}

- name: Set LFS output sha
working-directory: ${{github.workspace}}/lfs-data-cache/lfs_source
working-directory: ${{github.workspace}}/lfs_data_cache/lfs_source
id: lfs_sha_recover
shell: bash
run: echo "lfs_sha=${{env.lfs_data_cache_sha}}" >> $GITHUB_OUTPUT
Expand All @@ -77,7 +77,7 @@ runs:
with:
repository: ${{ inputs.repository }}
ref: ${{env.lfs_data_cache_sha}}
path: 'lfs-data-cache/lfs_data'
path: 'lfs_data_cache/lfs_data'
fetch-depth: 0
lfs: true

Expand All @@ -88,7 +88,7 @@ runs:
uses: actions/upload-artifact@v4
with:
name: lfs-data-${{inputs.cache_postfix}}
path: 'lfs-data-cache/lfs_data'
path: 'lfs_data_cache/lfs_data'
overwrite: true
include-hidden-files: true

Expand All @@ -101,7 +101,7 @@ runs:
continue-on-error: true
with:
name: lfs-data-${{inputs.cache_postfix}}
path: 'lfs-data-cache/lfs_data'
path: 'lfs_data_cache/lfs_data'

- name: Checkout LFS data (last resort)
if: |
Expand All @@ -112,11 +112,11 @@ runs:
with:
repository: ${{ inputs.repository }}
ref: ${{env.lfs_data_cache_sha}}
path: 'lfs-data-cache/lfs_data'
path: 'lfs_data_cache/lfs_data'
fetch-depth: 0
lfs: true

- name: Setup LFS data
working-directory: ${{github.workspace}}
shell: bash
run: cmake -P $GITHUB_ACTION_PATH/copy_lfs.cmake 'lfs-data-cache/lfs_data' ${{ inputs.target_directory }}
run: cmake -P $GITHUB_ACTION_PATH/copy_lfs.cmake 'lfs_data_cache/lfs_data' ${{ inputs.target_directory }}

0 comments on commit 0e17b0a

Please sign in to comment.