openpilot process replay logs lfs storage

DRAFT

Openpilot allows recording process replay reference logs for master-based custom forks (https://github.com/commaai/openpilot/blob/master/selfdrive/test/process_replay/README.md)

This guide is for setting up git lfs backend repo to store openpilot models and process replay logs (and to avoid github lfs quota limits).

• Create new git repo on gitlab for lfs storage backend (eg openpilot-lfs) and set up access using ssh key
• Add upstream repo, fetch lfs objects and push them to your own lfs backend repo

# add upstream repo
git remote add upstream https://github.com/commaai/openpilot

# fetch lfs objects referenced by upstream master branch
git fetch upstream
git lfs install
git lfs fetch --all upstream

# switch lfs backend
git config -f .lfsconfig lfs.url https://gitlab.com/martinlillepuu/openpilot-lfs.git/info/lfs
git config -f .lfsconfig lfs.pushurl ssh://[email protected]/martinlillepuu/openpilot-lfs.git
git lfs update

# push local lfs objects
git lfs push origin subaru-community-c2-2022-06-16

# push all upstream objects
git lfs push --all upstream

# add process replay logs to git lfs tracking
git lfs track "*.bz2"
git commit -am 'Update git lfs config'

Run process replay tests (only works on Ubuntu)

export PYTHONPATH=$PWD
pipenv shell
pipenv install
scons -i
git rm selfdrive/test/process_replay/fakedata/*.bz2
FILEREADER_CACHE='1' ./selfdrive/test/process_replay/test_processes.py
../update-ref-commit.sh
# add new reference process replay logs to git
git add -f selfdrive/test/process_replay/fakedata/*.bz2

# for c2 branches use
# ./selfdrive/test/process_replay/update_refs.py --no-upload
# git add -f selfdrive/test/process_replay/*.bz2

# check git lfs
git lfs status
# commit and push
git commit -am 'Add process replay reference logs, update ref commit'
git push

Diff for adding git lfs pull command to github process replay workflow

--- a/.github/workflows/selfdrive_tests.yaml
+++ b/.github/workflows/selfdrive_tests.yaml
@@ -321,6 +321,8 @@ jobs:
     - uses: actions/checkout@v3
       with:
         submodules: true
+    - name: Pull LFS
+      run: git lfs pull
     - name: Cache dependencies
       id: dependency-cache
       uses: actions/cache@v2

Scripts

update-ref-commit.sh

ref_commit=$(git rev-parse HEAD)
  
echo -n $ref_commit > selfdrive/test/process_replay/ref_commit

Update lfs objects after merge

# switch to upstream lfs backend
git config -f .lfsconfig lfs.url https://gitlab.com/commaai/openpilot-lfs.git/info/lfs
git config -f .lfsconfig lfs.pushurl ssh://[email protected]/commaai/openpilot-lfs.git
git lfs update
git lfs fetch --all upstream

# switch to local lfs backend
git config -f .lfsconfig lfs.url https://gitlab.com/martinlillepuu/openpilot-lfs.git/info/lfs
git config -f .lfsconfig lfs.pushurl ssh://[email protected]/martinlillepuu/openpilot-lfs.git
git lfs update

# push all upstream master objects
git lfs push --all upstream master
git lfs push --all origin

References:

https://www.atlassian.com/git/tutorials/git-lfs#creating-new-repository