Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add container jobs performing chores #31

Merged
merged 11 commits into from
Mar 7, 2025
19 changes: 16 additions & 3 deletions .github/workflows/per_patch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,22 @@ jobs:
autorun:
needs: get_qcow
runs-on: ubuntu-latest
timeout-minutes: 35
timeout-minutes: 60
env:
REPOSITORY_TARBALL_PATH: ${{ github.workspace }}/repository.tar.gz
ABI_TARBALL_PATH: ${{ github.workspace }}/abi.tar.gz
CI_CFGS_REPOSITORY_PATH: ${{ github.workspace }}/ci/cijoe/configs/autorun_configs
strategy:
matrix:
workflow:
- autorun_unittest
- autorun_nvme
- unittest-gcc-container-autotest
- check-format-container-autotest
- release-build-gcc-container-autotest
- doc-container-autotest
- build-files-container-autotest
- scan-build-container-autotest
- check-so-deps-container-autotest
container:
image: ghcr.io/refenv/cijoe-docker:v0.9.50
options: --device=/dev/kvm
Expand All @@ -54,9 +61,15 @@ jobs:
with:
name: repository

- name: Download the SPDK ABI repository
uses: actions/[email protected]
with:
name: abi

- name: Extract the SPDK repository
run: |
tar xzf repository.tar.gz --strip 1
tar xzf abi.tar.gz --strip 1

- name: Download VM Qcow2 image artifact
uses: actions/[email protected]
Expand All @@ -68,7 +81,7 @@ jobs:
- name: qemu-guest, provision
run: |
cd ci/cijoe
cijoe guest_initialize guest_start guest_check tgz_transfer ci_configs_transfer tgz_unpack \
cijoe guest_initialize guest_start guest_check tgz_transfer abi_transfer ci_configs_transfer tgz_unpack output_create \
--monitor \
--config configs/qemuhost-with-guest-fedora-40.toml \
--workflow workflows/autorun_in_qemu.yaml \
Expand Down
17 changes: 16 additions & 1 deletion .github/workflows/spdk-common-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,21 @@ jobs:
with:
name: repository
path: repository.tar.gz

- name: Checkout the spdk-abi repo
uses: actions/[email protected]
with:
repository: 'spdk/spdk-abi'
path: './spdk-abi'
fetch-depth: '0'
fetch-tags: 'true'
- name: Create abi tarball
run: tar -C './spdk-abi' -czf abi.tar.gz .
- name: Upload the abi tarball as an artifact
uses: actions/[email protected]
with:
name: abi
path: abi.tar.gz
outputs:
is_called: ${{ env.is_called }}

Expand All @@ -67,7 +82,7 @@ jobs:
- name: Download artifact tarballs
uses: actions/[email protected]
with:
pattern: 'autorun_*_artifacts'
pattern: '*_artifacts'

- name: Show artifacts
run: |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CHECK_FORMAT_ONLY_DIFF=yes
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
SPDK_RUN_VALGRIND=0
SPDK_TEST_UNITTEST=0
SPDK_TEST_BLOCKDEV=1
SPDK_TEST_ISAL=1
SPDK_TEST_REDUCE=1
SPDK_TEST_VBDEV_COMPRESS=1
SPDK_TEST_CRYPTO=1
SPDK_TEST_FTL=1
SPDK_TEST_OCF=1
SPDK_TEST_RAID=1
SPDK_TEST_RBD=1
SPDK_TEST_URING=1
SPDK_TEST_NVME_CUSE=1
SPDK_TEST_BLOBFS=1
SPDK_TEST_VFIOUSER=1
SPDK_TEST_DAOS=0
SPDK_RUN_ASAN=1
SPDK_RUN_UBSAN=1
SPDK_TEST_XNVME=1
Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
SPDK_TEST_UNITTEST=1
SPDK_TEST_BLOCKDEV=1
SPDK_TEST_PMDK=1
SPDK_TEST_ISAL=1
SPDK_TEST_REDUCE=1
SPDK_TEST_VBDEV_COMPRESS=1
SPDK_TEST_CRYPTO=1
SPDK_TEST_FTL=1
SPDK_TEST_OCF=1
SPDK_TEST_RAID=1
SPDK_TEST_RBD=1
SPDK_TEST_URING=1
SPDK_TEST_NVME_CUSE=1
SPDK_TEST_BLOBFS=1
SPDK_TEST_VFIOUSER=1
SPDK_TEST_XNVME=1
SPDK_TEST_RELEASE_BUILD=1
18 changes: 18 additions & 0 deletions cijoe/configs/autorun_configs/scan-build-container-autotest.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
SPDK_TEST_SCANBUILD=1
SPDK_TEST_BLOCKDEV=1
SPDK_TEST_ISAL=1
SPDK_TEST_REDUCE=1
SPDK_TEST_VBDEV_COMPRESS=1
SPDK_TEST_CRYPTO=1
SPDK_TEST_FTL=1
SPDK_TEST_OCF=1
SPDK_TEST_RAID=1
SPDK_TEST_RBD=1
SPDK_TEST_URING=1
SPDK_TEST_NVME_CUSE=1
SPDK_TEST_BLOBFS=1
SPDK_TEST_VFIOUSER=1
SPDK_TEST_DAOS=0
SPDK_RUN_ASAN=1
SPDK_RUN_UBSAN=1
SPDK_TEST_XNVME=1
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ SPDK_TEST_REDUCE=1
SPDK_TEST_VBDEV_COMPRESS=1
SPDK_TEST_CRYPTO=1
SPDK_TEST_FTL=1
SPDK_TEST_OCF=0
SPDK_TEST_OCF=1
SPDK_TEST_RAID=1
SPDK_TEST_RBD=1
SPDK_TEST_URING=1
Expand Down
44 changes: 39 additions & 5 deletions cijoe/workflows/autorun_in_qemu.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,12 @@ steps:
src: "{{ local.env.REPOSITORY_TARBALL_PATH }}"
dst: /tmp/repository.tar.gz

- name: abi_transfer
uses: core.put
with:
src: "{{ local.env.ABI_TARBALL_PATH }}"
dst: /tmp/abi.tar.gz

- name: ci_configs_transfer
uses: core.put
with:
Expand All @@ -42,17 +48,45 @@ steps:
run: |
mkdir -p /opt/spdk
tar xzf /tmp/repository.tar.gz -C /opt/spdk
git config --global --add safe.directory /opt/spdk
mkdir -p /opt/spdk-abi
tar xzf /tmp/abi.tar.gz -C /opt/spdk-abi
git config --global --add safe.directory '*'

- name: autorun_unittest
- name: output_create
run: |
mkdir -p /opt/output
output_dir=/opt/output /opt/spdk/autorun.sh /opt/configs/unittest.conf

- name: autorun_nvme
run: |
mkdir -p /opt/output
output_dir=/opt/output /opt/spdk/autorun.sh /opt/configs/nvme.conf
/opt/spdk/autorun.sh /opt/configs/nvme.conf

- name: unittest-gcc-container-autotest
run: |
rootdir=/opt/spdk && cd $rootdir && source ./test/common/autobuild_common.sh /opt/configs/unittest-gcc-container-autotest.conf && ocf_precompile && unittest_build && ./test/unit/unittest.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Total nit: break the line a bit?
It's ~190 chars, so a bit unwieldy when displayed in editor with split scree.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see jobs in other commits having similar line length, so applies there as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://cijoe.readthedocs.io/en/latest/workflows/index.html#inline-commands
Each line is separate command intentionally. This implies all the variables being set and workdir has to be in one line.
I could have created separate script that does all of that, but then it goes a bit against the simplicity we wanted here. We are already three 'workflows' deep.


- name: check-format-container-autotest
run: |
rootdir=/opt/spdk && cd $rootdir && source ./test/common/autobuild_common.sh /opt/configs/check-format-container-autotest.conf && check_format

- name: release-build-gcc-container-autotest
run: |
rootdir=/opt/spdk && cd $rootdir && source ./test/common/autobuild_common.sh /opt/configs/release-build-gcc-container-autotest.conf && build_release

- name: doc-container-autotest
run: |
rootdir=/opt/spdk && cd $rootdir && source ./test/common/autobuild_common.sh /opt/configs/empty.conf && build_doc

- name: build-files-container-autotest
run: |
rootdir=/opt/spdk && cd $rootdir && source ./test/common/autobuild_common.sh /opt/configs/empty.conf && build_files && dpdk_pci_api

- name: scan-build-container-autotest
run: |
rootdir=/opt/spdk && cd $rootdir && source ./test/common/autobuild_common.sh /opt/configs/scan-build-container-autotest.conf && ocf_precompile && scanbuild_make

- name: check-so-deps-container-autotest
run: |
rootdir=/opt/spdk && cd $rootdir && source ./test/common/autobuild_common.sh /opt/configs/check-so-deps-container-autotest.conf && ocf_precompile && SPDK_ABI_DIR=/opt/spdk-abi check_so_deps
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a lot of repetitions of this rootdir=/opt/spdk && cd $rootdir && source ./test/common/autobuild_common.sh.

This is ci-joe specific config so I have no idea how to handle that. Using yaml's aliases may reduce the number of repetitions, but overall generates more total lines in this file (lol). Something to clean up later I guess.


- name: output_listing
run: |
Expand Down