Skip to content

Commit

Permalink
Merge pull request #255 from truenas/truenas/zfs-2.3-sync-with-master
Browse files Browse the repository at this point in the history
Sync truenas/zfs-2.3-release with upstream zfs-2.3-rc2 tag
  • Loading branch information
amotin authored Oct 10, 2024
2 parents eecb1bc + a06e9f4 commit 3b78e86
Show file tree
Hide file tree
Showing 457 changed files with 13,384 additions and 8,996 deletions.
18 changes: 7 additions & 11 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,22 +145,18 @@ Once everything is in good shape and the details have been worked out you can re
Any required reviews can then be finalized and the pull request merged.

#### Tests and Benchmarks
* Every pull request will by tested by the buildbot on multiple platforms by running the [zfs-tests.sh and zloop.sh](
* Every pull request is tested using a GitHub Actions workflow on multiple platforms by running the [zfs-tests.sh and zloop.sh](
https://openzfs.github.io/openzfs-docs/Developer%20Resources/Building%20ZFS.html#running-zloop-sh-and-zfs-tests-sh) test suites.
`.github/workflows/scripts/generate-ci-type.py` is used to determine whether the pull request is nonbehavior, i.e., not introducing behavior changes of any code, configuration or tests. If so, the CI will run on fewer platforms and only essential sanity tests will run. You can always override this by adding `ZFS-CI-Type` line to your commit message:
* If your last commit (or `HEAD` in git terms) contains a line `ZFS-CI-Type: quick`, quick mode is forced regardless of what files are changed.
* Otherwise, if any commit in a PR contains a line `ZFS-CI-Type: full`, full mode is forced.
* To verify your changes conform to the [style guidelines](
https://github.com/openzfs/zfs/blob/master/.github/CONTRIBUTING.md#style-guides
), please run `make checkstyle` and resolve any warnings.
* Static code analysis of each pull request is performed by the buildbot; run `make lint` to check your changes.
* Test cases should be provided when appropriate.
This includes making sure new features have adequate code coverage.
* Code analysis is performed by [CodeQL](https://codeql.github.com/) for each pull request.
* Test cases should be provided when appropriate. This includes making sure new features have adequate code coverage.
* If your pull request improves performance, please include some benchmarks.
* The pull request must pass all required [ZFS
Buildbot](http://build.zfsonlinux.org/) builders before
being accepted. If you are experiencing intermittent TEST
builder failures, you may be experiencing a [test suite
issue](https://github.com/openzfs/zfs/issues?q=is%3Aissue+is%3Aopen+label%3A%22Type%3A+Test+Suite%22).
There are also various [buildbot options](https://openzfs.github.io/openzfs-docs/Developer%20Resources/Buildbot%20Options.html)
to control how changes are tested.
* The pull request must pass all CI checks before being accepted.

### Testing
All help is appreciated! If you're in a position to run the latest code
Expand Down
57 changes: 0 additions & 57 deletions .github/workflows/build-dependencies.txt

This file was deleted.

5 changes: 0 additions & 5 deletions .github/workflows/checkstyle-dependencies.txt

This file was deleted.

23 changes: 14 additions & 9 deletions .github/workflows/checkstyle.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ on:
push:
pull_request:

concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true

jobs:
checkstyle:
runs-on: ubuntu-22.04
Expand All @@ -13,15 +17,11 @@ jobs:
ref: ${{ github.event.pull_request.head.sha }}
- name: Install dependencies
run: |
# https://github.com/orgs/community/discussions/47863
sudo apt-mark hold grub-efi-amd64-signed
sudo apt-get update --fix-missing
sudo apt-get upgrade
sudo xargs --arg-file=${{ github.workspace }}/.github/workflows/build-dependencies.txt apt-get install -qq
sudo xargs --arg-file=${{ github.workspace }}/.github/workflows/checkstyle-dependencies.txt apt-get install -qq
sudo python3 -m pip install --quiet flake8
sudo apt-get clean
# for x in lxd core20 snapd; do sudo snap remove $x; done
sudo apt-get purge -y snapd google-chrome-stable firefox
ONLY_DEPS=1 .github/workflows/scripts/qemu-3-deps.sh ubuntu22
sudo apt-get install -y cppcheck devscripts mandoc pax-utils shellcheck
sudo python -m pipx install --quiet flake8
# confirm that the tools are installed
# the build system doesn't fail when they are not
checkbashisms --version
Expand All @@ -31,8 +31,13 @@ jobs:
shellcheck --version
- name: Prepare
run: |
sed -i '/DEBUG_CFLAGS="-Werror"/s/^/#/' config/zfs-build.m4
./autogen.sh
- name: Configure
run: |
./configure
- name: Make
run: |
make -j$(nproc) --no-print-directory --silent
- name: Checkstyle
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:
run: mv ../*.deb artifacts
if: success()

- uses: actions/upload-artifact@v1
- uses: actions/upload-artifact@v4
with:
name: zfs-native
path: artifacts
Expand Down
10 changes: 7 additions & 3 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ on:
push:
pull_request:

concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true

jobs:
analyze:
name: Analyze
Expand All @@ -27,15 +31,15 @@ jobs:
uses: actions/checkout@v4

- name: Initialize CodeQL
uses: github/codeql-action/init@v2
uses: github/codeql-action/init@v3
with:
config-file: .github/codeql-${{ matrix.language }}.yml
languages: ${{ matrix.language }}

- name: Autobuild
uses: github/codeql-action/autobuild@v2
uses: github/codeql-action/autobuild@v3

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v2
uses: github/codeql-action/analyze@v3
with:
category: "/language:${{matrix.language}}"
14 changes: 14 additions & 0 deletions .github/workflows/scripts/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@

Workflow for each operating system:
- install qemu on the github runner
- download current cloud image of operating system
- start and init that image via cloud-init
- install dependencies and poweroff system
- start system and build openzfs and then poweroff again
- clone build system and start 2 instances of it
- run functional testings and complete in around 3h
- when tests are done, do some logfile preparing
- show detailed results for each system
- in the end, generate the job summary

/TR 14.09.2024
107 changes: 107 additions & 0 deletions .github/workflows/scripts/generate-ci-type.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
#!/usr/bin/env python3

"""
Determine the CI type based on the change list and commit message.
Prints "quick" if (explicity required by user):
- the *last* commit message contains 'ZFS-CI-Type: quick'
or if (heuristics):
- the files changed are not in the list of specified directories, and
- all commit messages do not contain 'ZFS-CI-Type: full'
Otherwise prints "full".
"""

import sys
import subprocess
import re

"""
Patterns of files that are not considered to trigger full CI.
Note: not using pathlib.Path.match() because it does not support '**'
"""
FULL_RUN_IGNORE_REGEX = list(map(re.compile, [
r'.*\.md',
r'.*\.gitignore'
]))

"""
Patterns of files that are considered to trigger full CI.
"""
FULL_RUN_REGEX = list(map(re.compile, [
r'cmd.*',
r'configs/.*',
r'META',
r'.*\.am',
r'.*\.m4',
r'autogen\.sh',
r'configure\.ac',
r'copy-builtin',
r'contrib',
r'etc',
r'include',
r'lib/.*',
r'module/.*',
r'scripts/.*',
r'tests/.*',
r'udev/.*'
]))

if __name__ == '__main__':

prog = sys.argv[0]

if len(sys.argv) != 3:
print(f'Usage: {prog} <head_ref> <base_ref>')
sys.exit(1)

head, base = sys.argv[1:3]

def output_type(type, reason):
print(f'{prog}: will run {type} CI: {reason}', file=sys.stderr)
print(type)
sys.exit(0)

# check last (HEAD) commit message
last_commit_message_raw = subprocess.run([
'git', 'show', '-s', '--format=%B', 'HEAD'
], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

for line in last_commit_message_raw.stdout.decode().splitlines():
if line.strip().lower() == 'zfs-ci-type: quick':
output_type('quick', f'explicitly requested by HEAD commit {head}')

# check all commit messages
all_commit_message_raw = subprocess.run([
'git', 'show', '-s',
'--format=ZFS-CI-Commit: %H%n%B', f'{head}...{base}'
], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
all_commit_message = all_commit_message_raw.stdout.decode().splitlines()

commit_ref = head
for line in all_commit_message:
if line.startswith('ZFS-CI-Commit:'):
commit_ref = line.lstrip('ZFS-CI-Commit:').rstrip()
if line.strip().lower() == 'zfs-ci-type: full':
output_type('full', f'explicitly requested by commit {commit_ref}')

# check changed files
changed_files_raw = subprocess.run([
'git', 'diff', '--name-only', head, base
], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
changed_files = changed_files_raw.stdout.decode().splitlines()

for f in changed_files:
for r in FULL_RUN_IGNORE_REGEX:
if r.match(f):
break
else:
for r in FULL_RUN_REGEX:
if r.match(f):
output_type(
'full',
f'changed file "{f}" matches pattern "{r.pattern}"'
)

# catch-all
output_type('quick', 'no changed file matches full CI patterns')
Loading

0 comments on commit 3b78e86

Please sign in to comment.