Skip to content

Commit

Permalink
Port instrumented testing to python
Browse files Browse the repository at this point in the history
Since an unknown amount of time the instrumented CI has been a bit
flawed, explained here
#5185. It also
experiences random timeout issues where restarting the workflow fixes it
or very long run times (more than other workflows) and is not very
portable.

The intention of this commit is to port the instrumented.sh to python
which also works on other operating systems. It should also be
relatively easy for beginners to add new tests to assert stockfish's
output and to run it.
From the source directory the following command can be run.

`python3 ../tests/instrumented.py --none ./stockfish`

A test runner will go over the test suites and run the test cases.

All instrumented tests should have been ported over.
The required python version for this is should be 3.7 (untested),
testing.py includes some infrastructure code which setups the testing.

fixes #5185
closes #5583

No functional change
  • Loading branch information
Disservin committed Sep 13, 2024
1 parent 2680c9c commit ebf4f19
Show file tree
Hide file tree
Showing 5 changed files with 898 additions and 302 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/sanitizers.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,4 +75,4 @@ jobs:
export CXXFLAGS="-O1 -fno-inline"
make clean
make -j4 ARCH=x86-64-sse41-popcnt ${{ matrix.sanitizers.make_option }} debug=yes optimize=no build > /dev/null
../tests/instrumented.sh --${{ matrix.sanitizers.instrumented_option }}
python3 ../tests/instrumented.py --${{ matrix.sanitizers.instrumented_option }} ./stockfish
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,8 @@ src/-lstdc++.res
# Neural network for the NNUE evaluation
**/*.nnue

# Files generated by the instrumented tests
tsan.supp
__pycache__/
tests/syzygy
tests/bench_tmp.epd
Loading

0 comments on commit ebf4f19

Please sign in to comment.