Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci/transformers: run tests in utils, benchmark, generation, models #1190

Merged
merged 7 commits into from
Dec 24, 2024
+80 −13
Merged

ci/transformers: run tests in utils, benchmark, generation, models #1190

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c716cf1
ci/transformers: run tests in tests/utils
dvrogozh Dec 20, 2024
1073af0
ci/transformers: run all test groups even if one failed
dvrogozh Dec 20, 2024
989cb48
ci/transformers: run tests in tests/benchmark
dvrogozh Dec 21, 2024
80d5192
ci/transformers: run tests in tests/generation
dvrogozh Dec 21, 2024
29a1969
ci/transformers: print number of deselected tests
dvrogozh Dec 21, 2024
cb2af7e
ci/transformers: run tests in tests/*.py
dvrogozh Dec 21, 2024
0a0ed3d
ci/transformers: run tests in tests/models
dvrogozh Dec 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 80 additions & 13 deletions .github/workflows/_linux_transformers.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,18 +122,64 @@ jobs:
pip show torchvision | grep Version | grep xpu
python -c 'import torch; exit(not torch.xpu.is_available())'
- name: Run -k backbone tests
env:
TEST_CASE: 'tests_backbone'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a parameter test_cases for this workflow? And this parameter can be set as combination of tests_backbone,tests_py,tests_benchmark,tests_generation,tests_models,tests_pipelines,tests_trainer,tests_utils, and we also can set a default value for it as default test scope. Each test step need according to the value of test_cases to decide whether to do test.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. I also prefer chuanqi's comments about a new parameter for choose test scopes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, test cases parameter is one of those things we should do next. My thoughts on that are as follows:

  • As a developer I want some flexibility to set test cases when running on-demand workflow. For example, I see that running tests for one specific model might be needed, or running all the tests matching some conditions. With that in mind I incline to avoid setting pre-defined test scopes only (like tests_backbone, tests_py, etc.). Instead I think that specifying any input which is then passed to the pytest is preferable. Like:
test_cases="tests/model/bert/ -k sdpa"
...
python -m pytest --make-reports=... <other-common-params> $test_cases
  • I wonder how we will structure ci for other ecosystem projects? I mean that currently w/o test cases parameter we can add, for example, Huggingface Accelerate tests into the same workflow. But introducing test cases parameter might complicate that addition.

run: |
source activate huggingface_transformers_test
cd transformers
python3 -m pytest -rsf --make-reports=tests_backbone -k backbone tests
python3 -m pytest -rsf --make-reports=$TEST_CASE -k backbone tests || \
(echo "FAILED_CASES=$FAILED_CASES,$TEST_CASE" >> $GITHUB_ENV)
- name: Run tests/*.py
env:
TEST_CASE: 'tests_py'
run: |
source activate huggingface_transformers_test
cd transformers
python3 -m pytest -rsf --make-reports=$TEST_CASE tests/*.py || true
- name: Run tests/benchmark
env:
TEST_CASE: 'tests_benchmark'
run: |
source activate huggingface_transformers_test
cd transformers
python3 -m pytest -rsf --make-reports=$TEST_CASE tests/benchmark || true
- name: Run tests/generation
env:
TEST_CASE: 'tests_generation'
run: |
source activate huggingface_transformers_test
cd transformers
# Excluding tests due to:
# * torch.distributed.* not yet supported by XPU
pattern="not TestFSDPGeneration"
python3 -m pytest -rsf --make-reports=$TEST_CASE tests/generation -k "$pattern" || true
- name: Run tests/models
env:
TEST_CASE: 'tests_models'
run: |
source activate huggingface_transformers_test
cd transformers
# Excluding tests due to:
# * https://github.com/huggingface/transformers/issues/35252 (CUDA specific tests)
# * https://github.com/pytorch/pytorch/issues/140965 (aten::_linalg_eigvals)
pattern=" \
not test_model_parallelization and \
not test_model_parallel_equal_results and \
not test_resize_embeddings_untied and \
not test_resize_tokens_embeddings"
python3 -m pytest -rsf --make-reports=$TEST_CASE tests/models -k "$pattern" || true
- name: Run tests/pipelines
env:
TEST_CASE: 'tests_pipelines'
run: |
source activate huggingface_transformers_test
cd transformers
# Some tests are known to fail w/o clear pattern
# TODO: drop ||true after triage and fixes
python3 -m pytest -rsf --make-reports=tests_pipelines tests/pipelines || true
python3 -m pytest -rsf --make-reports=$TEST_CASE tests/pipelines || true
- name: Run tests/trainer
env:
TEST_CASE: 'tests_trainer'
run: |
source activate huggingface_transformers_test
cd transformers
Expand All @@ -145,7 +191,24 @@ jobs:
not TestTrainerDistributed and \
not TestTrainerDistributedXPU and \
not TestFSDPTrainer"
python3 -m pytest -rsf --make-reports=tests_trainer tests/trainer -k "$pattern"
python3 -m pytest -rsf --make-reports=$TEST_CASE tests/trainer -k "$pattern" || \
(echo "FAILED_CASES=$FAILED_CASES,$TEST_CASE" >> $GITHUB_ENV)
- name: Run tests/utils
env:
TEST_CASE: 'tests_utils'
run: |
source activate huggingface_transformers_test
cd transformers
# Excluding tests due to:
# * Network proxy connection issue, reason unknown
pattern="not test_load_img_url_timeout"
python3 -m pytest -rsf --make-reports=$TEST_CASE tests/utils -k "$pattern" || \
(echo "FAILED_CASES=$FAILED_CASES,$TEST_CASE" >> $GITHUB_ENV)
- name: Check for errors in tests
run: |
FAILED_CASES=$(echo $FAILED_CASES | sed 's/^,//')
echo "Failed cases: [$(echo $FAILED_CASES | sed 's/,/, /g')]"
test -z "$FAILED_CASES"
- name: Print results table
if: ${{ ! cancelled() }}
run: |
Expand All @@ -160,18 +223,19 @@ jobs:
cd transformers
{
echo "### Results"
echo "| Test group | Errors | Failed | Passed | Skipped |"
echo "| --- | --- | --- | --- | --- |"
echo "| Test group | Errors | Failed | Deselected | Passed | Skipped |"
echo "| --- | --- | --- | --- | --- | --- |"
for stat in $(find reports -name stats.txt); do
# Each stat.txt is located in: reports/$test_group/stats.txt
test_group=$(echo $stat | cut -f 2 -d/)
# Get failed, passed, skipped, etc. counters
failed=$(parse_stat $stat failed)
passed=$(parse_stat $stat passed)
deselected=$(parse_stat $stat deselected)
skipped=$(parse_stat $stat skipped)
warnings=$(parse_stat $stat warnings)
errors=$(parse_stat $stat errors)
echo "| $test_group | $errors | $failed | $passed | $skipped |"
echo "| $test_group | $errors | $failed | $deselected | $passed | $skipped |"
done
} >> $GITHUB_STEP_SUMMARY
- name: Print failure lines
Expand All @@ -180,24 +244,27 @@ jobs:
cd transformers
{
echo "### Failure lines"
echo "| File | Error | Comment |"
echo "| --- | --- | --- |"
echo "| Test group |File | Error | Comment |"
echo "| --- | --- | --- | --- |"
rm -rf _failures.txt
for failure in $(find reports -name failures_line.txt); do
tail -n +2 $failure >> _failures.txt
# Each failure_line.txt is located in: reports/$test_group/failure_line.txt
test_group=$(echo $failure | cut -f2 -d/)
tail -n +2 $failure | sed "s/^/$test_group /" >> _failures.txt
done
# failures_line.txt file does not have test case information,
# so we can just sort the output and report uniq values
sort _failures.txt | uniq > _failures_uniq.txt
while read line; do
file=$(echo $line | cut -f1 -d" " | sed "s/\(.*\):$/\1/")
error=$(echo $line | cut -f2 -d" " | sed "s/\(.*\):$/\1/")
test_group=$(echo $line | cut -f1 -d" ")
file=$(echo $line | cut -f2 -d" " | sed "s/\(.*\):$/\1/")
error=$(echo $line | cut -f3 -d" " | sed "s/\(.*\):$/\1/")
# Failure comments often contain special characters which complicate
# parsing failure lines. But fortunately we know for sure where comments
# start. So we just output all contents starting from this position and
# wrap everything in <pre></pre> to avoid collisions with Markdown formatting.
comment="<pre>$(echo $line | cut -f3- -d' ' | sed 's/\(.*\):$/\1/')</pre>"
echo "| $file | $error | $comment |"
comment="<pre>$(echo $line | cut -f4- -d' ' | sed 's/\(.*\):$/\1/')</pre>"
echo "| $test_group | $file | $error | $comment |"
done <_failures_uniq.txt
} >> $GITHUB_STEP_SUMMARY
- name: Print annotations
Expand Down
Loading