Skip to content

Commit 503d360

Browse files
jacobtylerwallsPierre-Sassoulas
jacobtylerwalls
and
Pierre-Sassoulas
authored
[primer] Parallelize primer workflow into package batches (#8697)
Co-authored-by: Pierre Sassoulas <[email protected]>
1 parent 9a2f113 commit 503d360

File tree

12 files changed

+145
-19
lines changed

12 files changed

+145
-19
lines changed

‎.github/workflows/primer_comment.yaml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,8 +103,9 @@ jobs:
103103
. venv/bin/activate
104104
python tests/primer/__main__.py compare \
105105
--commit=${{ github.event.workflow_run.head_sha }} \
106-
--base-file=output_${{ steps.python.outputs.python-version }}_main.txt \
107-
--new-file=output_${{ steps.python.outputs.python-version }}_pr.txt
106+
--base-file=output_${{ steps.python.outputs.python-version }}_main_BATCHIDX.txt \
107+
--new-file=output_${{ steps.python.outputs.python-version }}_pr_BATCHIDX.txt \
108+
--batches=4
108109
- name: Post comment
109110
id: post-comment
110111
uses: actions/[email protected]

‎.github/workflows/primer_run_main.yaml

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,14 @@ permissions:
2424

2525
jobs:
2626
run-primer:
27-
name: Run / ${{ matrix.python-version }}
27+
name: Run / ${{ matrix.python-version }} / batch index ${{ matrix.batchIdx }}
2828
runs-on: ubuntu-latest
29-
timeout-minutes: 120
29+
timeout-minutes: 45
3030
strategy:
3131
matrix:
3232
python-version: ["3.8", "3.11"]
33+
batches: [4]
34+
batchIdx: [0, 1, 2, 3]
3335
steps:
3436
- name: Check out code from GitHub
3537
uses: actions/[email protected]
@@ -91,7 +93,10 @@ jobs:
9193
run: |
9294
. venv/bin/activate
9395
pip install -e .
94-
python tests/primer/__main__.py run --type=main 2>warnings.txt
96+
python tests/primer/__main__.py run --type=main --batches=${{ matrix.batches }} --batchIdx=${{ matrix.batchIdx }} 2>warnings.txt
97+
- name: Echo warnings
98+
if: success() || failure()
99+
run: |
95100
WARNINGS=$(head -c 65000 < warnings.txt)
96101
if [[ $WARNINGS ]]
97102
then echo "::warning ::$WARNINGS"
@@ -102,4 +107,4 @@ jobs:
102107
name: primer_output
103108
path: >-
104109
tests/.pylint_primer_tests/output_${{ steps.python.outputs.python-version
105-
}}_main.txt
110+
}}_main_batch${{ matrix.batchIdx }}.txt

‎.github/workflows/primer_run_pr.yaml

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -33,12 +33,14 @@ permissions:
3333

3434
jobs:
3535
run-primer:
36-
name: Run / ${{ matrix.python-version }}
36+
name: Run / ${{ matrix.python-version }} / batch index ${{ matrix.batchIdx }}
3737
runs-on: ubuntu-latest
38-
timeout-minutes: 120
38+
timeout-minutes: 45
3939
strategy:
4040
matrix:
4141
python-version: ["3.8", "3.11"]
42+
batches: [4]
43+
batchIdx: [0, 1, 2, 3]
4244
steps:
4345
- name: Check out code from GitHub
4446
uses: actions/[email protected]
@@ -156,7 +158,10 @@ jobs:
156158
run: |
157159
. venv/bin/activate
158160
pip install -e .
159-
python tests/primer/__main__.py run --type=pr 2>warnings.txt
161+
python tests/primer/__main__.py run --type=pr --batches=${{ matrix.batches }} --batchIdx=${{ matrix.batchIdx }} 2>warnings.txt
162+
- name: Echo warnings
163+
if: success() || failure()
164+
run: |
160165
WARNINGS=$(head -c 65000 < warnings.txt)
161166
if [[ $WARNINGS ]]
162167
then echo "::warning ::$WARNINGS"
@@ -167,12 +172,14 @@ jobs:
167172
name: primer_output_pr
168173
path:
169174
tests/.pylint_primer_tests/output_${{ steps.python.outputs.python-version
170-
}}_pr.txt
175+
}}_pr_batch${{ matrix.batchIdx }}.txt
171176
- name: Upload output of 'main'
172177
uses: actions/[email protected]
173178
with:
174179
name: primer_output_main
175-
path: output_${{ steps.python.outputs.python-version }}_main.txt
180+
path:
181+
output_${{ steps.python.outputs.python-version }}_main_batch${{
182+
matrix.batchIdx }}.txt
176183

177184
# Save PR number so we know which PR to comment on
178185
- name: Save PR number

‎pylint/testutils/_primer/primer.py

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,18 @@ def __init__(self, primer_directory: Path, json_path: Path) -> None:
5656
run_parser.add_argument(
5757
"--type", choices=["main", "pr"], required=True, help="Type of primer run."
5858
)
59+
run_parser.add_argument(
60+
"--batches",
61+
required=False,
62+
type=int,
63+
help="Number of batches",
64+
)
65+
run_parser.add_argument(
66+
"--batchIdx",
67+
required=False,
68+
type=int,
69+
help="Portion of primer packages to run.",
70+
)
5971

6072
# All arguments for the compare parser
6173
compare_parser = self._subparsers.add_parser("compare")
@@ -74,6 +86,12 @@ def __init__(self, primer_directory: Path, json_path: Path) -> None:
7486
required=True,
7587
help="Commit hash of the PR commit being checked.",
7688
)
89+
compare_parser.add_argument(
90+
"--batches",
91+
required=False,
92+
type=int,
93+
help="Number of batches (filepaths with the placeholder BATCHIDX will be numbered)",
94+
)
7795

7896
# Storing arguments
7997
self.config = self._argument_parser.parse_args()

‎pylint/testutils/_primer/primer_compare_command.py

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,24 @@
1818

1919
class CompareCommand(PrimerCommand):
2020
def run(self) -> None:
21-
main_data = self._load_json(self.config.base_file)
22-
pr_data = self._load_json(self.config.new_file)
21+
if self.config.batches is None:
22+
main_data = self._load_json(self.config.base_file)
23+
pr_data = self._load_json(self.config.new_file)
24+
else:
25+
main_data = {}
26+
pr_data = {}
27+
for idx in range(self.config.batches):
28+
main_data.update(
29+
self._load_json(
30+
self.config.base_file.replace("BATCHIDX", "batch" + str(idx))
31+
)
32+
)
33+
pr_data.update(
34+
self._load_json(
35+
self.config.new_file.replace("BATCHIDX", "batch" + str(idx))
36+
)
37+
)
38+
2339
missing_messages_data, new_messages_data = self._cross_reference(
2440
main_data, pr_data
2541
)

‎pylint/testutils/_primer/primer_run_command.py

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -30,14 +30,21 @@ class RunCommand(PrimerCommand):
3030
def run(self) -> None:
3131
packages: PackageMessages = {}
3232
fatal_msgs: list[Message] = []
33-
for package, data in self.packages.items():
33+
package_data_iter = (
34+
self.packages.items()
35+
if self.config.batches is None
36+
else list(self.packages.items())[
37+
self.config.batchIdx :: self.config.batches
38+
]
39+
)
40+
for package, data in package_data_iter:
3441
messages, p_fatal_msgs = self._lint_package(package, data)
3542
fatal_msgs += p_fatal_msgs
3643
local_commit = Repo(data.clone_directory).head.object.hexsha
3744
packages[package] = PackageData(commit=local_commit, messages=messages)
38-
path = (
39-
self.primer_directory
40-
/ f"output_{'.'.join(str(i) for i in sys.version_info[:3])}_{self.config.type}.txt"
45+
path = self.primer_directory / (
46+
f"output_{'.'.join(str(i) for i in sys.version_info[:3])}_{self.config.type}"
47+
+ (f"_batch{self.config.batchIdx}.txt" if self.config.batches else "")
4148
)
4249
print(f"Writing result in {path}")
4350
with open(path, "w", encoding="utf-8") as f:

‎tests/testutils/_primer/fixtures/batched/expected.txt

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
🤖 **Effect of this PR on checked open source code:** 🤖
2+
3+
4+
5+
**Effect on [astroid](https://github.com/pylint-dev/astroid):**
6+
The following messages are no longer emitted:
7+
8+
<details>
9+
10+
1) locally-disabled:
11+
*Locally disabling redefined-builtin (W0622)*
12+
https://github.com/pylint-dev/astroid/blob/1234567890abcdef/astroid/__init__.py#L91
13+
2) unknown-option-value:
14+
*Unknown option value for 'disable', expected a valid pylint message and got 'Ellipsis'*
15+
https://github.com/pylint-dev/astroid/blob/1234567890abcdef/astroid/__init__.py#L91
16+
17+
</details>
18+
19+
*This comment was generated for commit v2.14.2*

‎tests/testutils/_primer/fixtures/batched/main_batch0.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{}

‎tests/testutils/_primer/fixtures/batched/main_batch1.json

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
{
2+
"astroid": {
3+
"commit": "1234567890abcdef",
4+
"messages": [
5+
{
6+
"type": "info",
7+
"module": "astroid",
8+
"obj": "",
9+
"line": 91,
10+
"column": 0,
11+
"endLine": null,
12+
"endColumn": null,
13+
"path": "tests/.pylint_primer_tests/pylint-dev/astroid/astroid/__init__.py",
14+
"symbol": "locally-disabled",
15+
"message": "Locally disabling redefined-builtin (W0622)",
16+
"message-id": "I0011"
17+
},
18+
{
19+
"type": "warning",
20+
"module": "astroid",
21+
"obj": "",
22+
"line": 91,
23+
"column": 0,
24+
"endLine": null,
25+
"endColumn": null,
26+
"path": "tests/.pylint_primer_tests/pylint-dev/astroid/astroid/__init__.py",
27+
"symbol": "unknown-option-value",
28+
"message": "Unknown option value for 'disable', expected a valid pylint message and got 'Ellipsis'",
29+
"message-id": "W0012"
30+
}
31+
]
32+
}
33+
}

‎tests/testutils/_primer/fixtures/batched/pr_batch0.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{}

0 commit comments

Comments
 (0)