Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Fix Test workflow concurrency for code coverage #41061

Merged
merged 1 commit into from
Jan 14, 2025

Conversation

anomiex
Copy link
Contributor

@anomiex anomiex commented Jan 14, 2025

Proposed changes:

A few adjustments are needed to make the coverage stuff more reliable:

  1. For trunk runs, don't cancel for concurrency. We need each trunk commit to upload.
  2. If the actual run is cancelled, don't upload partial data.

Other information:

  • Have you written new tests for your changes, if applicable?
  • Have you checked the E2E test CI results, and verified that your changes do not break them?
  • Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Jetpack product discussion

Does this pull request change what data or activity we track or use?

Testing instructions:

  • CI happy?

A few adjustments are needed to make the coverage stuff more reliable:

1. For trunk runs, don't cancel for concurrency. We need each trunk
   commit to upload.
2. If the actual run is cancelled, don't upload partial data.
@anomiex anomiex added [Type] Bug When a feature is broken and / or not performing as intended [Status] Needs Review To request a review from fellow Jetpack developers. Label will be renamed soon. [Pri] Normal labels Jan 14, 2025
@anomiex anomiex requested a review from a team January 14, 2025 19:35
@anomiex anomiex self-assigned this Jan 14, 2025
@github-actions github-actions bot added the Actions GitHub actions used to automate some of the work around releases and repository management label Jan 14, 2025
Copy link
Contributor

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

  • ✅ Include a description of your PR changes.
  • ✅ Add a "[Status]" label (In Progress, Needs Team Review, ...).
  • ✅ Add a "[Type]" label (Bug, Enhancement, Janitorial, Task).
  • ✅ Add testing instructions.
  • ✅ Specify whether this PR includes any changes to data or privacy.
  • ✅ Add changelog entries to affected projects

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖


The e2e test report can be found here. Please note that it can take a few minutes after the e2e tests checks are complete for the report to be available.


Follow this PR Review Process:

  1. Ensure all required checks appearing at the bottom of this PR are passing.
  2. Choose a review path based on your changes:
    • A. Team Review: add the "[Status] Needs Team Review" label
      • For most changes, including minor cross-team impacts.
      • Example: Updating a team-specific component or a small change to a shared library.
    • B. Crew Review: add the "[Status] Needs Review" label
      • For significant changes to core functionality.
      • Example: Major updates to a shared library or complex features.
    • C. Both: Start with Team, then request Crew
      • For complex changes or when you need extra confidence.
      • Example: Refactor affecting multiple systems.
  3. Get at least one approval before merging.

Still unsure? Reach out in #jetpack-developers for guidance!

@@ -54,7 +56,7 @@ jobs:

# Note matrix-job outputs are kind of weird. Last-to-run job that sets a non-empty value wins.
outputs:
did-coverage: ${{ ( steps.process-coverage.conclusion == 'success' && steps.upload-artifacts.conclusion == 'success' ) && 'true' || '' }}
did-coverage: ${{ ( steps.run-tests.conclusion != 'cancelled' && steps.process-coverage.conclusion == 'success' && steps.upload-artifacts.conclusion == 'success' ) && 'true' || '' }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we even try to run process-coverage if run-tests is cancelled? 🤔 Though I guess if it runs that's an easy way to determine if coverage was enabled and I'm not sure if matrix.script == 'test-coverage' is accessible here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For manual analysis, it probably doesn't hurt to save whatever coverage data we did manage to generate. But, particularly for trunk runs, it'll throw things off if that partial data gets uploaded.

The check here is now threefold: (1) we finished running the actual tests, (2) we successfully processed the coverage data, and (3) we successfully uploaded the artifact. If 2 or 3 is false, the upload job won't even be able to run successfully because there will be no data. If 1 is false, it's partial data that we don't want to upload.

Copy link
Contributor

@tbradsha tbradsha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable.

@anomiex anomiex merged commit 7172def into trunk Jan 14, 2025
59 checks passed
@anomiex anomiex deleted the fix/tests-concurrency-for-codecov branch January 14, 2025 20:01
@github-actions github-actions bot removed the [Status] Needs Review To request a review from fellow Jetpack developers. Label will be renamed soon. label Jan 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Actions GitHub actions used to automate some of the work around releases and repository management [Pri] Normal [Type] Bug When a feature is broken and / or not performing as intended
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants