Draft CI requirements #61

wusatosi · 2024-11-15T17:18:00Z

This PR drafts basic requirements for the use of Continuous Integration in beman project.

Note that I am not good with writing documentation so please scrutinize scrutinize scrutinize.

Most of the wordings here are more expressions of ideas than correctly worded standard.

Closes: #34 .

Discussion may concurrently happen on discourse.

wusatosi · 2024-11-15T17:18:33Z

docs/BEMAN_STANDARD.md

+
+**[CI.SCHEDULED_TEST]** RECOMMENDATION: CI enabled checks should run periodically across the entire repository.
+
+**[CI.SCEDULED_TEST_NOTIFY_FAILURE]** REQUIREMENT: There must be a established protocol to report CI failure (e.g. opening a GitHub issue) when scheduled CI fails.


Alternative wording: A CI run must not fail silently.

I probably don't want an issue opened every time there's a failure...

While it's very useful to know the state of success across compilers, I think it would be fine for say, AppleClang, to not be able to compile something because it's too old to have some feature that's needed, or even just desired, for the library component.

Requiring C++26 is OK. In fact it may be necessary for adoption of some proposals.

I would not put this rule in the current PR, as we don't have yet implemented in any repo. Please remove for now.

wusatosi · 2024-11-15T17:19:14Z

docs/BEMAN_STANDARD.md

+
+**[CI.TEST.CROSS_COMPILER_VERSIONS]** RECOMMENDATION: CI enabled tests should be executed across support major versions of appripriate compiler families.
+
+**[CI.DURATION_PER_COMMIT]** RECOMMENDATION: CI pipeline associated with commits should finish in a reasonable amount of time.


Should we include a quantifiable "reasonable amount of time" as suggestion here?

My recommendation is 10 mins for small projects, no longer than 30 mins.

wusatosi · 2024-11-15T17:19:46Z

docs/BEMAN_STANDARD.md

+
+## Continous Integration
+
+**[CI.PLATFORM]** REQUIREMENT: All CI should be implemented using GitHub Actions or an CI platform that appripriatly interacts with the code control system.


an CI platform that appropriately interacts with the code control system may need better wording here. My intent here is to say "GitHub Actions is preferred, but if something checks PR, commits and write a check status to each commit and is reasonable, it is acceptable"

If someone wants to pay for an external CI system, that might be acceptable? We can cross that bridge if and when.

But for beman standards purposes, configuration as code is important, and that code should live with the project. That's built in with GH Actions, but it does mean I can mostly coax Gitea to use them, too. I haven't tried Gitlab.

Internally, we use GHE, but not Actions, and instead use our own systems for CI and CD, but they're sufficiently agnostic that it's not an issue. It is why I prefer as minimal magic in the actions as possible, though, so it's straightforward to run tests on change.

Yeah I agree, we need to come up with a wording for allowing acceptable external CI or make this item a recommendation.

docs/BEMAN_STANDARD.md

JeffGarland · 2024-11-20T07:06:45Z

docs/BEMAN_STANDARD.md

+**[CI.TEST.CROSS_PLATFORM]** REQUIREMENT: CI enabled tests must be executed across major platforms if appripriate.
+
+**[CI.TEST.CROSS_COMPILER_FAMILIES]** REQUIREMENT: CI enabled tests must be executed across major compiler families if appripriate.
+


These are listed as Requirements, but then come with the conditional 'if appropriate' -- which kinda makes them recommendations I'd say.

This is fair.

My thinking here is, let's say we have a feature that relies on trunk version of gcc/ clang, this requirement could be waived.

Co-authored-by: Jeff Garland <[email protected]>

neatudarius

Thanks for helping with #34!

Btw, that issue has a bigger scope, we still have much to do there. This is just a step.

My proposal is to apply the feedback from all reviewers and just keep an initial set of rules so we can advance with this PR.

neatudarius · 2025-01-21T06:43:08Z

docs/BEMAN_STANDARD.md

@@ -422,3 +422,21 @@ copyright notice following the SPDX license identifier.

 **[CPP.NAMESPACE]** RECOMMENDATION: Headers in `include/beman/<short_name>/` should export
 entities in the `beman::<short_name>` namespace.
+
+## Continous Integration


Suggested change

## Continous Integration

## Continuous integration

neatudarius · 2025-01-21T06:44:37Z

docs/BEMAN_STANDARD.md

+
+## Continous Integration
+
+**[CI.PLATFORM]** REQUIREMENT: All CI should be implemented using GitHub Actions or an CI platform that appripriatly interacts with the code control system.


Suggested change

**[CI.PLATFORM]** REQUIREMENT: All CI should be implemented using GitHub Actions or an CI platform that appripriatly interacts with the code control system.

**[CI.PLATFORM]** REQUIREMENT: The CI pipeline must be implemented using GitHub Actions.

check REQUIREMENT and "must" in this docs

I don't think we need to have external cases for start

We should pick between REQUIREMENT: The CI pipeline must be implemented using GitHub Actions. and `RECOMMANDATION: The CI pipeline should be implemented using GitHub Actions. These options are consistent with current doc wording.

neatudarius · 2025-01-21T06:47:20Z

docs/BEMAN_STANDARD.md

+
+**[CI.PLATFORM]** REQUIREMENT: All CI should be implemented using GitHub Actions or an CI platform that appripriatly interacts with the code control system.
+
+**[CI.TEST]** REQUIREMENT: CI must execute appropriate testing of reasonable scope tagged with each commit, and must fail with test cases.


Suggested change

**[CI.TEST]** REQUIREMENT: CI must execute appropriate testing of reasonable scope tagged with each commit, and must fail with test cases.

**[CI.ENABLE_TESTS]** REQUIREMENT: The CI must execute tests for each commit in each PR and must provide explicit output on failure.

and I would remove

neatudarius · 2025-01-21T06:49:19Z

docs/BEMAN_STANDARD.md

+
+**[CI.TEST]** REQUIREMENT: CI must execute appropriate testing of reasonable scope tagged with each commit, and must fail with test cases.
+
+**[CI.TEST.CROSS_PLATFORM]** REQUIREMENT: CI enabled tests must be executed across major platforms if appripriate.


We don't have multiple dot usage in the name of the rules in this doc!

remove CI.TEST.CROSS_COMPILER_FAMILIES and CI.TEST.CROSS_COMPILER_VERSIONS

Suggested change

**[CI.TEST.CROSS_PLATFORM]** REQUIREMENT: CI enabled tests must be executed across major platforms if appripriate.

**[CI.USE_CROSS_PLATFORM_TESTS]** RECOMMANDATION: The CI tests should be executed across major platforms and compilers.

neatudarius · 2025-01-21T06:52:16Z

docs/BEMAN_STANDARD.md

+
+**[CI.TEST.CROSS_COMPILER_VERSIONS]** RECOMMENDATION: CI enabled tests should be executed across support major versions of appripriate compiler families.
+
+**[CI.DURATION_PER_COMMIT]** RECOMMENDATION: CI pipeline associated with commits should finish in a reasonable amount of time.


Suggested change

**[CI.DURATION_PER_COMMIT]** RECOMMENDATION: CI pipeline associated with commits should finish in a reasonable amount of time.

**[CI.EXECUTION_TIME]** RECOMMENDATION: The entire CI pipeline should run as fast as possible.

Again, I would make it short or remote it.

neatudarius · 2025-01-21T06:53:12Z

docs/BEMAN_STANDARD.md

+
+**[CI.DURATION_PER_COMMIT]** RECOMMENDATION: CI pipeline associated with commits should finish in a reasonable amount of time.
+
+**[CI.SCHEDULED_TEST]** RECOMMENDATION: CI enabled checks should run periodically across the entire repository.


I would not put this rule in the current PR, as we don't have yet implemented in any repo. Please remove for now.

neatudarius · 2025-01-21T06:53:19Z

docs/BEMAN_STANDARD.md

+
+**[CI.SCHEDULED_TEST]** RECOMMENDATION: CI enabled checks should run periodically across the entire repository.
+
+**[CI.SCEDULED_TEST_NOTIFY_FAILURE]** REQUIREMENT: There must be a established protocol to report CI failure (e.g. opening a GitHub issue) when scheduled CI fails.


I would not put this rule in the current PR, as we don't have yet implemented in any repo. Please remove for now.

wusatosi · 2025-02-07T23:43:13Z

Thanks for helping with #34!

Btw, that issue has a bigger scope, we still have much to do there. This is just a step.

My proposal is to apply the feedback from all reviewers and just keep an initial set of rules so we can advance with this PR.

Hey I realized I have never respond to this, I will bring this up in a weekly sync for discussion.

Mainly because I was working on drafting beman test standard (#49), and I realize CI requirement should be there to compliment test requirement. A standalone CI requirement is definitely missing something.

wusatosi · 2025-02-07T23:44:39Z

I would not put this rule in the current PR, as we don't have yet implemented in any repo. Please remove for now.

We do! exemplar has this. I think this was one of my first PRs. It is pretty trivial to implement and has been working (bemanproject/exemplar#110).

neatudarius · 2025-02-28T18:27:11Z

I think this PR should be temporary closed. Let's finish CI in 2-3 repos and the make rules.

wusatosi · 2025-02-28T19:27:30Z

I think this PR should be temporary closed. Let's finish CI in 2-3 repos and the make rules.

yeah agreed, I can revisit once I have time for this.

draft CI requirements

c966ee7

wusatosi requested review from bretbrownjr, camio, dietmarkuehl, inbal2l, JeffGarland, neatudarius and steve-downey as code owners November 15, 2024 17:18

wusatosi commented Nov 15, 2024

View reviewed changes

JeffGarland reviewed Nov 20, 2024

View reviewed changes

docs/BEMAN_STANDARD.md Outdated Show resolved Hide resolved

JeffGarland reviewed Nov 20, 2024

View reviewed changes

typo fix

e3c7d3b

Co-authored-by: Jeff Garland <[email protected]>

neatudarius requested changes Jan 21, 2025

View reviewed changes

neatudarius self-assigned this Jan 21, 2025

wusatosi closed this Feb 28, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Draft CI requirements #61

Draft CI requirements #61

wusatosi commented Nov 15, 2024 •

edited

Loading

wusatosi Nov 15, 2024

JeffGarland Nov 20, 2024

steve-downey Nov 20, 2024

neatudarius Jan 21, 2025

wusatosi Nov 15, 2024

wusatosi Nov 15, 2024

wusatosi Nov 15, 2024 •

edited

Loading

steve-downey Nov 15, 2024

wusatosi Nov 15, 2024

JeffGarland Nov 20, 2024

wusatosi Nov 20, 2024

neatudarius left a comment

neatudarius Jan 21, 2025

neatudarius Jan 21, 2025

neatudarius Jan 21, 2025

neatudarius Jan 21, 2025

neatudarius Jan 21, 2025

neatudarius Jan 21, 2025

neatudarius Jan 21, 2025

wusatosi commented Feb 7, 2025

wusatosi commented Feb 7, 2025

neatudarius commented Feb 28, 2025

wusatosi commented Feb 28, 2025


		[CI.SCHEDULED_TEST] RECOMMENDATION: CI enabled checks should run periodically across the entire repository.

		[CI.SCEDULED_TEST_NOTIFY_FAILURE] REQUIREMENT: There must be a established protocol to report CI failure (e.g. opening a GitHub issue) when scheduled CI fails.


		[CI.TEST.CROSS_COMPILER_VERSIONS] RECOMMENDATION: CI enabled tests should be executed across support major versions of appripriate compiler families.

		[CI.DURATION_PER_COMMIT] RECOMMENDATION: CI pipeline associated with commits should finish in a reasonable amount of time.


		## Continous Integration

		[CI.PLATFORM] REQUIREMENT: All CI should be implemented using GitHub Actions or an CI platform that appripriatly interacts with the code control system.

		[CI.TEST.CROSS_PLATFORM] REQUIREMENT: CI enabled tests must be executed across major platforms if appripriate.

		[CI.TEST.CROSS_COMPILER_FAMILIES] REQUIREMENT: CI enabled tests must be executed across major compiler families if appripriate.

	[CI.PLATFORM] REQUIREMENT: All CI should be implemented using GitHub Actions or an CI platform that appripriatly interacts with the code control system.
	[CI.PLATFORM] REQUIREMENT: The CI pipeline must be implemented using GitHub Actions.


		[CI.PLATFORM] REQUIREMENT: All CI should be implemented using GitHub Actions or an CI platform that appripriatly interacts with the code control system.

		[CI.TEST] REQUIREMENT: CI must execute appropriate testing of reasonable scope tagged with each commit, and must fail with test cases.

	[CI.TEST] REQUIREMENT: CI must execute appropriate testing of reasonable scope tagged with each commit, and must fail with test cases.
	[CI.ENABLE_TESTS] REQUIREMENT: The CI must execute tests for each commit in each PR and must provide explicit output on failure.

	[CI.TEST.CROSS_PLATFORM] REQUIREMENT: CI enabled tests must be executed across major platforms if appripriate.
	[CI.USE_CROSS_PLATFORM_TESTS] RECOMMANDATION: The CI tests should be executed across major platforms and compilers.

	[CI.DURATION_PER_COMMIT] RECOMMENDATION: CI pipeline associated with commits should finish in a reasonable amount of time.
	[CI.EXECUTION_TIME] RECOMMENDATION: The entire CI pipeline should run as fast as possible.

Draft CI requirements #61

Draft CI requirements #61

Conversation

wusatosi commented Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wusatosi Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neatudarius left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wusatosi commented Feb 7, 2025

wusatosi commented Feb 7, 2025

neatudarius commented Feb 28, 2025

wusatosi commented Feb 28, 2025

wusatosi commented Nov 15, 2024 •

edited

Loading

wusatosi Nov 15, 2024 •

edited

Loading