Improve local and CI flows

[neatudarius]

TLDR: scripts for local and CI flows. #1 #2 #11 #16
The codebase (e.g. C++ files) are not updated in this PR. I will make a new PR an update main and rebase this branch until this PR becomes green!

===========================================================================

Updates already applied:

• Update toolchain files (etc/ tree)

  • Remove unused toolchain files from etc/ . Add license. Run linters.
  • Expose CMAKE_CXX_STANDARD (tested with C++20, C++23 and C++26)

• Configure linters:

  • C++: clang-format
    • pulled version: 19, tested on Ubuntu 24.04
    • config: .clang-format
  • Shell: shell-check
    • pulled version: default apt-get version on Ubuntu 24.04
    • config: .shellcheckrc
  • Markdown: markdownlint
    • pulled version: default version available with npm on Ubuntu 24.04
    • config: .markdownlint.json
  • YAML: yamllint
    • pulled version: default apt-get version, on Ubuntu 24.04
    • config: yamllint
  • CMake: cmake-format
    • pulled version: default apt-get version, on Ubuntu 24.04
    • config: .cmake-format

• Scripts: Uniform testing and linting local and CI environments! Full usage in scripts/README.md.

  • scripts/lint.sh: Script used to lint some or all files in the project.

    • Can detect errors. Can automatically fix detected errors (--fix).

  • scripts/run-tests.sh: Script used to run tests on local and CI environments!

    • Can use a single or multiple toolchains (--preset). Can set a C++ standard (--std).

• New CI flows:

  • github/workflows/ci_tests.yml: use scripts/run-test.sh and a matrix config to run all tests.

    • toolchans: {gcc-13, gcc-14, clang-17, clang-18, clang-19}. Note: Expected gcc-13 to fail with C++26.
    • stds: {20, 23, 26}

  • github/workflows/ci_lint.yml: use scripts/lint.sh to make a PR have failed jobs if linting fails.

    • types of checks: {--cpp, --shell, --markdown, --yaml, --cmake}
    • examples of failed jobs:
      • clang-format failed job
      • shellcheck failed job
      • markdownlint failed job
      • yamllint failed jobs
      • cmake-format failed job

===========================================================================

Next steps:

• All jobs from ci_tests.yml are green and can be deployed in main.
• Intentionally, all jobs from ci_lint.yml are red. I will push start making commits to fix linter errors, after we agree a direciton.

[steve-downey]

This will make it slightly harder to keep the .clang-format up to date because clang-format will want to rewrite it when you dump the config.
I believe it's right now at clang-format-17.

clang-format-19 --style=file:.clang-format    --dump-config

will produce the file that clang-format believes it read, adding the options that were defaulted.

[neatudarius]

This will make it slightly harder to keep the .clang-format up to date because clang-format will want to rewrite it when you dump the config.

I'm OK to remove this file, but wouldn't be consistent for other linters? I think I would like to change it manually (using --dump-config into a temporary file). What do you think?

btw, I tried with clang 17/18/19, and I cannot get the current config. Would you like to update it with clang-format-19 in this PR? (The required changes will be applied in later commits or in a new PR).

[steve-downey]

It's the comment itself that gets lost. clang-format configs are generally one way upgradable but otherwise stable. That is, clang-format-N+1 will read an N file and default the elements that are added, so that when you run dump-config you get a file that only cf-N+1 understands but that produces the same output as the old config.

Caveats around syntax that clang-format didn't used to understand, of course.

We don't need to be eager about upgrading unless we decide we need the new knobs.

I've left myself a TODO to see if clang-format changes matter for the few files we have, and if so, we probably don't need to upgrade.

It's just that the comments are easy to get lost, and I wouldn't want anyone to read too much into that.

[steve-downey]

I'll take a look at what the lint fixes would do before approving this, but I'm in principle in favor of all it.

I have no strong feelings about markdown, shell, yaml, and json formatting -- all of them are terrible and I don't particularly care.

Slight preferences about C++ formatting, although as long as clang-format can put things back in canonical form it's only a slight preference. And with Lint Cpp currently green, it looks like I'm OK.

[steve-downey]

Remove the execution trace option?
It's very noisy to use the script with this set.

[steve-downey]

👍

[neatudarius]

Related PR - #52 . Will discuss with @steve-downey the approach.

[neatudarius]

Put this PR on hold until we solve similar issue/discussion for beman.exampler.

[camio]

Put this PR on hold until we solve similar issue/discussion for beman.exampler.

I think a good way to do that would be to mark it as a draft PR.

[neatudarius]

Put this PR on hold until we solve similar issue/discussion for beman.exampler.

I think a good way to do that would be to mark it as a draft PR.

It seems it cannot be put back to draft. I will temporary close it.