Dev/CI/CD/Testing Infrastructure

[sergey-shandar]

I was thinking about different kinds of tests, such as unit, perf (WSTS benchmarks), code coverage, and E2E (manual observation of a working system). Then I see I a huge problem which is caused by different vendor-specific architectures of modern tools and services which were not designed for decentralized software open-source development.

Other Challenges

• Workflow is different in our repositories (like stacks-blockchain and sbtc). I think we should use a simple mono-repo style workflow.
  • Work in one main branch.
  • Iterate fast, and merge to main ASAP.
  • For breaking changes, don't create a new branch. Instead, create a new library, application, type, or command-line option.
• Different languages Rust, Clarity, JavaScript, etc.
• Publishing and versioning
  • in particular, Cargo.lock problem. For example, The new version of Regex (1.9.0) is not compatible with 1.8 #3779.
    • for applications, it's important to have strong dependencies,
    • for libraries, it's important to make sure it works on the current dependencies.
• Multiple repositories (with different licenses):
  • https://github.com/Trust-Machines/wsts
  • https://github.com/Trust-Machines/stacks-sbtc
  • https://github.com/Trust-Machines/p256k1
  • https://github.com/stacks-network/stacks-blockchain
  • https://github.com/stacks-network/sbtc
  • and others, including multiple web sites.
• Testing. This is huge. Testing Infrastructure #3788
  • testing on PRs
  • long-running tests
    • periodically running?
  • E2E testing on different networks (testnet/mainnet).
  • security
    • black-box testing
  • some testing can be very time-consuming (several hours); possible solutions:
    • Parallel testing. Split a set of tests to multiple subsets and run them on multiple machines.
    • Incremental testing (test only affected code).
• Downstream builds

Infrastructure Principles

1. As simple as possible
2. Cloud/Vendor Agnostic

Why?

We shouldn't depend on specific vendors because we develop open-source, decentralized software. Also, we had a lot of problems with different providers
such as

• GitHub Actions,
  • Pages are terrible, for example, there are no previews on PRs
• BuildJets, stuck after repo rename.
• CloudFlare can't install the required software (such as LaTeX). No source-based config files. (we may use conventions).
• Netlify, used by Stacks Foundation,
• Google Cloud,
• GitLab uses docker containers. Docker is not always good; sometimes, we need to test on very specific OSes, such as Windows, Android, iOs, and Mac OS.
• codecov.io

Every provider has its language to describe executions.

Alternatives:

• Shell. It cannot be used because
  • we need to be able to create new machines and send artifacts between them
  • not cross-platform

How?

To be vendor agnostic, we should have a way to describe and configure pipelines in simple terms. FunctionalScript can be used for such things.

Another problem is that we must execute the scripts to generate other scripts and configuration files. The solution is that we can commit the results of the scripts into the repo.

[Scripts] -> [generating CI configurations]

Generated scripts should be marked, for example, with a comment # Generated file. In the same type, developers may edit these files for experimenting and debugging. Also, sometimes, generated files may not include everything we need.

Another reason to do this is that we have multiple repositories, and we would like to provide a similar infrastructure for all of these repositories. We would like to have consistency between repositories.

The solution is to add it to FunctionalScript repo.

Plan

Short term

• generating GitHub Action yaml files for multiple repos using FunctionalScript. The config files should be generated and kept in the repo so we can minimize risk if things are going south

Notes

• Rust: cargo-nexttest
• Clarity: Gas profiler. https://blog.tenderly.co/how-to-reduce-smart-contract-gas-usage/

[sergey-shandar]

Copied from https://github.com/orgs/Trust-Machines/discussions/502

[sergey-shandar]

Roadmap

Tasks:

• Rust workflow:
  • fix Rust workspace (1 day)
  • improve code coverage by applying no IO principles
    • sBTC repo (2 weeks),
    • Stacks repo (unknown, more than 2 weeks).
    • documentation/articles to share the experience (1 week).
• Clarity workflow
  • investigation (2 weeks)
  • property testing (unknown)
• Improve infrastructure
  • generic pipelines (from 4 weeks and up), including generating and publishing documentation
  • CI/CD based on generic pipelines (from 4 weeks and up). Depends on the generic pipeline.
  • testing infrastructure based on generic pipelines (from 4 weeks and up). Depends on the generic pipeline.
  • shipping packages and versioning
    • Rust (1 week)
• E2E manual testing (step-by-step instructions must be executed manually at least once before releasing a new version)
  • sBTC (2 weeks)
  • Stacks (unknown)

Questions: should we simplify our branching strategy and use mono-repo? We can use either command-line arguments or separate executables for experimental and incompatible versions.

@kantai could you help me to identify tasks for Clarity workflow?

[wileyj]

Since i'm working specifically on the stacks-blockchain testing, ci/cd infra - this is of interest to me:

- Rust workflow:
    - fix Rust workspace (1 day)
    - improve code coverage by applying no IO principles
      - sBTC repo (2 weeks),
      - Stacks repo (unknown, more than 2 weeks).
      - documentation/articles to share the experience (1 week).

can you provide a little extra context around what you mean by "fix Rust workspace" and "no IO principles"?
if these are methods I can add to the work I'm doing, i'd very much like our approaches to match.

briefly, my plan is to remove our dependence on docker for running tests. rather, we'll use native Rust-based actions utilizing github actions caches to speed up testing via rust's nextest.

[kantai]

@kantai could you help me to identify tasks for Clarity workflow?

As far as I understand, I think there's a handful of tasks for the testing workstream related to clarity. The most immediate are:

1. Property testing of PoX-2 and PoX-3 via clarinet test. This requires applying a property testing framework (likely the one discussed by moodmosaic in Automated testing for the stacks-blockchain #3732), writing the property assertions, and building clarinet with this PR (Draft: Supporting PoX locks in Clarinet hirosystems/clarinet#1074).
2. Automation of the tests in https://github.com/hirosystems/stacks-e2e-testing
3. Sufficient automatic epoch-gated testing that the trait bug necessitating Stacks 2.3 would have been caught (as discussed in Automated testing for the stacks-blockchain #3732)
4. Mutation testing to highlight areas where improved testing should be possible.
5. Property testing of the Clarity VM itself -- this would be rust property tests written for each Clarity native function

[sergey-shandar]

@wileyj for fix workspace, I have an issue #3809 and a PR.

For no I/O principle, it's relative simple:

• our code should be mostly libraries, the libraries shouldn't have direct I/O calls (like read/write file, TCP/IP, current time, environment variables, etc.), I/O is used indirectly by dependency injection (in Rust, it's a zero-cost abstraction).
• applications should just bind the libraries with real I/O.

This way we can have almost 100% code coverage in unit tests without relaying on complicated/flaky dependencies, like network.

As an example, see the source code for a relay server https://github.com/Trust-Machines/stacks-sbtc/tree/main/relay-server.

[moodmosaic]

the libraries shouldn't have direct I/O calls

@sergey-shandar, agreed 100% on that. Rust doesn't use the type system to make side effects explicit in the same way Haskell does, for example.

How can this be enforced, then, apart from manual code review(s)? Perhaps via custom monadic structures to encapsulate and sequence side-effects and/or Result and Option types.

[moodmosaic]

Property testing of PoX-2 and PoX-3 via clarinet test. This requires applying a property testing framework (likely the one discussed by @moodmosaic in #3732), writing the property assertions, and building clarinet with this PR (hirosystems/clarinet#1074).

@kantai, is hirosystems/clarinet#1074 ready to be pulled and built?

[AshtonStephens]

Before we get into any language specific decisions for keeping cloud infrastructure vendor agnostic (e.g. FunctionalScript), I think we should analyze the tools that exist for agnostic cloud deployment and then find the programming language that allows us to best utilize those.

We discussed this a week ago, but I think we will likely end up defining our agnostic code in some front end that calls upon TerraformCDK so we can specify the vendor agnostic cloud infrastructure top level and turn it into vendor specific specification bottom level with Terraform. There may be tools that allow us to do this for FunctionalScript explicitly, but I've played around with TerraformCDK and can confirm that it works nicely with straight TypeScript (not sure why all these things use some JavaScript spinoff) and think it will be better to use raw TypeScript since that's what the framework is really designed for.

@sergey-shandar I'm wondering if there are other existing vendor agnostic IaC frameworks that you know of.

[AshtonStephens]

Also - I think for communication purposes in this design phase it would be good to create a flow chart of what we want the pipeline to look like for pull, push, etc. separately, with the bulkier pipeline stages maybe in red. I'm looking into issue-466 and the turnaround time for testing my changes against the pipeline in my private repo is 15 minutes, which imo is at least 10 minutes too long.

Also - I'm not entirely clear on why we're spinning up cloud infrastructure for the tests. Are we not trusting the OS specifications in GitHub?

[sergey-shandar]

@jo-tm could you add your thoughts, needs, links to tickets for sBTC related to testing and CI/CD?

• code coverage is broken for sBTC

[jio-gl]

• Some changes by friedger are needed to work with clarinet integrate (the Clarinet tests run on a Devnet) https://github.com/Trust-Machines/stacks-sbtc/pull/587

[jio-gl]

Added new issues for GH Action: https://github.com/Trust-Machines/stacks-sbtc/issues/611

[sergey-shandar]

@CharlieC3 feel free to comment

[sergey-shandar]

@netrome @jferrant @stjepangolemac @soju-drinker please, add your thoughts/links to issues and documents related to deployment and testing to this thread. Especially, we are interested in your current needs for sBTC from deployment and testing point of view. For example, a clear set of deployment steps or E2E test setup.

[netrome]

Sure, for sBTC the deployment and testing needs for sub-components are

• Signer binary: No central deployment needs but we need to ensure it's easy to set it up and that anyone can participate, that means testing on all major platforms. @jferrant correct me if I'm wrong.
• sBTC SDK: Ensure the SDK can be linked and built by downstream apps, meaning we should test compiling it for all major platforms. @stjepangolemac please elaborate if you have any thoughts.
• sBTC Docs. Initially we can serve the docs from GH pages, but I don't have strong preferences here.
• sBTC Clarity contracts. Developing and testing with Clarinet, I believe that's sufficient for now.
• sBTC E2E testing. I would love the possibility to run an isolated testing environment for sBTC with a bitcoin regtest + Stacks testnet node. This is quite complex to set up, so any simplifications here would be greatly appreciated.

[sergey-shandar]

Thank you, @netrome! Do we have a step-by-step scenario of how we would do E2E test? For example,

1. configure and run a coordinator
2. configure and run signers on different machines
3. wait for this...
4. observe this...
5. shutdown the machines

No need to specify deployment details, like Docker, VM settings, but we need details like what and how to configure E2E test for sBTC.

[netrome]

We don't but it's a good idea to have this in place for the next release. I created this issue to define one: https://github.com/stacks-network/sbtc/issues/11

[stjepangolemac]

Yeah testing the build for multiple platforms is a good thing but I don't think that it needs to be a priority. It would be a good thing to have just not critical.

[netrome]

GitLab uses docker containers. Docker is not always good; sometimes, we need to test on very specific OSes, such as Windows, Android, iOs, and Mac OS.

Just chiming in to say that you can set up your own custom runner with GitLab. Their CI solution is pretty good imo.

[sergey-shandar]

For Sprint 3:

• sBTC:
  • https://github.com/stacks-network/sbtc/issues/11 @CAGS295
  • test coverage
• Calrity related issues.
  • Dev/CI/CD/Testing Infrastructure #3787 (reply in thread)
• investigate further CI/CD and testing infra that will fit all repositories and requirements. @AshtonStephens @sergey-shandar
  • TM Google Cloud Infra @igorsyl
  • Checks issues Blockchain DevEx Survey Results & Quality of Life Action Items #3839
  • defines requirements, workflows, and standards for CI and Tests, for example xtask (Rust), nexttest?. Another example https://github.com/wileyj/stacks-blockchain/tree/master/.github/workflows
  • review https://github.com/orgs/stacks-network/discussions/490

[sergey-shandar]

Best Practices

• Avoid direct use of I/O
• Use conventions over configuration. For example,
  • a package name and a directory name should be the same. Make sure that the name of the package is not reserved on https://crates.io/. Publish an empty version of the package to https://crates.io/ to reserve the name. Otherwise, you may need to rename the package later.
  • the package should be a library without any use of I/O. Put all executables with I/O binding into src/bin. No need to specify the binaries in the Cargo.toml file.
• Branches
  • use main as the default branch. The branch should be used for active development. Please, don't create long lived feature branches. You may create a long living branch to create a fix for a critical bug for an old version of a package.
    For example,
    • main - active development
    • v1.0 - bug fixes for version 1.0.
    • v1.1 - bug fixes for version 1.1.
    • v2.0 - bug fixes for version 2.0.

    gitGraph
      commit id: "v0.1.0"
      commit id: "v1.0.0"
      branch v1.0
      checkout main
      commit id: "v1.1.0-alpha"
      checkout v1.0
      commit id: "v1.0.1 (critical bug fix)"
      checkout main
      commit id: "v1.1"
      branch v1.1
      checkout main
      commit id: "v2.0.0-alpha"
      checkout v1.1
      commit id: "v1.1.1 (critical bug fix)"
      checkout v1.0
      commit id: "v1.0.2 (critical bug fix)"
      checkout main
      commit id: "v2.0.0"
    

    Loading
  • use only squashed merge in PRs to keep the history clean and linear. The PR branch should be deleted after the merge.
    • we are working on issue12 and issue13:

      %%{init:{'themeVariables':{'git0':'#00f','git1':'#0f0','git2':'#0ff'}}}%%
      gitGraph
        commit id: "issue9"
        commit id: "issue10"
        branch issue12
        commit id: "AB34..."
        checkout main
        branch issue13
        commit id: "56FE..."
        commit id: "DDE0..."
        checkout issue12
        commit id: "82E4..."
      

      Loading
    • issue13 PR is merged and we are also working issue14:

      %%{init:{'themeVariables':{'git0':'#00f','git1':'#0f0','git2':'#f0f'}}}%%
      gitGraph
        commit id: "issue9"
        commit id: "issue10"
        branch issue12
        commit id: "AB34..."
        checkout main
        commit id: "issue13"
        branch issue14
        commit id: "3480..."
        checkout issue12
        commit id: "82E4..."
        merge main
      

      Loading
    • issue12 PR is merged and we are working on issue14:

      %%{init:{'themeVariables':{'git0':'#00f','git1':'#f0f'}}}%%
      gitGraph
        commit id: "issue9"
        commit id: "issue10"
        commit id: "issue13"
        branch issue14
        commit id: "3480..."
        checkout main
        commit id: "issue12"
        checkout issue14
        merge main
      

      Loading
    Try to avoid merges between branches. Merge changes from main into your branch instead. Multiple small atomic PRs are preferred to a single large PR. This will help to avoid merge conflicts and make the review process easier.
• use cargo fmt and cargo clippy to format and lint the code.
• Structure:
  • every Rust repository should either have only one package or a workspace with multiple packages.
  • A workspace Cargo.toml file should be at the root level of the repository. All packages should be in the packages directories.
  • CI should generate documentation and publish on a temporary URL. For example, CloudFlare can do this.
• Documentation:
  • use Markdown for documentation.
  • every Rust package should have a README.md file. This README.md file is also used to generate a front page for a create when published on https://crates.io/. the README.md should contain instruction on how to build, test, and run the package and prerequisites if needed. Use Mermaid for diagrams and LaTeX for math equations within Markdown. GitHub supports both:
    • https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/writing-mathematical-expressions
    • https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-diagrams
  • Rust comments supports examples. Please, provide comments with correct examples. Use cargo test --doc to run the examples as tests.
  • use cargo doc --open to generate documentation and open it in a browser.

[kantai]

Branches
use main as the default branch. The branch should be used for active development. Please, don't create long lived feature branches.
You may create a long living branch to create a fix for a critical bug for an old version of a package.

This repository uses gitflow (https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow) intentionally. While it'd be fine to use main instead of master, there is a strong rationale for having both main and develop. The idea is that active development occurs in develop, while main tracks only release code. This makes bug fixes, hot fixes, or patch releases much easier -- you open a patch PR based off main so that you do not need to also handle releasing whatever features are staged in the active development branch.

[netrome]

you open a patch PR based off main so that you do not need to also handle releasing whatever features are staged in the active development branch.

If you need to do hot fixes in a single-branch repo you can always base the fix directly of the latest release tag and ignore any unreleased changes. I don't see why you'd need multiple branches for that use case.