Testing Infrastructure

[sergey-shandar]

Testing Infrastructure

By Scale

• Unit Tests
• Integration Tests
• End-To-End Tests

Unit Tests

Unit tests should be as deterministic as possible, avoiding external processes, communication with servers, and using I/O. Instead, employ dependency injection to mock I/O. With proper I/O mocking, code coverage should approach 100%. Read about I/O dependency injection in this article: FunctionalScript. Though the article focuses on the FunctionalScript language, similar techniques can be applied to Rust and other languages.

These tests should run on Continuous Integration (CI) platforms (e.g., GitHub Actions), be locally executable and are part of CodeCoverage. See The test attribute for Rust unit tests.

Note System random and current time is also I/O functions and should not be used directly in the code and unit tests.

Process When a bug is found, add a new unit test that demonstrates the bug and then fix it.

Current State: We know how to implement these practices and should continue to apply them.

Integration Test

Integration tests may use I/O and other processes.

These tests should run on Continuous Integration (CI) platforms (e.g., GitHub Actions), be locally executable and are part of CodeCoverage. See Integration Tests for Rust integration tests.

These tests are part of Code Coverage.

Note current time and system random functions return different values every run, so log these value in order to reproduce a bug.

Current State: we know how to do this and we need to write more such tests.

End-to-end Test

End-to-end tests involve manual instructions on how to deploy software and validate the deployment. Although some scripts may help with deployment and validation, ultimately, someone should ensure that the system is working correctly.

End-to-end tests typically involve simulating real-world user interactions with the system to validate its functionality and performance. These tests can be time-consuming and resource-intensive but are crucial in ensuring that the software works as expected in various scenarios.

Current State: The deployment and manual testing process is currently unclear. We need to create step-by-step instructions to improve our end-to-end testing.

Other Types of Tests

• Performance/Benchmark Tests: Run these tests periodically on CI platforms to ensure that the software maintains its performance levels and meets required benchmarks.

• Stress and Security Attack Tests: Simulate different types of attacks on a deployed system to test its resilience and security measures.

Current State: WTFROST has some benchmark tests, but they are not currently run on CI.

Resources

CI

To ensure comprehensive testing across different platforms (CPU, OS, installed packages, etc.), we should/can utilize different CI platforms (like GH Actions, BuildJet, Circle CI, Appveyor) that support various environments.

OS	State
Linux	yes
MacOS	yes
Windows	missing
Android	missing
iOS	missing

CPU	State
amd64	yes
x86-32	no
ARM64	yes
ARM64(Apple)	no
ARM32	no

Additional Components for a Comprehensive Testing Infrastructure

• Monitoring production systems: Implement tools and processes to continuously monitor the performance and health of production systems.

• Telemetry: Collect and analyze system data to gain insights into performance, user behavior, and potential improvements.

• Bug reporting system: Establish a structured and efficient method for users to report bugs.

Current State: These components are currently only ideas and need to be developed and integrated into our testing infrastructure.

[sergey-shandar]

Copied from https://github.com/orgs/Trust-Machines/discussions/322

[moodmosaic]

Assuming that those tests will run over and over again, as part of a continuous process (build agent(s), CI/CD, nightly builds), why not take advantage of that and do some arbitrary execution of the contract's functions with arbitrary input? (Instead of hard coding e2e scenarios.)

Then, when an edge case is detected, the underlying framework/library used to test-drive this should, at the very least, be able to:

1. Replay the failed test — to reproduce locally and troubleshoot
2. Shrink the failed test — what's the simplest possible execution path (if any) that reproduces the error?

Since those are fundamental features of a property testing system, we may consider combining (some, if not all) integration tests and e2e tests into the form of testing described in #3732 (comment).

Here's some example output from a failed test-run:

I've never tried replaying this, but shrinking seems to be working well.

[CAGS295]

I remember trying the replaying feature at some point, and as far as I remember, it worked properly.

I also lean towards having a unified suite with property-based testing. At the moment I'm trying to think if sBTC-mini can be the 'simplified model' with the goal of also implementing invariant testing.

This way we would have a robust kit sharing components and tools (clarinet & devnetJS & fast-check) with a single logic source (.clar's):

Unit tests.
Property-based e2e.
Regression-based e2e.
Invariant-testing/simulation e2e.

[moodmosaic]

I remember trying the replaying feature at some point, and as far as I remember, it worked properly.

You mean you tried in general, right? I apologize if I wasn’t clear; I meant replaying there in that particular scenario (in the screenshot — this is not open sourced yet).