fix: cargo test takes too long to test

[AlexD10S]

Fixing Issue #88

• Run only unit tests when cargo test

• unit tests that are a bit slower, build a contract move it into feature unit_contract. To run cargo test --features unit_contract

• Run e2e tests using a feature flag cargo test --features e2e_contract for contracts and cargo test --features e2e_parachain for parachains

• Moved the test that builds a parachain to e2e test

• In the CI run everything cargo test --all-features -- --include-ignored

[weezy20]

cargo test --bins runs fast with 3 ignored test. However, if I run cargo test, I get the same followed by a somewhat time consuming:

running 3 tests
test test_contract_build_fails_if_no_contract_exists ... ok
test test_contract_build_specify_path ... ok
test test_contract_build ... ok

Why is test_contract_build running even after being #[ignore]d?

[AlexD10S]

cargo test --bins runs fast with 3 ignored test. However, if I run cargo test, I get the same followed by a somewhat time consuming:

running 3 tests
test test_contract_build_fails_if_no_contract_exists ... ok
test test_contract_build_specify_path ... ok
test test_contract_build ... ok

Why is test_contract_build running even after being #[ignore]d?

The ignored one was not being run, there were 2 tests called test_contract_build. I changed that. To run only unit tests now cargo test, to run e2e related to contracts: cargo test --features e2e_contract and to run everything ignored too: cargo test --all-features -- --include-ignored.

My doubt is still what to run in the CI, once we have the up tests run everything cargo test --all-features -- --include-ignored can take more than 2h but we will be sure the product is stable.

[weezy20]

My doubt is still what to run in the CI, once we have the up tests run everything cargo test --all-features -- --include-ignored can take more than 2h but we will be sure the product is stable.

So here's my suggestion. For product stability, i.e. sanity of generated template builds, it doesn't make sense to have them run whilst a PR is open for review or constantly being worked on. This slows us down.

At bare minimum unit tests must be run on PRs.
For e2e tests, specifically build_* tests, they can be merged only on merge to main. See https://stackoverflow.com/a/61565445/6591533

[brunopgalvao]

I am not a fan of using ignore. I would rather use a tiered approach.

cargo test
cargo test --features all_tests
cargo test --features all_unit_tests
cargo test --features all_contract_unit_tests
cargo test --features all_parachain_unit_tests
cargo test --features all_e2e_tests
cargo test --features all_e2e_contract_tests
cargo test --features all_e2e_parachain_tests

OR

cargo test
cargo test --features tier1_tests
cargo test --features tier2_tests
cargo test --features tier3_tests

where tier 1 would be least amount of tests and tier 3 would be full testing.

[AlexD10S]

I am not a fan of using ignore. I would rather use a tiered approach.

cargo test
cargo test --features all_tests
cargo test --features all_unit_tests
cargo test --features all_contract_unit_tests
cargo test --features all_parachain_unit_tests
cargo test --features all_e2e_tests
cargo test --features all_e2e_contract_tests
cargo test --features all_e2e_parachain_tests

OR

cargo test
cargo test --features tier1_tests
cargo test --features tier2_tests
cargo test --features tier3_tests

where tier 1 would be least amount of tests and tier 3 would be full testing.

Done! No more ignored tests

[weezy20]

Lgtm. Thanks @AlexD10S