Use less nix in CI

[ch1bo]

By running the tests, benchmarks and haddock in one job, we have overall lower concurrency and require less runners. But maybe having a single job is a bit too much serialization and having 2-3 jobs maps better onto runners.?

Also, this is not using nix to build/cache the haskell binaries (test suites and benchmarks) so the build step is done on each execution and we might not want to merge this because of the overall longer run-time.

OTOH, this also directly shows the quite high "clean build" development workflow time in this project (while orders of magnitude less than building all the cardano dependencies too). Maybe something we want to tackle and not ignore it by claiming "just use nix"?

---

• CHANGELOG updated or not needed
• Documentation updated or not needed
• Haddocks updated or not needed
• No new TODOs introduced or explained herafter

[github-actions]

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2024-09-20 15:47:53.563268939 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	2fac819a1f4f14e29639d1414220d2a18b6abd6b8e444d88d0dda8ff	3799
νCommit	2043a9f1a685bcf491413a5f139ee42e335157c8c6bc8d9e4018669d	1743
νHead	2ee477c60839936be49a50030690865b5bed4db8cd2f05bf255ac680	10068
μHead	a1610f6e64843161f4a88229c0286176f5325de3e2f773eec2b1d818*	4508
νDeposit	c2117fd9ebdee3e96b81fd67ff7092d638926415c10f1f7e5a267ad0	2791

• The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per head.

Init transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	5094	5.93	2.35	0.44
2	5298	7.07	2.79	0.46
3	5498	8.46	3.34	0.48
5	5902	11.32	4.48	0.53
10	6908	18.06	7.14	0.65
57	16353	82.81	32.75	1.77

Commit transaction costs

This uses ada-only outputs for better comparability.

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	566	10.52	4.15	0.29
2	759	13.86	5.65	0.34
3	947	17.33	7.20	0.38
5	1316	24.65	10.44	0.48
10	2259	45.22	19.36	0.75
20	4132	95.99	40.76	1.40

CollectCom transaction costs

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	560	21.46	8.41	0.41
2	114	671	32.20	12.78	0.53
3	171	782	47.41	18.93	0.70
4	228	893	62.77	25.24	0.88
5	283	1004	78.39	31.78	1.06
6	339	1116	91.73	37.56	1.21

Cost of Decrement Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	649	17.91	7.87	0.38
2	775	19.34	9.16	0.41
3	953	21.44	10.69	0.44
5	1272	25.55	13.74	0.51
10	1952	31.12	19.52	0.63
49	8069	99.53	74.47	1.85

Close transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	674	20.07	9.03	0.41
2	819	21.91	10.73	0.44
3	873	22.62	11.49	0.45
5	1293	26.79	15.32	0.53
10	1958	33.94	22.28	0.67
50	8313	98.90	84.61	1.93

Contest transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	676	25.93	11.14	0.47
2	876	28.07	12.99	0.51
3	956	29.38	14.13	0.53
5	1204	32.94	17.16	0.59
10	2135	43.67	26.36	0.79
39	6260	96.46	72.01	1.72

Abort transaction costs

There is some variation due to the random mixture of initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4991	17.43	7.59	0.57
2	5104	28.50	12.49	0.70
3	5244	41.43	18.27	0.85
4	5322	56.81	25.12	1.02
5	5544	77.16	34.34	1.27
6	5631	95.57	42.53	1.48

FanOut transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
5	0	0	4935	7.70	3.25	0.45
5	1	57	4968	8.43	3.80	0.46
5	5	285	5105	13.35	6.81	0.53
5	10	569	5274	19.60	10.62	0.62
5	20	1138	5612	30.33	17.48	0.77
5	30	1708	5955	41.65	24.61	0.93
5	40	2274	6290	52.78	31.65	1.08
5	50	2845	6631	64.51	38.94	1.24
5	81	4606	7679	99.08	60.82	1.73

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2024-09-20 15:52:38.684812961 UTC

Baseline Scenario

Number of nodes	1
Number of txs	3000
Avg. Confirmation Time (ms)	12.681555637
P99	17.235234429999956ms
P95	14.572044550000001ms
P50	12.859226ms
Number of Invalid txs	0

Three local nodes

Number of nodes	3
Number of txs	9000
Avg. Confirmation Time (ms)	43.221895139
P99	86.86788676000015ms
P95	57.31235434999998ms
P50	40.7763725ms
Number of Invalid txs	0

[locallycompact]

Using self-hosted runners with cabal test all will still jam the ports.

[github-actions]

Test Results

506 tests  +3   500 ✅ +3   25m 15s ⏱️ + 3m 35s
162 suites +2     6 💤 ±0 
  7 files   ±0     0 ❌ ±0 

Results for commit 1c50bdb. ± Comparison against base commit d5729d3.

This pull request removes 4 and adds 7 tests. Note that renamed tests count towards both.

Plutus.Codec.CBOR.Encoding ‑ matches cborg library encoding
Plutus.MerkleTree ‑ can check membership of an element
Plutus.MerkleTree ‑ fromList . toList roundtrips MT
Plutus.MerkleTree ‑ tree is balanced

Hydra.ChainObserver ‑ All valid transitions for all possible states can be observed.
Hydra.ChainObserver ‑ Does not updates UTxO state given transactions outside of Head lifecycle
Hydra.ChainObserver ‑ Updates UTxO state given transaction part of Head lifecycle
Hydra.Explorer.ExplorerState/aggregate head observation into explorer state ‑ Any head observations (of some head id) must yield an entry of that head id
Hydra.Explorer.ExplorerState/aggregate head observation into explorer state ‑ Given any observations, the resulting list of head ids is a prefix of the original
Hydra.Explorer/API should respond correctly/GET /heads ‑ matches schema
Hydra.Explorer/API should respond correctly/GET /tick ‑ matches schema

♻️ This comment has been updated with latest results.

[ch1bo]

Using self-hosted runners with cabal test all will still jam the ports.

What do you mean with "jam the ports"? If you mean that it does not pass because there is some test polution, then yes, we should fix it and make cabal test all usable.

FWIW I updated the description and I might not even propose merging this. It's just an experiment.