Small changes to run benchmarks for lots of transactions

[noonio]

Fixes #1724

This slightly modifies the micro benchmark to hake a configurable number of transactions to run for, and shows how to use it to get memory usage output.

[github-actions]

Transaction cost differences

No cost or size differences found

[noonio]

Hmm, seems like tasty-bench doesn't support the HTML output option; so will need to see if there's a quick fix for that.

[github-actions]

Test Results

  5 files  ±0  162 suites  ±0   28m 3s ⏱️ - 4m 7s
554 tests ±0  548 ✅ ±0  6 💤 ±0  0 ❌ ±0 
556 runs  ±0  550 ✅ ±0  6 💤 ±0  0 ❌ ±0 

Results for commit 41fb994. ± Comparison against base commit 3be7b22.

♻️ This comment has been updated with latest results.

[noonio]

Running the benchmarks like so:

N_TXNS=1000000 cabal bench micro --benchmark-options '--json output.json +RTS -T'

Supposing you get peakMbAllocated from this:

jq '.[2][0].reportKeys[5]' hydra-node/output.json

You can find the mb this way:

jq '.[2][0].reportMeasured[0][5]' hydra-node/output.json

(Exercise for someone to do this properly for the JSON output.)

[github-actions]

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2024-11-13 13:45:59.043412945 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	00a6ddbc130ab92f5b7cb8d1ccd8d79eca5bfe25f6843c07b62841f0	2667
νCommit	3e5a776bcee213e3dfd15806952a10ac5590e3e97d09d62eb99266b2	690
νHead	00e0e6aceb51fcb4fcc5dc6032cc524a641163f0723b25d995de3ae8	12633
μHead	134f18365d53de72b07af2e703c96d8a6f0643c71d4c1a2050b8182a*	11111
νDeposit	2feb47889a4f658dc593cefcb0e37d584b9431944f08a687f3dab4af	4865

• The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per head.

Init transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	11700	8.83	2.88	0.76
2	11896	10.76	3.52	0.79
3	12097	12.83	4.21	0.82
5	12500	15.89	5.17	0.87
10	13503	24.65	8.02	1.00
24	16322	49.59	16.14	1.39

Commit transaction costs

This uses ada-only outputs for better comparability.

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	561	2.45	1.17	0.20
2	743	3.40	1.74	0.22
3	917	4.39	2.34	0.24
5	1274	6.46	3.61	0.28
10	2180	12.24	7.28	0.40
54	10063	99.20	68.72	1.89

CollectCom transaction costs

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	560	24.69	7.19	0.43
2	114	671	32.74	9.53	0.51
3	170	786	40.68	11.83	0.60
4	227	893	51.48	14.81	0.71
5	283	1004	62.90	17.95	0.83
6	340	1116	63.48	18.49	0.85
7	395	1227	73.24	21.27	0.95
8	451	1338	85.09	24.42	1.07
9	505	1449	87.27	25.45	1.10

Cost of Decrement Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	654	22.26	7.22	0.41
2	771	23.99	8.36	0.44
3	914	24.67	9.20	0.46
5	1248	28.77	11.68	0.52
10	1911	36.87	17.31	0.66
41	6783	99.53	55.53	1.65

Close transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	668	24.31	8.01	0.43
2	819	26.44	9.44	0.47
3	1036	28.61	10.90	0.51
5	1286	31.60	13.19	0.56
10	1998	39.80	19.23	0.70
44	7062	99.33	61.92	1.71

Contest transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	688	30.32	9.61	0.49
2	815	32.17	10.85	0.52
3	900	33.42	11.80	0.54
5	1344	38.85	15.13	0.63
10	1927	47.73	21.14	0.77
34	5529	97.00	53.20	1.56

Abort transaction costs

There is some variation due to the random mixture of initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	11575	25.80	8.76	0.93
2	11708	34.12	11.62	1.02
3	11861	43.69	14.91	1.13
4	12013	52.44	17.86	1.23
5	12035	56.15	19.08	1.27
6	12317	71.23	24.34	1.44
7	12431	79.38	27.06	1.53
8	12405	83.59	28.31	1.58
9	12525	87.13	29.54	1.62
10	12712	95.88	32.70	1.72

FanOut transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTXO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
10	0	0	11692	17.34	5.93	0.85
10	1	57	11727	19.19	6.70	0.87
10	5	285	11862	27.49	10.07	0.97
10	10	570	12032	35.13	13.28	1.06
10	20	1139	12371	54.02	21.03	1.28
10	30	1704	12708	73.27	28.92	1.51
10	40	2273	13047	91.81	36.54	1.73
10	44	2505	13187	98.49	39.32	1.81

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2024-11-13 13:49:05.436458294 UTC

Baseline Scenario

Number of nodes	1
Number of txs	300
Avg. Confirmation Time (ms)	4.725672723
P99	7.645229199999999ms
P95	6.0821347999999995ms
P50	4.552252ms
Number of Invalid txs	0

Three local nodes

Number of nodes	3
Number of txs	900
Avg. Confirmation Time (ms)	24.767187944
P99	114.54338578999996ms
P95	34.76249535ms
P50	22.1581765ms
Number of Invalid txs	0