Tune tx trace spec #1695

ch1bo · 2024-10-09T07:26:38Z

Rewrite of the model part and tuning of the TxTrace stateful property-based test suite comprised of:

Allow for empty decrements, but identify DecrementValueNegative errors when constructing transactions.
Added coverage to model test TxTrace/all valid transactions
Changed how snapshots are generated and modeled: Instead of generating snapshots ad-hoc, we create them in a dedicated NewSnapshot action and only pick from a list of knownSnapshots when generating Close and other actions. This resembles better what is actually happening in the real world, where the adversary could only pick from a list of plausible snapshots. It also makes the arbitraryAction simpler and requires less preconditions. Lastly, this also results in less discarded values as was the original motivation in Tune snapshot generation in TxTrace tests #1524

CHANGELOG update not needed
Documentation update not needed
Haddocks updated
No new TODOs introduced or explained herafter
- Three very minor XXX comments how further refactoring could be done in TxTrace

github-actions · 2024-10-09T07:36:47Z

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2024-10-27 10:53:38.002257152 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	b512161ccb0652d7e9a0b540e4a3c808f73d6558a4bcabf374d85880	3969
νCommit	ea444d37d226e71eef73ac78d149750da977feb588900135bf9e8221	692
νHead	2253ddd95837c7aacc8635a971caaea743434152dd8dd2849bdf4162	10797
μHead	4d648ca239040b0e87901835aa11423e7aa3bd947ce6befe7db1bae8*	4508
νDeposit	703883df80fe589a8f64a4127057f28c9100ea9ebbb3e726c05e89ca	3096

The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per head.

`Init` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	5094	5.65	2.23	0.44
2	5301	7.18	2.84	0.46
3	5498	8.45	3.34	0.48
5	5901	11.49	4.55	0.54
10	6907	18.11	7.16	0.65
57	16356	83.00	32.84	1.78

`Commit` transaction costs

This uses ada-only outputs for better comparability.

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	569	10.84	4.26	0.29
2	756	14.31	5.80	0.34
3	944	17.92	7.39	0.39
5	1322	25.56	10.73	0.49
10	2253	47.11	19.97	0.77
19	3942	94.71	39.81	1.38

`CollectCom` transaction costs

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	560	19.84	7.58	0.39
2	114	671	27.94	10.64	0.48
3	170	782	36.72	13.98	0.58
4	227	893	43.03	16.38	0.66
5	281	1004	53.39	20.28	0.77
6	337	1116	62.94	23.89	0.88
7	395	1227	71.85	27.29	0.98
8	450	1342	78.03	29.68	1.05
9	504	1449	79.33	30.26	1.07

Cost of Decrement Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	659	18.40	8.07	0.39
2	777	19.35	9.15	0.41
3	906	21.17	10.70	0.44
5	1318	27.14	14.52	0.53
10	2035	35.01	21.47	0.68
48	7624	96.43	74.69	1.80

`Close` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	602	20.46	8.99	0.41
2	787	22.34	10.77	0.44
3	872	23.37	11.87	0.46
5	1244	27.02	15.42	0.53
10	2061	35.56	23.66	0.70
50	8121	98.80	86.18	1.93

`Contest` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	679	26.81	11.48	0.48
2	871	28.94	13.35	0.52
3	946	30.35	14.59	0.54
5	1274	34.11	17.98	0.61
10	1972	43.53	26.15	0.78
39	6336	99.45	75.14	1.77

`Abort` transaction costs

There is some variation due to the random mixture of initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4972	15.47	6.61	0.54
2	5079	19.19	8.13	0.59
3	5216	28.72	12.39	0.70
4	5435	36.33	15.85	0.80
5	5449	41.66	18.02	0.86
6	5720	49.67	21.76	0.96
7	5709	52.71	22.81	0.99
8	5942	63.96	27.95	1.13
9	5933	66.92	28.95	1.16
10	6094	73.97	32.09	1.25
11	6374	83.88	36.63	1.37
12	6543	88.72	38.75	1.43

`FanOut` transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
10	0	0	5090	10.38	4.35	0.49
10	1	57	5123	11.35	4.98	0.50
10	5	284	5258	15.80	7.78	0.57
10	10	569	5428	21.67	11.41	0.65
10	20	1138	5768	33.55	18.72	0.81
10	30	1708	6109	45.63	26.13	0.98
10	40	2276	6447	57.13	33.29	1.14
10	50	2848	6789	69.03	40.62	1.30
10	76	4325	7668	98.68	59.14	1.71

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2024-10-27 10:57:11.336003324 UTC

Baseline Scenario

Number of nodes	1
Number of txs	300
Avg. Confirmation Time (ms)	4.087476810
P99	7.162337629999995ms
P95	4.79381135ms
P50	3.9692665ms
Number of Invalid txs	0

Three local nodes

Number of nodes	3
Number of txs	900
Avg. Confirmation Time (ms)	23.459491512
P99	109.55972107999997ms
P95	32.17772809999998ms
P50	20.720039ms
Number of Invalid txs	0

github-actions · 2024-10-09T07:40:45Z

Test Results

7 files ±0 164 suites ±0 28m 25s ⏱️ + 2m 29s
558 tests ±0 552 ✅ ±0 6 💤 ±0 0 ❌ ±0
560 runs ±0 554 ✅ ±0 6 💤 ±0 0 ❌ ±0

Results for commit 461282e. ± Comparison against base commit f0d2fa0.

♻️ This comment has been updated with latest results.

This will ensure we have good coverage of the actual model test later. For now, the efficiency is way to low to checkCoverage on it too (it would go up to 6400 tests like prop_traces).

Changing the way we generate actions will help in guiding the generators based on "what an honest node would approve" and reduce discarded values / improve coverage.

Otherwise we simpliy pick from the models knownSnapshots

…rom it" This reverts commit b92f051.

The decrement validator allows for this and consequently this is a possible scenario: An adversary can construct decrement transactions with any snapshot we signed.

This is more consistent with what the off-chain code does.

Also disables some preconditions to decide whether we want to test this.

This also makes the output easier to read and removes several preconditions as we now expect construction to fail if we try to decrement too much.

This is reusing the models pendingDecommit for this purpose. However, it would be cleaner to separate the model states between Open and Closed.

It is a list of A-E elements only where shuffling of the elements represents off-chain transactions that maintain balance.

Using lower Confidence values, we should not see way very long tests at the cost of occasional false positives. In such cases, the actual coverage numbers and "interesting values" test should give an indication whether it was rightfully failing or would have taken just very long.

github-actions · 2024-10-27T10:54:58Z

Transaction cost differences

No cost or size differences found

v0d1ch

Thank you!

ch1bo linked an issue Oct 9, 2024 that may be closed by this pull request

Tune snapshot generation in TxTrace tests #1524

Closed

ch1bo force-pushed the tune-tx-trace-spec branch 2 times, most recently from c050c44 to 1ef0670 Compare October 10, 2024 18:00

ch1bo mentioned this pull request Oct 10, 2024

Tune snapshot generation in TxTrace tests #1524

Closed

ch1bo added 10 commits October 27, 2024 10:49

Add coverage to prop_runActions

ad0715a

This will ensure we have good coverage of the actual model test later. For now, the efficiency is way to low to checkCoverage on it too (it would go up to 6400 tests like prop_traces).

Increase likelihood of closed state with deltaUTxO

523e2a1

Increase likelihood of multiple commits

85405f3

WIP: Draft a plan

495ae2e

Changing the way we generate actions will help in guiding the generators based on "what an honest node would approve" and reduce discarded values / improve coverage.

Switch to individual snapshot production while in Open state

a3f63da

Only use existing genSnapshot when producing new snapshots

7e7003b

Otherwise we simpliy pick from the models knownSnapshots

Only keep knownSnapshots and derive latest version / number from it

20913ca

Revert "Only keep knownSnapshots and derive latest version / number f…

bd42c46

…rom it" This reverts commit b92f051.

Generate plausible sequences of snapshots (version and number)

31b614f

Allow for decrement with empty utxo to decommit

8b03153

The decrement validator allows for this and consequently this is a possible scenario: An adversary can construct decrement transactions with any snapshot we signed.

ch1bo force-pushed the tune-tx-trace-spec branch from e1dbc95 to a273c5b Compare October 27, 2024 09:49

ch1bo changed the title ~~WIP: Tune tx trace spec~~ Tune tx trace spec Oct 27, 2024

ch1bo requested review from ffakenz, v0d1ch, locallycompact and noonio October 27, 2024 10:08

ch1bo marked this pull request as ready for review October 27, 2024 10:09

ch1bo added 7 commits October 27, 2024 11:48

Generate Nothing if realWorldModelUTxO is empty

9c6da41

This is more consistent with what the off-chain code does.

Not shrink relevant NewSnapshots away

1280f85

Only update pendingDecommitUTxO in contest if not outdated

272268b

Make negative decrements fail gracefully

3d6ae8f

Also disables some preconditions to decide whether we want to test this.

Fix excessive discarding by generating better snapshots

f8312ae

This also makes the output easier to read and removes several preconditions as we now expect construction to fail if we try to decrement too much.

Small tuning on generators

cdebb64

Model toDecommit evolution as we would expect it

d105327

This is reusing the models pendingDecommit for this purpose. However, it would be cleaner to separate the model states between Open and Closed.

ch1bo added 4 commits October 27, 2024 11:48

Fix nextState of Close/Contest

3a150cd

Futher simplify ModelUTxO

d96d55c

It is a list of A-E elements only where shuffling of the elements represents off-chain transactions that maintain balance.

Drop old, commented code

ea603b9

ch1bo force-pushed the tune-tx-trace-spec branch from a273c5b to 461282e Compare October 27, 2024 10:49

v0d1ch approved these changes Oct 28, 2024

View reviewed changes

v0d1ch added this pull request to the merge queue Oct 28, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 28, 2024

v0d1ch added this pull request to the merge queue Oct 28, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 28, 2024

v0d1ch added this pull request to the merge queue Oct 28, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 28, 2024

v0d1ch merged commit 357ed03 into master Oct 30, 2024
29 checks passed

v0d1ch deleted the tune-tx-trace-spec branch October 30, 2024 14:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tune tx trace spec #1695

Tune tx trace spec #1695

ch1bo commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 27, 2024

v0d1ch left a comment

Tune tx trace spec #1695

Tune tx trace spec #1695

Conversation

ch1bo commented Oct 9, 2024 • edited Loading

github-actions bot commented Oct 9, 2024 • edited Loading

Transaction costs

Script summary

Init transaction costs

Commit transaction costs

CollectCom transaction costs

Cost of Decrement Transaction

Close transaction costs

Contest transaction costs

Abort transaction costs

FanOut transaction costs

End-to-end benchmark results

Baseline Scenario

Three local nodes

github-actions bot commented Oct 9, 2024 • edited Loading

Test Results

github-actions bot commented Oct 27, 2024

Transaction cost differences

v0d1ch left a comment

Choose a reason for hiding this comment

ch1bo commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 9, 2024 •

edited

Loading

`Init` transaction costs

`Commit` transaction costs

`CollectCom` transaction costs

`Close` transaction costs

`Contest` transaction costs

`Abort` transaction costs

`FanOut` transaction costs

github-actions bot commented Oct 9, 2024 •

edited

Loading