Test consistency #239

nerfZael · 2024-04-23T16:11:22Z

Closes #166

nerfZael · 2024-04-23T16:15:52Z

/workflows/benchmarks agents/token/send,agents/token/test_swap.py,agents/token/test_swap_and_send.py 1

github-actions · 2024-04-23T16:16:05Z

Finished benchmarks
Download artifacts

Test Run Summary

Run from: ./autotx/tests/agents/token/send,./autotx/tests/agents/token/test_swap.py,./autotx/tests/agents/token/test_swap_and_send.py
Base path: autotx/tests/agents/token/
Iterations: 1
Total Cost: $1.20
Total Success Rate (%): ${\color{red} \LARGE \texttt {93.75} \large \texttt { (94/-6)} }$

Detailed Results

Test Name	Success Rate (%)	Passes	Fails	Avg Time	Avg Cost
`send/test_send.py::test_send_erc20`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	26s	$0.06
`send/test_send.py::test_send_erc20_parallel`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	26s	$0.07
`send/test_send.py::test_send_eth_multiple`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	39s	$0.08
`send/test_send.py::test_send_native`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	15s	$0.06
`send/test_send.py::test_send_native_sequential`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	31s	$0.12
`test_swap.py::test_swap_complex_1`	${\color{red} \large \texttt {0} \normalsize \texttt {(-100)} }$	${\color{red} \large \texttt {0}}$	${\color{red} \large \texttt {1}}$	39s	$0.08
`test_swap.py::test_swap_complex_2`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	30s	$0.08
`test_swap.py::test_swap_multiple_1`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	27s	$0.06
`test_swap.py::test_swap_multiple_2`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	28s	$0.06
`test_swap.py::test_swap_native`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	29s	$0.06
`test_swap.py::test_swap_triple`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	54s	$0.06
`test_swap.py::test_swap_with_non_default_token`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	26s	$0.06
`test_swap_and_send.py::test_send_and_swap_complex`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	57s	$0.08
`test_swap_and_send.py::test_send_and_swap_simple`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	27s	$0.08
`test_swap_and_send.py::test_swap_and_send_complex`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	37s	$0.11
`test_swap_and_send.py::test_swap_and_send_simple`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {1}}$	${\color{none} \large \texttt {0}}$	26s	$0.08

Total run time: 8.61 minutes

nerfZael · 2024-04-23T20:30:34Z

/workflows/benchmarks agents/token/test_swap.py,agents/token/test_swap_and_send.py

github-actions · 2024-04-23T20:30:46Z

Finished benchmarks
Download artifacts

Test Run Summary

Run from: ./autotx/tests/agents/token/test_swap.py,./autotx/tests/agents/token/test_swap_and_send.py
Base path: autotx/tests/agents/token/test_swap
Iterations: 5
Total Cost: $4.03
Total Success Rate (%): ${\color{none} \LARGE \texttt {100.00} \large \texttt {} }$

Detailed Results

Test Name	Success Rate (%)	Passes	Fails	Avg Time	Avg Cost
`.py::test_swap_complex_1`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	32s	$0.08
`.py::test_swap_complex_2`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	34s	$0.07
`.py::test_swap_multiple_1`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	31s	$0.06
`.py::test_swap_multiple_2`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	33s	$0.06
`.py::test_swap_native`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	25s	$0.06
`.py::test_swap_triple`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	40s	$0.06
`.py::test_swap_with_non_default_token`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	24s	$0.06
`_and_send.py::test_send_and_swap_complex`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	41s	$0.11
`_and_send.py::test_send_and_swap_simple`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	25s	$0.08
`_and_send.py::test_swap_and_send_complex`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	39s	$0.10
`_and_send.py::test_swap_and_send_simple`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	27s	$0.08

Total run time: 29.16 minutes

nerfZael · 2024-04-23T21:00:48Z

/workflows/benchmarks agents/token/research

github-actions · 2024-04-23T21:01:02Z

Finished benchmarks
Download artifacts

Test Run Summary

Run from: ./autotx/tests/agents/token/research
Base path: autotx/tests/agents/token/research/test_
Iterations: 5
Total Cost: $17.81
Total Success Rate (%): ${\color{red} \LARGE \texttt {83.08} \large \texttt { (83/-10)} }$

Detailed Results

Test Name	Success Rate (%)	Passes	Fails	Avg Time	Avg Cost
`advanced.py::test_research_and_swap_many_tokens_subjective_complex`	${\color{lightgreen} \large \texttt {20} \normalsize \texttt {(+10)} }$	${\color{lightgreen} \large \texttt {1}}$	${\color{lightgreen} \large \texttt {4}}$	2.23m	$0.68
`advanced.py::test_research_and_swap_many_tokens_subjective_simple`	${\color{red} \large \texttt {40} \normalsize \texttt {(-60)} }$	${\color{red} \large \texttt {2}}$	${\color{red} \large \texttt {3}}$	1.13m	$0.63
`research.py::test_get_token_exchanges`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	30s	$0.09
`research.py::test_get_top_5_memecoins`	${\color{red} \large \texttt {80} \normalsize \texttt {(-20)} }$	${\color{red} \large \texttt {4}}$	${\color{red} \large \texttt {1}}$	35s	$0.17
`research.py::test_get_top_5_memecoins_in_optimism`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	26s	$0.17
`research.py::test_get_top_5_most_traded_tokens_from_l1`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	29s	$0.19
`research.py::test_get_top_5_tokens_from_base`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	30s	$0.18
`research.py::test_price_change_information`	${\color{red} \large \texttt {80} \normalsize \texttt {(-20)} }$	${\color{red} \large \texttt {4}}$	${\color{red} \large \texttt {1}}$	15s	$0.08
`research_and_swap.py::test_research_and_buy_multiple`	${\color{red} \large \texttt {80} \normalsize \texttt {(-20)} }$	${\color{red} \large \texttt {4}}$	${\color{red} \large \texttt {1}}$	44s	$0.25
`research_and_swap.py::test_research_and_buy_one`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	31s	$0.23
`research_swap_and_send.py::test_research_buy_multiple_send_multiple`	${\color{red} \large \texttt {80} \normalsize \texttt {(-20)} }$	${\color{red} \large \texttt {4}}$	${\color{red} \large \texttt {1}}$	1.02m	$0.33
`research_swap_and_send.py::test_research_buy_one_send_multiple`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	42s	$0.30
`research_swap_and_send.py::test_research_buy_one_send_one`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	40s	$0.27

Total run time: 48.74 minutes

nerfZael · 2024-04-24T15:28:51Z

/workflows/benchmarks agents/token/research/test_research.py::test_get_top_5_memecoins,agents/token/research/test_research.py::test_get_top_5_memecoins,agents/token/research/test_research.py::test_price_change_informationm,agents/token/research/test_research_and_swap.py::test_research_and_buy_multiple,agents/token/research/test_research_swap_and_send.py::test_research_buy_multiple_send_multiple 5

github-actions · 2024-04-24T15:29:05Z

Finished benchmarks
Download artifacts

Test Run Summary

Run from: ./autotx/tests/agents/token/research/test_research.py::test_get_top_5_memecoins,./autotx/tests/agents/token/research/test_research.py::test_get_top_5_memecoins,./autotx/tests/agents/token/research/test_research.py::test_price_change_informationm,./autotx/tests/agents/token/research/test_research_and_swap.py::test_research_and_buy_multiple,./autotx/tests/agents/token/research/test_research_swap_and_send.py::test_research_buy_multiple_send_multiple
Base path: autotx/tests/agents/token/research/test_research
Iterations: 5
Total Cost: $3.71
Total Success Rate (%): ${\color{red} \LARGE \texttt {93.33} \large \texttt { (93/-7)} }$

Detailed Results

Test Name	Success Rate (%)	Passes	Fails	Avg Time	Avg Cost
`.py::test_get_top_5_memecoins`	${\color{red} \large \texttt {80} \normalsize \texttt {(-20)} }$	${\color{red} \large \texttt {4}}$	${\color{red} \large \texttt {1}}$	40s	$0.19
`_and_swap.py::test_research_and_buy_multiple`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	44s	$0.20
`_swap_and_send.py::test_research_buy_multiple_send_multiple`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	1.16m	$0.35

Total run time: 12.80 minutes

nerfZael · 2024-04-24T16:24:41Z

/workflows/benchmarks agents/token/research/test_research.py::test_get_top_5_memecoins,agents/token/research/test_research.py::test_price_change_information

github-actions · 2024-04-24T16:24:54Z

Finished benchmarks
Download artifacts

Test Run Summary

Run from: ./autotx/tests/agents/token/research/test_research.py::test_get_top_5_memecoins,./autotx/tests/agents/token/research/test_research.py::test_price_change_information
Base path: autotx/tests/agents/token/research/test_research.py::
Iterations: 5
Total Cost: $1.35
Total Success Rate (%): ${\color{none} \LARGE \texttt {100.00} \large \texttt {} }$

Detailed Results

Test Name	Success Rate (%)	Passes	Fails	Avg Time	Avg Cost
`test_get_top_5_memecoins`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	33s	$0.19
`test_price_change_information`	${\color{none} \large \texttt {100} \normalsize \texttt {} }$	${\color{none} \large \texttt {5}}$	${\color{none} \large \texttt {0}}$	21s	$0.08

Total run time: 4.47 minutes

nerfZael · 2024-04-24T17:25:20Z

/workflows/benchmarks agents/token/research/test_advanced.py::test_research_and_swap_many_tokens_subjective_simple 5

github-actions · 2024-04-24T17:25:32Z

Finished benchmarks
Download artifacts

Test Run Summary

Run from: ./autotx/tests/agents/token/research/test_advanced.py::test_research_and_swap_many_tokens_subjective_simple
Iterations: 5
Total Cost: $2.59
Total Success Rate (%): ${\color{red} \LARGE \texttt {80.00} \large \texttt { (80/-20)} }$

Detailed Results

Test Name	Success Rate (%)	Passes	Fails	Avg Time	Avg Cost
`test_research_and_swap_many_tokens_subjective_simple`	${\color{red} \large \texttt {80} \normalsize \texttt {(-20)} }$	${\color{red} \large \texttt {4}}$	${\color{red} \large \texttt {1}}$	1.75m	$0.52

Total run time: 8.76 minutes

nerfZael · 2024-04-25T12:26:48Z

/workflows/benchmarks agents/token/research/test_research.py::test_get_top_5_memecoins,agents/token/research/test_advanced.py::test_research_and_swap_many_tokens_subjective_simple 10

github-actions · 2024-04-25T12:27:00Z

Finished benchmarks
Download artifacts

Test Run Summary

Run from: ./autotx/tests/agents/token/research/test_research.py::test_get_top_5_memecoins,./autotx/tests/agents/token/research/test_advanced.py::test_research_and_swap_many_tokens_subjective_simple
Base path: autotx/tests/agents/token/research/test_
Iterations: 10
Total Cost: $7.09
Total Success Rate (%): ${\color{none} \LARGE \texttt {90.00} \large \texttt {} }$

Detailed Results

Test Name	Success Rate (%)	Passes	Fails	Avg Time	Avg Cost
`advanced.py::test_research_and_swap_many_tokens_subjective_simple`	${\color{red} \large \texttt {80} \normalsize \texttt {(-10)} }$	${\color{red} \large \texttt {8}}$	${\color{red} \large \texttt {2}}$	1.50m	$0.51
`research.py::test_get_top_5_memecoins`	${\color{lightgreen} \large \texttt {100} \normalsize \texttt {(+10)} }$	${\color{lightgreen} \large \texttt {10}}$	${\color{lightgreen} \large \texttt {0}}$	45s	$0.20

Total run time: 22.46 minutes

autotx/agents/SwapTokensAgent.py

autotx/tests/agents/token/research/test_advanced.py

cbrzn · 2024-04-25T15:08:31Z

autotx/tests/agents/token/research/test_research_swap_and_send.py

+
+    receiver = test_accounts[0]
+
+    shib_address = ETHAddress(auto_tx.network.tokens["shib"])


we have a lot of tests that fetch the meme coin with largest market cap - since this is dynamic, should we get the data from coingecko in each test?

You're right, I just made an issue for it here: #247
Seems a bit lower prio though since the highest market cap coins won't likely change that soon

…ency

nerfZael added 2 commits April 23, 2024 18:10

test refactor

e841e19

prompt fixes

c8afae9

agentcoinorg deleted a comment from github-actions bot Apr 23, 2024

minimized swap prompt while fixing issue

b5d8644

agentcoinorg deleted a comment from github-actions bot Apr 23, 2024

agentcoinorg deleted a comment from github-actions bot Apr 24, 2024

updated couple of tests to 90% since they're inconsistent

9610536

nerfZael changed the title ~~[WIP] Test consistency~~ Test consistency Apr 25, 2024

nerfZael requested review from cbrzn and dOrgJelli and removed request for cbrzn April 25, 2024 12:25

cbrzn reviewed Apr 25, 2024

View reviewed changes

nerfZael added 3 commits April 26, 2024 20:48

Merge remote-tracking branch 'origin/main' into nerfzael/test-consist…

9684447

…ency

Merge remote-tracking branch 'origin/main' into nerfzael/test-consist…

f3a75b9

…ency

typos

f5c7ddc

nerfZael requested a review from cbrzn April 30, 2024 16:10

cbrzn approved these changes Apr 30, 2024

View reviewed changes

cbrzn merged commit d94506c into main Apr 30, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test consistency #239

Test consistency #239

nerfZael commented Apr 23, 2024 •

edited

Loading

nerfZael commented Apr 23, 2024

github-actions bot commented Apr 23, 2024 •

edited

Loading

nerfZael commented Apr 23, 2024

github-actions bot commented Apr 23, 2024 •

edited

Loading

nerfZael commented Apr 23, 2024

github-actions bot commented Apr 23, 2024 •

edited

Loading

nerfZael commented Apr 24, 2024

github-actions bot commented Apr 24, 2024 •

edited

Loading

nerfZael commented Apr 24, 2024

github-actions bot commented Apr 24, 2024 •

edited

Loading

nerfZael commented Apr 24, 2024

github-actions bot commented Apr 24, 2024 •

edited

Loading

nerfZael commented Apr 25, 2024

github-actions bot commented Apr 25, 2024 •

edited

Loading

cbrzn Apr 25, 2024

nerfZael Apr 30, 2024


		receiver = test_accounts[0]

		shib_address = ETHAddress(auto_tx.network.tokens["shib"])

Test consistency #239

Test consistency #239

Conversation

nerfZael commented Apr 23, 2024 • edited Loading

nerfZael commented Apr 23, 2024

github-actions bot commented Apr 23, 2024 • edited Loading

Test Run Summary

Detailed Results

nerfZael commented Apr 23, 2024

github-actions bot commented Apr 23, 2024 • edited Loading

Test Run Summary

Detailed Results

nerfZael commented Apr 23, 2024

github-actions bot commented Apr 23, 2024 • edited Loading

Test Run Summary

Detailed Results

nerfZael commented Apr 24, 2024

github-actions bot commented Apr 24, 2024 • edited Loading

Test Run Summary

Detailed Results

nerfZael commented Apr 24, 2024

github-actions bot commented Apr 24, 2024 • edited Loading

Test Run Summary

Detailed Results

nerfZael commented Apr 24, 2024

github-actions bot commented Apr 24, 2024 • edited Loading

Test Run Summary

Detailed Results

nerfZael commented Apr 25, 2024

github-actions bot commented Apr 25, 2024 • edited Loading

Test Run Summary

Detailed Results

cbrzn Apr 25, 2024

Choose a reason for hiding this comment

nerfZael Apr 30, 2024

Choose a reason for hiding this comment

nerfZael commented Apr 23, 2024 •

edited

Loading

github-actions bot commented Apr 23, 2024 •

edited

Loading

github-actions bot commented Apr 23, 2024 •

edited

Loading

github-actions bot commented Apr 23, 2024 •

edited

Loading

github-actions bot commented Apr 24, 2024 •

edited

Loading

github-actions bot commented Apr 24, 2024 •

edited

Loading

github-actions bot commented Apr 24, 2024 •

edited

Loading

github-actions bot commented Apr 25, 2024 •

edited

Loading