Release V0.9.1: Allow import of graph files when imported from other scripts

[Tritium-VLK]

• Use json files instead of generating new data on chain load.
• Twiddle setup.py to include gql files in sub-dirs. Update version to 0.9.1
• Abstract use of web3 into utils to allow for different names for functions in various version and make this module able to run with both Web3 v5 and Web3 v6.
• Use python path objects to find gql files to make sure we are always looking in the right place.

Please squash merge

[Tritium-VLK]

@gosuto-inzasheru some other places where I am including this are crashing when trying to build the core pools data. So this PR tries to make the pathing a bit robust (although build_core_pools still seems to be crashing when you run it from an import), and uses the cached core pools unless you explicitly ask it otherwise.

[Tritium-VLK]

see: https://github.com/BalancerMaxis/protocol_fee_allocator/actions/runs/8302243127/attempts/3 for an example

[Tritium-VLK]

This still isn't working. Have to figure out how to get setup.py to include the gql files when you import the package.
@gosuto-inzasheru maybe you can figure it out, I will take another stab at it as well.

[Tritium-VLK]

Example successful run from another repo. https://github.com/BalancerMaxis/protocol_fee_allocator/actions/runs/8332269118

[Tritium-VLK]

@gosuto-inzasheru ok this all looks good to me now, except the core pools tests are failing. A bit unsure/unclear on that.

[gosuto-inzasheru]

works now as a standalone!

Python 3.10.10 (main, Jul  7 2023, 13:26:12) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bal_addresses
>>> b = bal_addresses.pools_gauges.BalPoolsGauges('mainnet', False)
Pool 0x2191df821c198600499aa1f0031b1a7514d7a7d9000200000000000000000639 on mainnet has no preferential gauge
Pool 0x32296969ef14eb0c6d29669c550d4a0449130230000200000000000000000080 on mainnet has no preferential gauge
Pool 0xb9debddf1d894c79d2b2d09f819ff9b856fca55200000000000000000000062a on mainnet has no preferential gauge
>>> b.
b.build_core_pools()                          b.get_gauge_deposit_shares(                   b.is_core_pool(
b.chain                                       b.get_last_join_exit(                         b.is_pool_exempt_from_yield_fee(
b.core_pools                                  b.get_liquid_pools_with_protocol_yield_fee()  b.query_preferential_gauges(
b.get_bpt_balances(                           b.has_alive_preferential_gauge(               b.subgraph

[Tritium-VLK]

@gosuto-inzasheru https://github.com/BalancerMaxis/bal_addresses/actions/runs/8349543802/job/22853956890

Are you sure all the subgraph host urls are resolving correctly?

Maybe also just a downtime, can try running gen_addressbooks on this branch again later.

[Tritium-VLK]

@gosuto-inzasheru ok so that seemed to just be temporary. Ran again later and it worked, but core pools still failing due to trying to generate on testnets. I'm kinda finsihing up for the day, so won't get ot this today, but can take a look tomororw (afternoon/evenint CET) if you haven't figured it out yet. We're getting close I think to getting a few of these guys merged.

[gosuto-inzasheru]

sorted testnet issues, action runs successfully now: https://github.com/BalancerMaxis/bal_addresses/actions/runs/8356226775/job/22872904183

[Tritium-VLK]

Test results after merging in main with root gauges addition and dealign with merge conflicts:
#237
https://github.com/BalancerMaxis/bal_addresses/actions/runs/8362674396

[Tritium-VLK]

@gosuto-inzasheru is it me or are the tests just failing cuz teh arbi subgraph is flakey?

[Tritium-VLK]

@gosuto-inzasheru is it me or are the tests just failing cuz teh arbi subgraph is flakey?

Looking into it more seems like python 3.9 doesn't work anymore. I think this is probably somewhat important as it is used in places. Maybe we just need to get over that.

[Tritium-VLK]

@gosuto-inzasheru is it me or are the tests just failing cuz teh arbi subgraph is flakey?

Looking into it more seems like python 3.9 doesn't work anymore. I think this is probably somewhat important as it is used in places. Maybe we just need to get over that.

Ok dug in a bit more. It looks liek the issue here is that the gql client library we are using to fetch these gql files is not working with python v3.9.

Some thoughts on this:

• I think we want this base package to work with as many version as possible.
• The problem is in a build command, users of the package don't tend to run this command by default and get this data from the eventually consistent data in the outputs/ dir. For that reason this could be ok as is, but would be nice if you got a better error when trying to run build_core_pools() with v3.9. Could say something like "This feature only works in V3.10, please use the prepopulated core_pools data or upgrade to Python 3.10"
• We want this package to be as robust as possible. What do we get from using these gql libraries? Could we solve the problem by skipping them and just using requests to execute queries?

[gosuto-inzasheru]

@gosuto-inzasheru is it me or are the tests just failing cuz teh arbi subgraph is flakey?

i think you are right, arbitrum subgraph was wonky. tests pass now without issue, also on python 3.9

[Tritium-VLK]

@gosuto-inzasheru you just merged this, but we didn't deal with the gql, v3.9 problem.

[gosuto-inzasheru]

@gosuto-inzasheru you just merged this, but we didn't deal with the gql, v3.9 problem.

i cant reproduce. this is in a fresh env using latest commit on main branch, installed as a standalone:

Python 3.9.16 (main, Apr  4 2023, 00:13:23) 
[Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bal_addresses
>>> b = bal_addresses.pools_gauges.BalPoolsGauges('arbitrum', False)
>>> b.get_bpt_balances('0x451b0afd69ace11ec0ac339033d54d2543b088a80000000000000000000004d5', 192721622)
{'0x40F6bE0b26B229C4C3b2C3344e91f8eEe22a62fF': 1128038.509832892, '0x54B2bC420f9d59A4a4Eda1c5121D6FA43Ad80193': 580.8081601395012, '0xaf2A0aCd19f914DEc99CD786d2C1676E93EF1b00': 428.81987340396284, '0x72eb41AE7f943445Da9932662d31A06D56A3cfAa': 96.30946108563089, '0x88C88bE09f5b934F4D4c5B3d8681b66128BE9067': 14.771170411672987, '0xce88686553686DA562CE7Cea497CE749DA109f9F': 1.601849434871351, '0xd08e942f510668ACb7c07ba3DfB1daF6d2d01f0e': 1.0004345105069716, '0x8EF8349A5a9665d9EFFF4F27E5dFB1D83213F39E': 0.4944554311219424, '0x034d6Ce667956091d6a809687FAE51D6959D0499': 0.007890658567123414, '0x3Cd2cc9E6C2Aba88Aa7fEFF88e465Cb93FE1D232': 0.003120882650862887, '0x0000000000000000000000000000000000000000': 1e-12, '0xDef8fA37eb9f72aa6b0cc7712D81a41E68DCca1d': 1e-18}

[gosuto-inzasheru]

also tests are passing on python 3.9. can you give a reproducable situation, or write a test that highlights the issue you are talking about?

[Tritium-VLK]

also tests are passing on python 3.9. can you give a reproducable situation, or write a test that highlights the issue you are talking about?

Hmm you're right. Yesterday 3.9 was failing with somehow not getting json data from some reqeust call I thought, but I can't find those runs in the jobs and they seem to be passing now. Will tag this and test it a bit. Thanks.