✨ feat: Week 12

[raxhvl]

No description provided.

[taxmeifyoucan]

Looking at your script, you are manually calling the RPC https://github.com/raxhvl/evm-memory-analysis/blob/4bb22ec09c12e957f83bc5f7102a95231dad110e/memory-analysis/trace_memory.py#L15

That's not really practical or effective. I would strongly suggest to use web3.py instead https://web3py.readthedocs.io/en/stable/quickstart.html

[raxhvl]

Exactly what I tried yesterday.

web3.py has single threaded concurrency and I was able to get the block time down to 40 sec. Inspired by their code, I then rewrote the script using python's native ThreadPoolExecutor and now I'm able to process a block in 16 sec.

[taxmeifyoucan]

That's awesome! Many ways to improve efficiency.

One more thing I wanted to comment on in your update:

archive requires over 12TB of storage.

This is old geth database approach which is basically deprecated at this point. Erigon and reth can do the same with faster rpc processing in >3TB

[raxhvl]

Ahh.. I picked the data from geth's docs. You know i'm trying to archive node :p

File size is where I'm also currently struggling at. Currently each block is producing 5mb (500G for 100K blocks) of debug data. The flat csv file has redundant data (block number, tx hash, tx gas) for each frame:

block,tx_hash,tx_gas,to,call_depth,memory_instruction,memory_access_offset,memory_gas_cost,pre_active_memory_size,post_active_memory_size,memory_expansion
20569001,0x82f6413a2658ebb83f27e44fe1e815ec0979a7dcc0bc9dbdbbbe058d547195a6,1051297,0x1f2f10d1c40777ae1da742455c65828ff36df387,1,MSTORE,0,6,0,32,32
20569001,0x82f6413a2658ebb83f27e44fe1e815ec0979a7dcc0bc9dbdbbbe058d547195a6,1051297,0x1f2f10d1c40777ae1da742455c65828ff36df387,1,MSTORE,20,6,32,64,32
20569001,0x82f6413a2658ebb83f27e44fe1e815ec0979a7dcc0bc9dbdbbbe058d547195a6,1051297,0x1f2f10d1c40777ae1da742455c65828ff36df387,1,MSTORE,21,3,96,96,0
20569001,0x82f6413a2658ebb83f27e44fe1e815ec0979a7dcc0bc9dbdbbbe058d547195a6,1051297,0x1f2f10d1c40777ae1da742455c65828ff36df387,1,MSTORE,4,3,96,96,0
20569001,0x82f6413a2658ebb83f27e44fe1e815ec0979a7dcc0bc9dbdbbbe058d547195a6,1051297,0x1f2f10d1c40777ae1da742455c65828ff36df387,2,MSTORE,64,12,0,96,96
20569001,0x82f6413a2658ebb83f27e44fe1e815ec0979a7dcc0bc9dbdbbbe058d547195a6,1051297,0x1f2f10d1c40777ae1da742455c65828ff36df387,2,MLOAD,64,3,96,96,0
20569001,0x82f6413a2658ebb83f27e44fe1e815ec0979a7dcc0bc9dbdbbbe058d547195a6,1051297,0x1f2f10d1c40777ae1da742455c65828ff36df387,2,MSTORE,64,3,96,96,0

Trying ways to compress this. Once i omit the redundant I/O to file I should (hopefully) beat mainnet block time of 12 sec !