Refactor and reduce technical debt

[fxamacker]

Refactoring and reducing technical debt increases productivity and reduces risks.

It would be good to refactor before adding new features and optimizations like Lazy Decoding (which was postponed since 2023-Q4 OKR due to higher priority projects like Crescendo migrations and Atree releases).

I had postponed some refactoring as a tradeoff due to deadlines, higher priority work, etc.

Status

Completed. 🎉 Around 26074 insertions(+), 26002 deletions in ~97 files.

We made a lot of improvements and have bigger fish to fry.

This refactoring was briefly smoke tested on my desktop with no problems detected.

Also, backward compatibility tests succeeded using:

• 1 million mainnet blocks (on March 5, 2025).
• 1 million testnet blocks (on March 6, 2025).

Thank you @turbolent for all the PR reviews!

Progress

Delayed start: this was postponed until Jan 22, 2025 because of higher priority urgent work from Dec 20 - Jan 22 in private repos (debug/identify/fix security bug, contribute ~9500 lines of code that got ported to 2 public repos, detailed non-public PR reviews, etc.)

Since Jan 22, first phase refactored unit tests and smoke tests.

🔍 Refactoring tests

From Feb 12-14, second phase refactored non-test code.

🔍 Refactoring non-test code

Next, from Feb 24 - Mar 4: resume refactoring to simplify and use newer Go programming language features, etc. Also run brief smoke tests as sanity check.

🔍 Resume and wrap up refactoring (mostly) non-test code

Caveats

We should run longer smoke tests after all PRs are merged and before next release tag.

After this issue is completed, immediately starting work on Lazy Decoding (approved OKR postponed since 2023) will save time and money by avoiding re-rampup time.

[fxamacker]

This was postponed due to more urgent work (e.g. work with Cadence team since Friday Dec. 20, 2024).

[fxamacker]

The last PR (#534) and some of the prior PRs were briefly smoke tested with no problems detected. However, longer smoke tests should be run before next Atree release tag.

[fxamacker]

The last PR (#534) passed backward compatibility test using:

• 1 million mainnet blocks (on March 5, 2025).
• 1 million testnet blocks (on March 6, 2025).

[fxamacker]

Closing this as completed. I will reopen (or open new issue) if longer smoke testing on gcp finds anything.

So far, brief smoke tests on my desktop didn't find any problems.

cc: @j1010001