v0.28.6

Mainnet 20

This release includes the following major changes,

1. Laying the ground work for Permissionless AN with several upgrades to the underlying peer-to-peer networking layer to make it more Byzantine fault tolerant.
2. A fully revamped execution node checkpointing process (v6) that is six time faster, eliminates execution node memory spikes and reduces spork downtime.
3. Rolling upgrade support for Execution and Verification nodes, working towards zero-downtime network upgrades in the future.
4. FVM performance improvements.

💥 Breaking Changes

Cadence type checking of resource has been fixed and significantly improved. Correct Cadence code should continue to function as before. However, any Cadence code that relied on implementation bugs must be corrected (onflow/cadence#2033)

Execution node

• [Exec] Add logging to collection fetch and handling by @SaveTheRbtz in #3316
• [Execution] Adding more logging and a fix to block upload retry by @Tonix517 in #3315
• [Execution] Fix TestScriptStorageMutationsDiscarded timeout by @SaveTheRbtz in #3109
• [Execution] Fixing improper use of WaitGroup in a goroutine for upload retry by @Tonix517 in #3285
• [Execution] Return error when closing gcp connection fails by @peterargue in #3290
• [Execution] Worker-based Chunk Data Pack Requests by @yhassanzadeh13 in #2951
• [Execution] state migration clean up by @ramtinms in #3192
• [EN Performance] Optimize checkpoint serialization for -37GB operational RAM, -2.7 minutes duration, -19.6 million allocs (50% fewer allocs) by @fxamacker in #3050
• [EN Performance] Reduce memory used for ledger.Payload by 32+ GB, eliminate 1+ billion allocs/op, speedup various ops by @fxamacker in #2930
• [Execution] Backport "Optimize GetEventsByBlockIDs" to master by @zhangchiqing in #3211
• [Execution] Fix error logging in script execution - port to master by @janezpodhostnik in #3071
• Refactor execution node builder (part 1) by @pattyshack in #3130
• Block computation result upload Retry implementation by @Tonix517 in https://github.com/onflow/flow-go/pull/
• Execution data format update by @smnzhu in #2841
• [Execution] remove duplicate blobservice on execution nodes by @peterargue in #3418

Sync engine

• [SyncEngine] not request pending ancestors by @zhangchiqing in #2879
• [ExecutoinStateSyn] Add bitswap metrics by @smnzhu in #2932
• Implement rate limite blob service by @smnzhu in #2859

Crypto

• [Crypto] DKG robustness against invalid messages by @tarakby in #2970
• [Crypto] Reformat using go 1.19 fmt by @SaveTheRbtz in #3141
• [Crypto] new sentinel errors and doc improvement by @tarakby in #2861
• [Crypto] update Go dependencies by @tarakby in #3085

Access and Observer node

• [Access] Option to Disable Conn Pooling by @lolpuddle in #2836
• [Access] Fix race condition in AN execution receipt metrics collection by @peterargue in #2957
• [Access] Added rpc metrics flag to Observer Node Builder by @danielholmes839 in #2973
• Observer metrics and logs using apiproxy by @danielholmes839 in #2920
• Crash Access Node if disk is full by @koko1123 in #1956
• Fix observer integration ports by @danielholmes839 in #2886

Performance benchmarking

• [Benchmark] Ability to dynamically adjust TPS by @SaveTheRbtz in #3246
• [Benchmark] Cleanup logging by @SaveTheRbtz in #3293
• [Benchmark] Expose execution rate from the loader by @SaveTheRbtz in #3080
• [Benchmark] Faster stopping of workers by @SaveTheRbtz in #3079
• [Benchmark] Fix crash on stats pusher exit by @SaveTheRbtz in #3089
• [Benchmark] Follow sealed blocks by @SaveTheRbtz in #3074
• [Benchmark] Group parameters to the loader library by @SaveTheRbtz in #3091
• [Benchmark] Keep prometheus metrics in-sync with TPS by @SaveTheRbtz in #3340
• [Benchmark] Remove noisy benchmarks by @SaveTheRbtz in #3090
• [Benchmark] Remove race in Digest() print by @SaveTheRbtz in #3076
• [Benchmark] Simplify follower inner loop by @SaveTheRbtz in #3078
• [Benchmark] Split TPS flags into initial and max by @SaveTheRbtz in #3292
• [Benchmark] upload data to BigQuery directly by @SaveTheRbtz in #3241
• [Performance] add fast path to programs cleanup by @SaveTheRbtz in #2982
• [Performance] initialize profiler after the state init by @SaveTheRbtz in #2985
• [Performance] run computation benchmarks on commit by @SaveTheRbtz in #2983
• [Benchmark] Add ability to auto-detect maximum TPS (#3338) @SaveTheRbtz
• Rewrite benchmarks for storing/loading checkpoint by @fxamacker in #3377

Cadence and FVM Performance improvement

• [FVM] Add expected event count to fuzzer by @janezpodhostnik in #3131
• [FVM] Add interaction limit to FVM fuzzing by @janezpodhostnik in #3114
• [FVM] Adding storage metering to Meter by @Tonix517 in #3236
• [FVM] Basic fuzzing and minor test change by @janezpodhostnik in #3112
• [FVM] Cleanup fvm logging by @janezpodhostnik in #3117
• [FVM] Disable limits on final state merge by @janezpodhostnik in #3182
• [FVM] Error Cleanup by @janezpodhostnik in #3244
• [FVM] Events in scripts do nothing instead of causing errors by @janezpodhostnik in #3183
• [FVM] Fix account reporter memory limit - master by @janezpodhostnik in #3105
• [FVM] Handle cadence runtime error early by @janezpodhostnik in #3204
• [FVM] Log err on TX failure by @janezpodhostnik in #3230
• [FVM] Move HandleRuntimeError to ContractFunctionInvoker by @janezpodhostnik in #3059
• [FVM] Moving event emit size metering logic to Meter interface by @Tonix517 in #3268
• [FVM] Pass context by value by @janezpodhostnik in #3220
• [FVM] Refactor checkAccountsAreNotFrozen by @janezpodhostnik in #3235
• [FVM] Remove old FVM errors by @janezpodhostnik in #3351
• [FVM] Remove old quick fixes by @janezpodhostnik in #3219
• [FVM] Replacing existing metering in state.go with new interface by @Tonix517 in #3249
• [FVM] Set ReusableCadenceRuntimePoolSize in bench tests by @janezpodhostnik in #3295
• [FVM] Simplifying FVM meter interface and implementation by @Tonix517 in #3252
• [FVM] Use embed for cadence scripts in blueprints by @janezpodhostnik in #3148
• Create one cadence runtime per ReusableCadenceRuntime object by @pattyshack in #3184
• Create virtual machine / cadence runtime inside computation.Manager by @pattyshack in #3132
• Reuse cadence runtime environment by @pattyshack in #3101
• Replace error/failure structs with CodedError/CodedFailure (Part 1) by @pattyshack in #3289
• Replace error/failure structs with CodedError/CodedFailure (Part 2) by @pattyshack in https://github.com/onflow/flow-go/pull/
• Reduce environment circular dependency by @pattyshack in #3223
• Refactor AccountKeyUpdater by @pattyshack in #3222
• Refactor GetAccountKey into AccountKeyReader by @pattyshack in #3186
• Refactor ProgramsCache into ChainPrograms by @pattyshack in #3205
• Refactor UUIDGenerator into environment by @pattyshack in #3057
• Refactor account freezer by @pattyshack in #3191
• Refactor contract reader into environment by @pattyshack in #3160
• Refactor contract updater by @pattyshack in #3193
• Refactor create account logic into AccountCreator by @pattyshack in #3209
• Refactor environment Programs by @pattyshack in #3250
• Refactor environment event emitter by @pattyshack in #3120
• Refactor environment meter into separate objects by @pattyshack in #2999
• Perform stricter unexpected nested transaction checks. by @pattyshack in #3269
• Clean up environment api and structs by @pattyshack in #3238
• Cleanup meter test error casting by @pattyshack in #3284
• Combine Failure into CodedError by @pattyshack in #3322
• Combined Error and CodedError interface. by @pattyshack in #3309
• Convert component/module initialization closures into methods by @pattyshack in #3134
• Convert more context fields into environment params by @pattyshack in #3272
• Convert programs cache into mini occ databases by @pattyshack in #3088
• Convert programs.Programs into an optional argument for fvm.GetAccount(V2) by @pattyshack in #3214
• Convert programs.Programs into an optional argument for fvm.Run(V2) by @pattyshack in #3194
• Dedup more environment code by @pattyshack in #2979
• Dedup setExecutionParameters by @pattyshack in #2942
• Deprecate/delete FVMInternalError by @pattyshack in #3281
• Don't pollute global program caches with non-address programs by @pattyshack in #2918
• Ensure cadence don't access unexpected operations while parsing programs (part 1) by @pattyshack in #3334
• Expose more StateHolder methods by @pattyshack in #2953
• Finish migrating execution node initialization closure to method by @pattyshack in #3181
• Fix HandleRuntimeError by @pattyshack in #3025
• Fix logger. Forgot to set the field during initialization by @pattyshack in #3124
• Fix more fvm tests to use incrementing tx index. by @pattyshack in #3234
• Fix tests to use correct increasing tx index (attempt #2) by @pattyshack in #3086
• Fix tests to use correct increasing tx index by @pattyshack in #3084
• Fix transaction invoker to not early exit once transaction started by @pattyshack in #2966
• Fully initialize meter parameters before transaction execution by @pattyshack in #2971
• Fully replace Programs with BlockPrograms by @pattyshack in #3239
• Internalize state/meter creation by @pattyshack in #2981
• Invalidate program cache entries when accounts are freezed. by @pattyshack in #2855
• Make ContractFunctionInvoker stateless by @pattyshack in #3159
• Make NewMeter's parameters non-optional by @pattyshack in #2947
• Make NewState Parameters non-optional by @pattyshack in #2948
• Merge handler/metrics.go into environment/program_logger.go by @pattyshack in #3116
• Merge meter (and interaction) unconditionally by @pattyshack in #3144
• Move Context fields into EnvironmentParams (Part 1) by @pattyshack in #3266
• Move accounts* from state to environment by @pattyshack in #3127
• Move address generator from state to environment + some renaming by @pattyshack in #3125
• Move environment specific meter constants from meter/ to environment/ by @pattyshack in #3128
• Move fully refactored code into new environment subpackage by @pattyshack in #2980
• Move invokeMetaTransaction from fvm.go into bootstrap.go by @pattyshack in #3210
• Move remaining ContractUpdater's code out of environment by @pattyshack in #3229
• Move various files out of fvm/ into subpackages by @pattyshack in #3195
• Refactor system contracts by @pattyshack in #3169
• Refactor value store into environment/ by @pattyshack in #3170
• Rename EnvContext to Tracer + Reduce dependency on Context by @pattyshack in #2909
• Rename StateHolder to TransactionState by @pattyshack in #3265
• Reuse the original transaction environment when error occurs by @pattyshack in #3206
• Simplify Process, Processor, Environment initialization by @pattyshack in #3216
• Simplify meter limits control by @pattyshack in #2931
• Split AccountInfo out of commonEnv by @pattyshack in #3208
• Split block info and crypto library out of env into logical units by @pattyshack in #3053
• Split transaction info out of env into a logical unit by @pattyshack in #3026
• Standardize memory and interaction limit disabling by @pattyshack in #2993
• Standardize nested transaction creation/restart/commitment by @pattyshack in #2963
• Stop exposing Context() as part of environment api by @pattyshack in #3168
• Stop type casting meter before setting weights / limits by @pattyshack in #2908
• Switch ProgramsHandler to use TransactionPrograms interface by @pattyshack in #3199
• Switch adding key and creating account to RunWithAllLimitsDisabled by @pattyshack in #3232
• add IsError to errors/ by @pattyshack in #3288
• dedup get execution weights by @pattyshack in #2907
• fix TransactionFeeDeductionFailedError TxFees initialization by @pattyshack in #3282
• fix error initialization in tests by @pattyshack in #3286
• fix out of bound slice access in StateValueSizeLimitError by @pattyshack in #3280
• move ImplementationDebugLog into ProgramLogger by @pattyshack in #3002
• move blocks.go into environment subpackage by @pattyshack in #3001
• move facadeEnv into environment/ by @pattyshack in #3296
• move getAccount from vm to AccountInfo by @pattyshack in #3264
• mv AllocateStorageIndex into ValueStore by @pattyshack in #3185
• mv TxIndex / TxID into TransactionInfo by @pattyshack in #3224
• mv handler/* environment/ by @pattyshack in #3260
• rm AccountFreezeEnabled. It should always be on by @pattyshack in #2939
• rm EventCollectionEnabled. It does nothing by @pattyshack in #2940
• rm MaxNumOfTxRetries + add comment to ServiceAccountEnabled by @pattyshack in https://github.com/onflow/flow-go/pull/
• Update to Cadence v0.26.0 and Go SDK v0.27.0 by @turbolent in #3077
• Update to Cadence v0.27.0 and Go SDK v0.28.0 by @turbolent in #3271
• Ensure cadence don't access unexpected operations while parsing (part 2) (#3345) @
• Ensure cadence don't access unexpected operations while parsing (part 2) (#3345) @pattyshack
• [FVM] Lazy seed unsafe random (#3354) @janezpodhostnik
• [FVM] Keeping event counter inside EventEmitter (#3371) @Tonix517
• clean up errors in txn verifier / seq num checker by @pattyshack in #3382
• Modify SplitError's behavior. by @pattyshack in #3332
• Ensure cadence don't access unexpected operations while parsing (part 3) by @pattyshack in #3359
• Clean up seq num checker's nested transaction commit/restart by @pattyshack in #3383
• [FVM] Log fees error (#3392) @janezpodhostnik
• [FVM] Refactoring TransactionProcedure trace span creation (#3395) @Tonix517
• Adding Event Index Sequence checking to UTs (#3386) @Tonix517
• [FVM] Log fees error (#3392) @janezpodhostnik

Networking improvements and changes to support BFT

• [Networking] Migrates topology management to GossipSub by @yhassanzadeh13 in #2875
• [Networking] Reduce test flakes by @SaveTheRbtz in #3036
• [Networking] Make protocolPeerCache threadsafe by @SaveTheRbtz in #3037
• [Networking] Minor mock generation cleanup by @SaveTheRbtz in #3027
• [Networking] Explicitly set resource limits in tests by @SaveTheRbtz in #3099
• [Networking] Quarantine flaky tests by @SaveTheRbtz in #3111
• [Networking] Fix network stub concurrency by @SaveTheRbtz in #3110
• [Networking] Upgrade LibP2P Core by @yhassanzadeh13 in #3225
• [Networking] Repackages P2P package by @yhassanzadeh13 in #3253
• [Networking] GossipSub Authorized Subscription by @yhassanzadeh13 in #3291
• [Networking] Log all inbound connections (#3333) @peterargue
• Extend P2P connection gater filters by @kc1116 in #2990
• Khalil/1702 unicast authorized sender by @kc1116 in #2832
• Khalil/1741 baselevel protections metrics by @kc1116 in #3146
• Khalil/1762 check codec sentinels in topic validator by @kc1116 in https://github.com/onflow/flow-go/pull
• Add searchable labels to bft related log events (#3346) @peterargue
• [Networking] Refactor libp2pNode and peer manager into components (#2808) @peterargue
• [Networking] blocking nodes on network level (#3344) @AlexHentschel
• [Routing Attacks] Push Access Nodes To Edge by @yhassanzadeh13 in https://github.com/onflow/flow-go/pull/
• [BFT] Authorize execution data requests over bitswap (#3373) @peterargue
• [BFT] add suspicious flag to a few new log messages (#3376) @peterargue
• [BFT] Improve network message validation (#3375) @peterargue

BFT Testing

• [BFT Testing] updated factory and network tests to support new ccf interface by @gomisha in #2768
• [BFT Testing] Crash Test utility by @gomisha in #2941
• [BFT Testing] Corrupting Ingress Traffic (part-1) by @yhassanzadeh13 in #2807
• [BFT Testing] Corrupting ingress traffic (part-2, add ingress message type) by @gomisha in #2811
• [BFT Testing] Corrupting ingress traffic (part-3, process ingress message) by @gomisha in #2936
• [BFT Testing] Adds integration test for ingress corruption by @yhassanzadeh13 in #3215
• [BFT Testing] ingress traffic - insecure/net/*_test.go, renaming Corruptible Conduit Factory to Corrupt Network (#3255) @gomisha
• [BFT Testing] Fixing flakey TestNetworkPartitionWithNoHonestPeerScoringInFullTopology by @yhassanzadeh13 in #3397

Testing

• [Testing] Bump golangci-lint action from v2 to v3 by @fxamacker in #2956
• [Testing] Remove legacy ioutil and os.Is* usage by @SaveTheRbtz in #2989
• [Testing] Format all files in the integration directory by @SaveTheRbtz in #2987
• [Testing] Remove the data races in zerolog Hook by @SaveTheRbtz in #2991
• [Testing] Add mock for the flow-go-sdk/access.Client by @SaveTheRbtz in #2986
• [Testing] fix TestCreateUploader flakes by @SaveTheRbtz in #2992
• [Testing] Fix lint issues in the integration sub-package by @SaveTheRbtz in #3007
• [Testing] Do not run flaky tests in bors by @SaveTheRbtz in #3014
• [Testing] End-to-End TPS benchmark by @AndrewM-SDET in #2961
• [Testing] consolidate benchmarking code by @SaveTheRbtz in #3024
• [Testing] Add proper mock-based TxFollower tests by @SaveTheRbtz in #3011
• [Testing][CI] Free up Github runners by disabling 2 flaky tests workflows by @gomisha in #3012
• [Testing] De-duplicate metric pusher code by @SaveTheRbtz in #3029
• [Testing] Enable unit-testing of the benchmark directory by @SaveTheRbtz in #3048
• [Testing] Enable Lint for ./crypto and ./integration by @SaveTheRbtz in #3006
• [Testing] Fix race in WaitForReceiptFromAny by @SaveTheRbtz in #3042
• [Testing] Diff mocks in CI by @SaveTheRbtz in #3033
• [Testing] Minor improvements to crash test by @SaveTheRbtz in #3028
• [Testing] Bump golangci-lint to v1.47.3 by @SaveTheRbtz in #3055
• [Testing] Fix races in BlockState by @SaveTheRbtz in #3041
• [Testing] Add LoggerForTest to avoid hardcoding Caller by @SaveTheRbtz in #3073
• [Testing] Add an ability to enable Localnet pprof uploader by @SaveTheRbtz in #3095
• [Testing] Enable race detector by default by @SaveTheRbtz in #3034
• [Testing] Disable race detector while we fix remaining flakes by @SaveTheRbtz in #3158
• [Testing] Bump golangci-lint to v1.49.0 by @SaveTheRbtz in #3137
• [Testing][CI] Decrease unit test execution time in CI by @gomisha in #3087
• [Testing] Use T.TempDir to create temporary test directory by @Juneezee in #3165
• [Testing] de-flake TestGenerateTestMatrix by @SaveTheRbtz in #3242
• [EN Testing] Fix Compactor flaky test when GitHub CI is slow by @fxamacker in #2926
• [Test] Fix flaky access engine unittest by @peterargue in #3171
• Quarantine execution state sync test by @kc1116 in #3107
• Fix flaky TopicValidatorTest by @kc1116 in #3147
• Fix job queue test races by @SaveTheRbtz in #3092
• Fix race in TestCompactorConcurrency by @fxamacker in #3038

Profiler/Telemetry changes

• [Profiler] Add heap allocation diffs by @SaveTheRbtz in #3043
• [Profiler] Remove pprof files on failure by @SaveTheRbtz in #3044
• [Profiler] fix race in SetProfilerEnabled by @SaveTheRbtz in #2923

Util changes

• [Util] set/get hotstuff view by @zhangchiqing in #2888
• [Util] transit script for new checkpointing by @vishalchangrani in #3301

Checkpointing

• [Checkpoint V6] Refactor load checkpoint by @zhangchiqing in #3310
• [Checkpoint V6] Revert "Refactor load checkpoint" by @zhangchiqing in #3342
• [Checkpoint V6] Add CRC checksum by @zhangchiqing in #3319
• [Checkpoint v6] Check part file exist by @zhangchiqing in #3321
• [Checkpoint V6] Checkpoint v6 support v5 by @zhangchiqing in #3326
• [Checkpoint V6] Add EOF check by @zhangchiqing in #3327
• [Checkpoint v6] add more tests by @zhangchiqing in #3328
• [Checkpoint v6] Control concurrency by @zhangchiqing in #3329
• [Checkpoint V6] Evict file after read by @zhangchiqing in #3330
• [Checkpoint V6] Add version to part files by @zhangchiqing in #3335
• [Checkpoint v6] refactor reader writer arguments by @zhangchiqing in #3336
• [Checkpoint v6] Cleanup temp files on writing error by @zhangchiqing in #3337
• [Checkpoint V6] refactor getNodeByIndex by @zhangchiqing in #3347
• [Checkpoint V6] Encode subtrie node count by @zhangchiqing in #3350
• [Checkpoint v6] Refactor file close error by @zhangchiqing in #3368
• [Checkpoint v6] support concurrent checkpoint encoding/decoding by @zhangchiqing in #3273

Rolling upgrade support

• #3045 stop EN and VN at height (#3213) @m4ksio

🛠 General Node Improvements

• Adding Upsert for BadgerDB operations by @Tonix517 in #3062
• [All] Deprecate ioutil by @SaveTheRbtz in #3136
• [All] Go 1.19 by @SaveTheRbtz in #3135
• [All] Reformat using go 1.19 fmt by @SaveTheRbtz in #3140
• [Build] Update docker-build-loader target by @AndrewM-SDET in #3051
• [CODEOWNERS] Move Patrick from integration to FVM by @SaveTheRbtz in #3081
• [CODEOWNERS] Stop spamming Simon with notifications by @SaveTheRbtz in #3251
• [CODEOWNERS] actualize performance team's ownership by @SaveTheRbtz in #2924
• [Go] Bump test client go version to 1.18 by @SaveTheRbtz in #2934
• [Localnet] deduplicate service creation by @SaveTheRbtz in #2928
• [Localnet] do not uninstall loki plugin on start by @SaveTheRbtz in #2925
• [Makefile] bump minimum golang version to 1.18 by @SaveTheRbtz in #2933
• Add CodedError / CodedFailure to the errors package by @pattyshack in #3261
• Add IsValueError to errors/ by @pattyshack in #3279
• Add more detailed logging for tracker inconsistency detection by @smnzhu in #3058
• bump up mockery version to 2.13.1 by @pattyshack in #2929
• Adding install-mock-generators as dep to generate-mocks in Makefile by @Tonix517 in #3094
• Documentation: Coding Conventions by @AlexHentschel in #2575
• Go 1.19 support (libp2p upgrade) by @SaveTheRbtz in #2988
• Improve error handling and documentation in protocol.Epoch by @jordanschalm in #3019
• Increase logger's time resolution from seconds to nanoseconds by @SaveTheRbtz in #3317
• Update debug logs and engine metrics (#3270) @kc1116
• [Metrics] Adding metrics interface for block computation effort vector (#3388) @Tonix517
• Align message processing between common and collection sync engines by @peterargue in #3407