Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement the RPCBlockHeaderSubscriber for indexing finalized results #728

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
+479 −135
Open

Implement the RPCBlockHeaderSubscriber for indexing finalized results #728

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
a276412
Implement the RPCBlockHeaderSubscriber for indexing finalized results
m-Peter Jan 21, 2025
7d36012
Rename RPCBlockHeaderSubscriber to RPCBlockTrackingSubscriber
m-Peter Jan 23, 2025
75a1d18
Remove redundant fetching of block header prior to SubscribeBlockHead…
m-Peter Jan 23, 2025
ce83ebb
Improve error handling in RPCBlockTrackingSubscriber subscription
m-Peter Jan 23, 2025
d159467
Simplify RPCBlockTrackingSubscriber by embedding RPCEventSubscriber f…
m-Peter Jan 23, 2025
5e8f388
Optimize RPCBlockTrackingSubscriber to avoid fetch EVM tx events for …
m-Peter Jan 24, 2025
bfe6188
Improve length checks for EVM related events
m-Peter Jan 27, 2025
037c234
Close events channel in RPCBlockTrackingSubscriber
m-Peter Jan 27, 2025
6afde4d
Improve error handling to use the gRPC status code instead of strings
m-Peter Jan 27, 2025
ff927f7
Simplify constructor of RPCBlockTrackingSubscriber
m-Peter Jan 27, 2025
4555263
Remove redundant fields from RPCBlockTrackingSubscriber
m-Peter Jan 27, 2025
34dbde5
Make us of GetEventsForBlockIDs method instead of GetEventsForHeightR…
m-Peter Jan 27, 2025
bf4626c
Add description on RPCBlockTrackingSubscriber type
m-Peter Jan 27, 2025
1f084be
Refactor function for fetching EVM events on a given block header
m-Peter Jan 27, 2025
a8329e4
Make PoC configurable
peterargue Jan 28, 2025
fa39603
fix enabled conditional
peterargue Jan 28, 2025
d68f2ba
Merge pull request #737 from onflow/petera/making-poc-configurable
peterargue Jan 28, 2025
823e965
improve error reconnection handling
peterargue Jan 29, 2025
91b1ec0
Remove redundant start-testnet & start-mainnet make recipes
m-Peter Jan 30, 2025
2ddd052
Merge branch 'main' into mpeter/poc-index-finalized-block-results
peterargue Jan 30, 2025
97f5f93
Merge branch 'main' into mpeter/poc-index-finalized-block-results
peterargue Jan 30, 2025
1e977a9
retry getting events on NotFound and ResourceExhausted
peterargue Feb 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,3 +227,36 @@ endif

docker run $(MODE) -p $(HOST_PORT):8545 -p $(HOST_METRICS_PORT):8080 $(MOUNT) "$(CONTAINER_REGISTRY)/flow-evm-gateway:$(IMAGE_TAG)" $(CMD_ARGS)

.PHONY: start-testnet
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment that the key is from the emulator account (so it doesn't freak people out 😄)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed them altogether in 91b1ec0 . They are not really relevant to the PR.

start-testnet:
rm -rf testnet-db-block-headers/
rm -rf metrics/data/
go run cmd/main.go run \
--database-dir=testnet-db-block-headers \
--access-node-grpc-host=access.devnet.nodes.onflow.org:9000 \
--access-node-spork-hosts=access-001.devnet51.nodes.onflow.org:9000 \
--flow-network-id=flow-testnet \
--init-cadence-height=211176670 \
--ws-enabled=true \
--coinbase=FACF71692421039876a5BB4F10EF7A439D8ef61E \
--coa-address=0x62631c28c9fc5a91 \
--coa-key=2892fba444f1d5787739708874e3b01160671924610411ac787ac1379d420f49 \
--gas-price=100 \
--log-level=info

m-Peter marked this conversation as resolved.
Show resolved Hide resolved
.PHONY: start-mainnet
start-mainnet:
rm -rf mainnet-db-block-headers/
rm -rf metrics/data/
go run cmd/main.go run \
--database-dir=mainnet-db-block-headers \
--access-node-grpc-host=access.mainnet.nodes.onflow.org:9000 \
--access-node-spork-hosts=access-001.mainnet25.nodes.onflow.org:9000 \
--flow-network-id=flow-mainnet \
--init-cadence-height=85981135 \
--ws-enabled=true \
--coinbase=FACF71692421039876a5BB4F10EF7A439D8ef61E \
--coa-address=0xf1ab99c82dee3526 \
--coa-key=2892fba444f1d5787739708874e3b01160671924610411ac787ac1379d420f49 \
--gas-price=100 \
--log-level=info
25 changes: 18 additions & 7 deletions bootstrap/bootstrap.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,13 +142,24 @@ func (b *Bootstrap) StartEventIngestion(ctx context.Context) error {
chainID := b.config.FlowNetworkID

// create event subscriber
subscriber := ingestion.NewRPCEventSubscriber(
b.logger,
b.client,
chainID,
b.keystore,
latestCadenceHeight,
)
var subscriber ingestion.EventSubscriber
if b.config.ExperimentalSoftFinalityEnabled {
subscriber = ingestion.NewRPCBlockTrackingSubscriber(
b.logger,
b.client,
chainID,
b.keystore,
latestCadenceHeight,
)
} else {
subscriber = ingestion.NewRPCEventSubscriber(
b.logger,
b.client,
chainID,
b.keystore,
latestCadenceHeight,
)
}

callTracerCollector, err := replayer.NewCallTracerCollector(b.logger)
if err != nil {
Expand Down
5 changes: 5 additions & 0 deletions cmd/run/cmd.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,8 @@ func parseConfigFromFlags() error {
return fmt.Errorf("unknown tx state validation: %s", txStateValidation)
}

cfg.ExperimentalSoftFinalityEnabled = experimentalSoftFinalityEnabled

return nil
}

Expand All @@ -246,6 +248,8 @@ var (

initHeight,
forceStartHeight uint64

experimentalSoftFinalityEnabled bool
)

func init() {
Expand Down Expand Up @@ -280,4 +284,5 @@ func init() {
Cmd.Flags().StringVar(&cfg.ProfilerHost, "profiler-host", "localhost", "Host for the Profiler server")
Cmd.Flags().IntVar(&cfg.ProfilerPort, "profiler-port", 6060, "Port for the Profiler server")
Cmd.Flags().StringVar(&txStateValidation, "tx-state-validation", "tx-seal", "Sets the transaction validation mechanism. It can validate using the local state index, or wait for the outer Flow transaction to seal. Available values ('local-index' / 'tx-seal'), defaults to 'tx-seal'.")
Cmd.Flags().BoolVar(&experimentalSoftFinalityEnabled, "experimental-soft-finality-enabled", false, "Sets whether the gateway should use the experimental soft finality feature. WARNING: This may result in incorrect results being returned in certain circumstances. Use only if you know what you are doing.")
}
4 changes: 4 additions & 0 deletions config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,4 +89,8 @@ type Config struct {
// TxStateValidation sets the transaction validation mechanism. It can validate
// using the local state index, or wait for the outer Flow transaction to seal.
TxStateValidation string
// ExperimentalSoftFinalityEnabled enables the experimental soft finality feature which syncs
// EVM block and transaction data from the upstream Access node before the block is sealed.
// CAUTION: This feature is experimental and may return incorrect data in certain circumstances.
ExperimentalSoftFinalityEnabled bool
}
37 changes: 18 additions & 19 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ require (
github.com/cockroachdb/pebble v1.1.1
github.com/goccy/go-json v0.10.2
github.com/hashicorp/go-multierror v1.1.1
github.com/onflow/atree v0.8.0
github.com/onflow/cadence v1.2.2
github.com/onflow/atree v0.8.1
github.com/onflow/cadence v1.3.0
github.com/onflow/flow-go v0.38.0-preview.0.4
github.com/onflow/flow-go-sdk v1.2.3
github.com/onflow/flow-go-sdk v1.3.0
github.com/onflow/go-ethereum v1.14.7
github.com/prometheus/client_golang v1.18.0
github.com/rs/cors v1.8.0
Expand All @@ -18,20 +18,19 @@ require (
github.com/sethvargo/go-limiter v1.0.0
github.com/sethvargo/go-retry v0.2.3
github.com/spf13/cobra v1.8.1
github.com/stretchr/testify v1.9.0
github.com/stretchr/testify v1.10.0
go.uber.org/ratelimit v0.3.1
golang.org/x/exp v0.0.0-20240119083558-1b970713d09a
golang.org/x/sync v0.8.0
google.golang.org/grpc v1.63.2
google.golang.org/grpc v1.64.1
)

require (
cloud.google.com/go v0.112.0 // indirect
cloud.google.com/go/compute v1.24.0 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go v0.112.1 // indirect
cloud.google.com/go/compute/metadata v0.5.0 // indirect
cloud.google.com/go/iam v1.1.6 // indirect
cloud.google.com/go/kms v1.15.7 // indirect
cloud.google.com/go/storage v1.36.0 // indirect
cloud.google.com/go/storage v1.38.0 // indirect
github.com/DataDog/zstd v1.5.2 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/SaveTheRbtz/mph v0.1.1-0.20240117162131-4166ec7869bc // indirect
Expand Down Expand Up @@ -88,7 +87,7 @@ require (
github.com/google/s2a-go v0.1.7 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.12.0 // indirect
github.com/googleapis/gax-go/v2 v2.12.2 // indirect
github.com/gorilla/websocket v1.5.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
Expand Down Expand Up @@ -181,10 +180,10 @@ require (
github.com/vmihailenco/msgpack/v4 v4.3.11 // indirect
github.com/vmihailenco/tagparser v0.1.1 // indirect
github.com/x448/float16 v0.8.4 // indirect
github.com/zeebo/blake3 v0.2.3 // indirect
github.com/zeebo/blake3 v0.2.4 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
go.opentelemetry.io/otel v1.24.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.21.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.21.0 // indirect
Expand All @@ -196,19 +195,19 @@ require (
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.26.0 // indirect
golang.org/x/crypto v0.28.0 // indirect
golang.org/x/net v0.25.0 // indirect
golang.org/x/oauth2 v0.17.0 // indirect
golang.org/x/net v0.26.0 // indirect
golang.org/x/oauth2 v0.18.0 // indirect
golang.org/x/sys v0.26.0 // indirect
golang.org/x/text v0.19.0 // indirect
golang.org/x/time v0.5.0 // indirect
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
gonum.org/v1/gonum v0.14.0 // indirect
google.golang.org/api v0.162.0 // indirect
google.golang.org/api v0.169.0 // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/genproto v0.0.0-20240227224415-6ceb2ff114de // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240227224415-6ceb2ff114de // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240227224415-6ceb2ff114de // indirect
google.golang.org/protobuf v1.33.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240814211410-ddb44dafa142 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240814211410-ddb44dafa142 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/blake3 v1.3.0 // indirect
Expand Down
Loading
Loading