Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

indexer lagging behind? #3921

Closed
dustinxie opened this issue Aug 14, 2023 · 3 comments · Fixed by #3924 · May be fixed by #3925
Closed

indexer lagging behind? #3921

dustinxie opened this issue Aug 14, 2023 · 3 comments · Fixed by #3924 · May be fixed by #3925
Assignees
@envestcc
Labels
bug Something isn't working

Comments

@dustinxie
Copy link
Member

dustinxie commented Aug 14, 2023 •
edited
Loading

What version of iotex-core image (or code branch) are you using?

v1.11.0

What operating system and processor architecture are you using?

GCP testnet cluster

What did you do? If possible, provide a recipe for reproducing the error.

node keeps generating this error

2023-08-14T19:43:55.571Z	ERROR	blockchain/pubsubmanager.go:121	Failed to handle new block.	{"ioAddr": "io1dg65erd07hrvyme0493f2kqj2utuvpyf6jeuhd", "error": "wrong block height 21768565, expecting 21693700: invalid input", "errorVerbose": "invalid input\ngithub.com/iotexproject/iotex-core/db.init\n\t/go/apps/iotex-core/db/counting_index.go:24\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6222\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6199\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6199\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6199\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6199\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571\nwrong block height 21768565, expecting 21693700\ngithub.com/iotexproject/iotex-core/blockindex.(*blockIndexer).putBlock\n\t/go/apps/iotex-core/blockindex/indexer.go:305\ngithub.com/iotexproject/iotex-core/blockindex.(*blockIndexer).PutBlock\n\t/go/apps/iotex-core/blockindex/indexer.go:140\ngithub.com/iotexproject/iotex-core/blockindex.(*IndexBuilder).ReceiveBlock\n\t/go/apps/iotex-core/blockindex/indexbuilder.go:99\ngithub.com/iotexproject/iotex-core/blockchain.(*pubSub).handler\n\t/go/apps/iotex-core/blockchain/pubsubmanager.go:120\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"}

What did you expect to see?

this should not happen, since indexer should commit every new block and update height

What did you see instead?

seems the indexer is lagging behind
@dustinxie dustinxie added the bug Something isn't working label Aug 14, 2023
@dustinxie
Copy link
Member Author

Suspect:

  1. something wrong in block 21693700, causing error in indexer handling code
  2. caused by recent resource shrink on the testnet cluster

@envestcc
Copy link
Member

Investigation of the issue could be traced here: https://iotex.larksuite.com/wiki/YmdMwBjGziddCjkgj3UuvlKwslf

@dustinxie
Copy link
Member Author

dustinxie commented Aug 15, 2023 •
edited
Loading

re-syncing the testnet from 20645456 (the block height where IIP-13 contract is deployed on testnet), hit the following error at block 21431485:

2023-08-15T21:08:46.076Z	FATAL	itx/server.go:219	Failed to start server.	{"ioAddr": "io1alkrg0glv63mgr809eyhux2kt48q8yppp2hh29", "error": "error when starting blockchain: invalid delta action 1 on state 1", "errorVerbose": "invalid delta action 1 on state 1\ngithub.com/iotexproject/iotex-core/blockindex/contractstaking.deltaState.Transfer\n\t/data/iotex-core/blockindex/contractstaking/delta_state.go:43\ngithub.com/iotexproject/iotex-core/blockindex/contractstaking.(*contractStakingDelta).DeleteBucketInfo\n\t/data/iotex-core/blockindex/contractstaking/delta_cache.go:127\ngithub.com/iotexproject/iotex-core/blockindex/contractstaking.(*contractStakingDirty).deleteBucketInfo\n\t/data/iotex-core/blockindex/contractstaking/dirty_cache.go:66\ngithub.com/iotexproject/iotex-core/blockindex/contractstaking.(*contractStakingEventHandler).handleMergedEvent\n\t/data/iotex-core/blockindex/contractstaking/event_handler.go:602\ngithub.com/iotexproject/iotex-core/blockindex/contractstaking.(*contractStakingEventHandler).HandleEvent\n\t/data/iotex-core/blockindex/contractstaking/event_handler.go:405\ngithub.com/iotexproject/iotex-core/blockindex/contractstaking.(*Indexer).PutBlock\n\t/data/iotex-core/blockindex/contractstaking/indexer.go:159\ngithub.com/iotexproject/iotex-core/blockindex.(*SyncIndexers).PutBlock\n\t/data/iotex-core/blockindex/sync_indexers.go:67\ngithub.com/iotexproject/iotex-core/blockchain/blockdao.(*BlockIndexerChecker).CheckIndexer\n\t/data/iotex-core/blockchain/blockdao/blockindexer.go:114\ngithub.com/iotexproject/iotex-core/blockchain/blockdao.(*blockDAO).checkIndexers\n\t/data/iotex-core/blockchain/blockdao/blockdao.go:98\ngithub.com/iotexproject/iotex-core/blockchain/blockdao.(*blockDAO).Start\n\t/data/iotex-core/blockchain/blockdao/blockdao.go:92\ngithub.com/iotexproject/iotex-core/pkg/lifecycle.(*Lifecycle).OnStart.func1\n\t/data/iotex-core/pkg/lifecycle/lifecycle.go:55\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/data/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\nerror when starting blockchain\ngithub.com/iotexproject/iotex-core/server/itx.(*Server).Start\n\t/data/iotex-core/server/itx/server.go:120\ngithub.com/iotexproject/iotex-core/server/itx.StartServer\n\t/data/iotex-core/server/itx/server.go:218\nmain.main\n\t/data/iotex-core/server/main.go:175\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"}

it is also a handleMergedEvent() call
question is how the contractStakingIndexer worked back in then to commit this block?

BTW, the re-syncing is using the code in branch https://github.com/iotexproject/iotex-core/commits/group, the indexer group PR seems to work fine

@millken millken mentioned this issue Aug 16, 2023
Merged
2 tasks
@envestcc envestcc mentioned this issue Aug 16, 2023
Merged
10 tasks
@dustinxie dustinxie mentioned this issue Aug 22, 2023
Open
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@envestcc envestcc

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@dustinxie @envestcc