Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent bsp-agent to crash in case of transaction errors #250

Open
hritique opened this issue Apr 18, 2024 · 0 comments
Open

Prevent bsp-agent to crash in case of transaction errors #250

hritique opened this issue Apr 18, 2024 · 0 comments

Comments

@hritique
Copy link

BSP agent crashes (after some retries) when there is any issue in producing BS proofs
From the last conversation a while back, we didn't have any monitoring, hence crashing the bsp-agent helped devs to be aware of some issue and help them debugging

But now that we have the metrics for successful block specimens, it would be nice to properly catch the errors, and avoid crashing the application

Specially recently we have been facing alot of crashes due to chain congestion
Some error logs recently:

bsp-agent-chorusone  | 2024-04-14T12:47:17.858870813Z time="2024-04-14T12:47:17Z" level=info msg="Process pending streams at: 2024-04-14T12:47:17Z" function=consumePendingEvents line=120
bsp-agent-chorusone  | 2024-04-14T12:47:19.621061918Z time="2024-04-14T12:47:19Z" level=error msg="error sending tx to deployed contract: submit transaction to pool failed: Ok(ImmediatelyDropped)" function=executeWithRetry line=153
bsp-agent-chorusone  | 2024-04-14T12:47:19.621102596Z time="2024-04-14T12:47:19Z" level=error msg="failed to avro encode, prove and upload block-result segment with err: failed to prove & upload block-replica segment event: 1-19653865-replica" function=processStream line=155
bsp-agent-chorusone  | 2024-04-14T12:47:19.623321289Z panic: failed to prove & upload block-replica segment event: 1-19653865-replica
bsp-agent-chorusone  | 2024-04-14T12:47:19.623383788Z 
bsp-agent-chorusone  | 2024-04-14T12:47:19.623388126Z goroutine 7320 [running]:
bsp-agent-chorusone  | 2024-04-14T12:47:19.623391242Z github.com/covalenthq/bsp-agent/internal/node.(*ethAgentNode).processStream(0xc00044b520, {{0xc000028510?, 0xc0003c0fd0?}, 0xc00012e090?}, 0xc00049b700)
bsp-agent-chorusone  | 2024-04-14T12:47:19.623394519Z   /bsp-agent/internal/node/eth_node.go:156 +0x95a
bsp-agent-chorusone  | 2024-04-14T12:47:19.623397534Z created by github.com/covalenthq/bsp-agent/internal/node.(*ethAgentNode).consumeEvents
bsp-agent-chorusone  | 2024-04-14T12:47:19.623404828Z   /bsp-agent/internal/node/eth_node.go:69 +0x38a
bsp-agent-chorusone exited with code 0
bsp-agent-chorusone  | 2024-04-14T14:15:26.879654466Z time="2024-04-14T14:15:26Z" level=info msg="tip cap: 2000000000 fee cap: 252000000000" function=setTransactionFeeParams line=104
bsp-agent-chorusone  | 2024-04-14T14:15:37.065873908Z time="2024-04-14T14:15:37Z" level=error msg="error sending tx to deployed contract: replacement transaction underpriced" function=executeWithRetry line=153
bsp-agent-chorusone  | 2024-04-14T14:15:37.065928362Z time="2024-04-14T14:15:37Z" level=error msg="failed to avro encode, prove and upload block-result segment with err: failed to prove & upload block-replica segment event: 1-19654040-replica" function=processStream line=155
bsp-agent-chorusone  | 2024-04-14T14:15:37.068130283Z panic: failed to prove & upload block-replica segment event: 1-19654040-replica
bsp-agent-chorusone  | 2024-04-14T14:15:37.068172384Z 
bsp-agent-chorusone  | 2024-04-14T14:15:37.068179037Z goroutine 2775 [running]:
bsp-agent-chorusone  | 2024-04-14T14:15:37.068184176Z github.com/covalenthq/bsp-agent/internal/node.(*ethAgentNode).processStream(0xc00049e2e0, {{0xc0033f57a0?, 0x2e72fd?}, 0xc000c102a0?}, 0xc000127460)
bsp-agent-chorusone  | 2024-04-14T14:15:37.068190168Z   /bsp-agent/internal/node/eth_node.go:156 +0x95a
bsp-agent-chorusone  | 2024-04-14T14:15:37.068195037Z created by github.com/covalenthq/bsp-agent/internal/node.(*ethAgentNode).consumePendingEvents
bsp-agent-chorusone  | 2024-04-14T14:15:37.068199977Z   /bsp-agent/internal/node/eth_node.go:116 +0x625
bsp-agent-chorusone exited with code 0
@hritique hritique changed the title Prevent bsp-agent to crashloop in case of transaction errors Prevent bsp-agent to crash in case of transaction errors Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant