Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle closed network connections by retrying #1166

Open
ajscholl opened this issue Oct 15, 2024 · 0 comments
Open

Handle closed network connections by retrying #1166

ajscholl opened this issue Oct 15, 2024 · 0 comments

Comments

@ajscholl
Copy link
Contributor

When running integration tests, they sometimes fail with the following log:

12:32:37.802 handler info    looking for ddb table xxx
    testcase_application.go:79: 
        	Error Trace:	/go/pkg/mod/github.com/justtrackio/[email protected]/pkg/test/suite/testcase_application.go:79
        	            				/go/pkg/mod/github.com/justtrackio/[email protected]/pkg/test/suite/testcase_httpserver.go:107
        	            				/go/pkg/mod/github.com/justtrackio/[email protected]/pkg/test/suite/run.go:166
        	Error:      	failed to create application under test
        	Test:       	TestGatewayTestSuite/TestXXX
        	Messages:   	can not build kernel factory: can not build module api: could not define routes: can not create xxx: can not create xxx: can not create xxx repo: can not create ddb repository: could not create ddb table xxx: can not check if the table already exists: can not describe table: operation error DynamoDB: DescribeTable, https response error StatusCode: 200, RequestID: af412002-2658-48a3-8915-506c3d64ec95, deserialization failed, failed to decode response body, read tcp 172.17.0.1:40560->172.17.0.1:32894: use of closed network connection

We should check for the "use of closed network connection" error and handle it like any other error which we just retry - it should be fine retrying here (or, if the container running dynamodb already exited, show a message about dynamodb being not (no longer) reachable).

Right now this sometimes randomly fails the automated tests in the phase loading the fixtures, i.e., no code under test is actually running yet, which makes for a frustrating developer experience.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant