Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent failure of Aws2KinesisTest.testMethodForDefaultCredentialsProvider #6342

Open
jamesnetherton opened this issue Aug 12, 2024 · 0 comments
Open

Intermittent failure of Aws2KinesisTest.testMethodForDefaultCredentialsProvider #6342

jamesnetherton opened this issue Aug 12, 2024 · 0 comments
Labels
area/native bug Something isn't working flaky-test Issues for tests that are known to be flaky native

Comments

@jamesnetherton
Copy link
Contributor

Bug description

Not sure what's going on here. Seems to fail frequently, often in native mode.

There's an exception in the logs which may be related. Seems the connection pool for the HTTP client is shutdown not long after the Camel routes start.

2024-08-12T08:45:42.7168286Z 2024-08-12 08:45:42,675 WARN  [org.apa.cam.com.aws.kin.Kinesis2Consumer] (Camel (camel-1) thread #3 - Kinesis_shard_monitor) Exception getting latest shard list: java.lang.IllegalStateException: Connection pool shut down
2024-08-12T08:45:42.7170396Z 	at org.apache.http.util.Asserts.check(Asserts.java:34)
2024-08-12T08:45:42.7171931Z 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269)
2024-08-12T08:45:42.7174606Z 	at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:75)
2024-08-12T08:45:42.7177046Z 	at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:57)
2024-08-12T08:45:42.7178365Z 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
2024-08-12T08:45:42.7179121Z 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
2024-08-12T08:45:42.7179910Z 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
2024-08-12T08:45:42.7180763Z 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
2024-08-12T08:45:42.7181588Z 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
2024-08-12T08:45:42.7182529Z 	at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
2024-08-12T08:45:42.7183503Z 	at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254)
2024-08-12T08:45:42.7184370Z 	at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104)
2024-08-12T08:45:42.7185207Z 	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231)
2024-08-12T08:45:42.7186016Z 	at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228)
2024-08-12T08:45:42.7186910Z 	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:99)
2024-08-12T08:45:42.7188077Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79)
2024-08-12T08:45:42.7189349Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57)
2024-08-12T08:45:42.7190555Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40)
2024-08-12T08:45:42.7191868Z 	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
2024-08-12T08:45:42.7193293Z 	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
2024-08-12T08:45:42.7194696Z 	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
2024-08-12T08:45:42.7196094Z 	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
2024-08-12T08:45:42.7197571Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
2024-08-12T08:45:42.7199092Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
2024-08-12T08:45:42.7200559Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
2024-08-12T08:45:42.7201963Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
2024-08-12T08:45:42.7203439Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
2024-08-12T08:45:42.7204989Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
2024-08-12T08:45:42.7206675Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.executeRequest(RetryableStage2.java:93)
2024-08-12T08:45:42.7208014Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:56)
2024-08-12T08:45:42.7209225Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36)
2024-08-12T08:45:42.7210485Z 	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
2024-08-12T08:45:42.7211681Z 	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
2024-08-12T08:45:42.7212665Z 	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
2024-08-12T08:45:42.7214097Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
2024-08-12T08:45:42.7215977Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
2024-08-12T08:45:42.7217360Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
2024-08-12T08:45:42.7218724Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
2024-08-12T08:45:42.7220161Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
2024-08-12T08:45:42.7221556Z 	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
2024-08-12T08:45:42.7222971Z 	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
2024-08-12T08:45:42.7224497Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
2024-08-12T08:45:42.7226140Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
2024-08-12T08:45:42.7227592Z 	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
2024-08-12T08:45:42.7228744Z 	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
2024-08-12T08:45:42.7229809Z 	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
2024-08-12T08:45:42.7230916Z 	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
2024-08-12T08:45:42.7232095Z 	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
2024-08-12T08:45:42.7233241Z 	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
2024-08-12T08:45:42.7234286Z 	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
2024-08-12T08:45:42.7235319Z 	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
2024-08-12T08:45:42.7236368Z 	at software.amazon.awssdk.services.kinesis.DefaultKinesisClient.listShards(DefaultKinesisClient.java:1615)
2024-08-12T08:45:42.7237393Z 	at org.apache.camel.component.aws2.kinesis.Kinesis2Consumer$ShardMonitor.getShardList(Kinesis2Consumer.java:431)
2024-08-12T08:45:42.7238385Z 	at org.apache.camel.component.aws2.kinesis.Kinesis2Consumer$ShardMonitor.run(Kinesis2Consumer.java:399)

After some googling, there's some issues that hint at working around it by configuring an explicit max connections value on the client.
@jamesnetherton jamesnetherton added the bug Something isn't working label Aug 12, 2024
@github-actions github-actions bot added native area/native flaky-test Issues for tests that are known to be flaky labels Aug 12, 2024
jamesnetherton added a commit to jamesnetherton/camel-quarkus that referenced this issue Aug 12, 2024
@jamesnetherton
Disable Aws2Kinesis on GitHub CI due to apache#6342
20302ba
jamesnetherton added a commit to jamesnetherton/camel-quarkus that referenced this issue Aug 12, 2024
@jamesnetherton
Disable Aws2KinesisTest on GitHub CI due to apache#6342
260ede2
jamesnetherton added a commit that referenced this issue Aug 12, 2024
@jamesnetherton
Disable Aws2KinesisTest on GitHub CI due to #6342
8186f67
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/native bug Something isn't working flaky-test Issues for tests that are known to be flaky native
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jamesnetherton