Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HikariCP pool suddenly drops to 0 despite minIdle and maxPoolSize being default and connections available on postgres. #2277

Open
naishal opened this issue Dec 20, 2024 · 1 comment

Comments

@naishal
Copy link

naishal commented Dec 20, 2024

I have postgres hosted on RDS with db.t4g.large instance which has 8 GB memory, which provides around 900 max connections.

Service is hosted on kubernetes with 10 pods running

HikariCP has default max pool size and min idle. So each pod is supposed to have 10 connections and total of 100 connections active all time (since min Idle and max pool size is default 10)

HikariConfig().also { hikari ->
    hikari.poolName = poolName
    hikari.dataSource = PGSimpleDataSource().also { pgDataSource ->
        ...
    }
    hikari.username = dataSourceProperties.username
    hikari.connectionTimeout = connectionTimeout.toMillis()
    hikari.maxLifetime = maxLifetime.toMillis()
}

I get the following error:

java.sql.SQLTransientConnectionException: Main - Connection is not available, request timed out after 10000ms (total=0, active=0, idle=0, waiting=0)
	at c.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:686)
	at c.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:179)
	at c.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:144)
	at c.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:99)
	at c.s.f.p.p.PostgresDataSourceFactory$createDataSource$1.getConnection(PostgresDataSourceFactory.kt:50)
	at jdk.proxy2/jdk.proxy2.$Proxy167.countUsersActivelyProcessing(Unknown Source) [26 skipped]
	at j.i.r.GeneratedMethodAccessor52.invoke(Unknown Source)
	at j.b.i.r.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at j.base/java.lang.reflect.Method.invoke(Method.java:568)
	at c.s.f.p.i.SummaryLogRepositoryAspect.recordQueryExecution(SummaryLogRepositoryAspect.kt:48) [6 skipped]
Wrapped by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: Main - Connection is not available, request timed out after 10000ms (total=0, active=0, idle=0, waiting=0)
	at jdk.proxy2/jdk.proxy2.$Proxy167.countUsersActivelyProcessing(Unknown Source) [24 skipped]
	at j.i.r.GeneratedMethodAccessor52.invoke(Unknown Source)
	at j.b.i.r.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at j.base/java.lang.reflect.Method.invoke(Method.java:568)
	at c.s.f.p.i.SummaryLogRepositoryAspect.recordQueryExecution(SummaryLogRepositoryAspect.kt:48) [6 skipped]
	at j.i.r.GeneratedMethodAccessor38.invoke(Unknown Source)
	at j.b.i.r.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at j.base/java.lang.reflect.Method.invoke(Method.java:568)
	at c.s.f.p.i.RetryOnDeadlockAspect.recordQueryExecution(RetryOnDeadlockAspect.kt:30) [7 skipped]
	at j.i.r.GeneratedMethodAccessor37.invoke(Unknown Source)

Interesting part is (total=0, active=0, idle=0, waiting=0) which should not be possible. This issue was happening in only 1 out of 10 pods.

Looks like suddenly Hikari is not setting up correct config of pool size, minIdle etc. To verify this analysis I also checked the RDS monitoring console for max connections ad seems like there were 92 connections active at the time. This proves that 9 pods were behaving right and only 1 was suddenly affected with hikari failing to setup proper config.

I see this issue opened long back and surfed through it but looks like there was no definitive conclusion on the problem and some fixes mentioned are different than the issue we are facing.

@qq31715879
Copy link

I also encountered the same problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants