Datasource does not gracefully reconnect after db is temporarily unavailable (outage) or db is restarted

[hdmengare]

Description:
When DB server is restarted, quartz is unable to connect it back even when DB server is up and running again.

quartz keeps on throwing the below error even when DB server is up:

QuartzScheduler_OtmmScheduler-HMENGAREDH3NWG31715062741536_ClusterManager 2024-05-07 11:51:26,255 | ERROR  | session=       | user=           | org.quartz.impl.jdbcjobstore.JobStoreCMT | ClusterManager: Error managing cluster: Failed to obtain DB connection from data source 'otmm-no-tx': java.sql.SQLException: Could not retrieve datasource via JNDI url 'java:openejb/Resource/jdbc/scheduler-db-no-tx' javax.naming.NameNotFoundException: Name [openejb/Resource/jdbc/scheduler-db-no-tx] is not bound in this Context. Unable to find [openejb].
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'otmm-no-tx': java.sql.SQLException: Could not retrieve datasource via JNDI url 'java:openejb/Resource/jdbc/scheduler-db-no-tx' javax.naming.NameNotFoundException: Name [openejb/Resource/jdbc/scheduler-db-no-tx] is not bound in this Context. Unable to find [openejb].
        at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:168) ~[quartz-2.3.2.jar:?]
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3322) ~[quartz-2.3.2.jar:?]
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3935) ~[quartz-2.3.2.jar:?]
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3972) ~[quartz-2.3.2.jar:?]
Caused by: java.sql.SQLException: Could not retrieve datasource via JNDI url 'java:openejb/Resource/jdbc/scheduler-db-no-tx' javax.naming.NameNotFoundException: Name [openejb/Resource/jdbc/scheduler-db-no-tx] is not bound in this Context. Unable to find [openejb].
        at org.quartz.utils.JNDIConnectionProvider.getConnection(JNDIConnectionProvider.java:163) ~[quartz-2.3.2.jar:?]
        at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108) ~[quartz-2.3.2.jar:?]
        at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165) ~[quartz-2.3.2.jar:?]
        ... 3 more

Analysis:

1. This issue is seen only in TomEE server and not in JBOSS.
2. This issue is reproducible in postgres as well as Microsoft SQL server DB.
3. Once the DB server is down, datasource object of org.quartz.utils.JNDIConnectionProvider is made null in catch block during the execution of getConnection() method.

 catch (Exception e) {
            this.datasource = null;
            throw new SQLException(
                    "Could not retrieve datasource via JNDI url '" + url + "' "
                            + e.getClass().getName() + ": " + e.getMessage());
        

4. Now since datasource object is null, trying to retrieve this datasource object from context when getConnection() method is executed again.

            if (ds == null || isAlwaysLookup()) {
                ctx = (props != null) ? new InitialContext(props): new InitialContext(); 

                ds = ctx.lookup(url);
                if (!isAlwaysLookup()) {
                    this.datasource = ds;
                }
            }

5. In above if block while executing ctx.lookup(url) below exception is thrown:
   javax.naming.NameNotFoundException: Name [openejb/Resource/jdbc/scheduler-db-no-tx] is not bound in this Context. Unable to find [openejb].
6. However, this same logic is used in init() method to get datasource object at the time of intialization, but here no exception is thrown.

    private void init() {

        if (!isAlwaysLookup()) {
            Context ctx = null;
            try {
                ctx = (props != null) ? new InitialContext(props) : new InitialContext(); 

                datasource = (DataSource) ctx.lookup(url);
            } catch (Exception e) {
                getLog().error(
                        "Error looking up datasource: " + e.getMessage(), e);
            } finally {
                if (ctx != null) {
                    try { ctx.close(); } catch(Exception ignore) {}
                }
            }
        }
    }

7. Only in getConnection() method when datasource object is null this issue seen.

Software version used:
quartz - 2.3.2
TomEE - 8.0.13
Postgresql - 6.12