failed to load plugin under supervisord management

[SimpleExpress]

I want to use supervisord to auto restart the trino node, but it always fails due to the class loader problem. Here are the steps I've tried:

Trino Version: 422
Manual command startup is ok: /opt/trino/bin/launcher start

supervisord config

[program:trino]
process_name=%(program_name)s
directory=/opt/trino
command=/opt/trino/bin/launcher start
autostart=false
autorestart=false
user=trino
startretries=0
startsecs=10
loglevel=critical

start by supervisord: supervisorctl start trino

trino server failure log

No service providers of type io.trino.spi.Plugin in the classpath: [file:/opt/trino_data/plugin/sqlserver/HdrHistogram-2.1.9.jar, file:/opt/trino_data/plugin/sqlserver/antlr4-runtime-4.13.0.jar, file:/opt/trino_data/plugin/sqlserver/aopalliance-1.0.jar, file:/opt/trino_data/plugin/sqlserver/bootstrap-234.jar, file:/opt/trino_data/plugin/sqlserver/byte-buddy-1.14.4.jar, file:/opt/trino_data/plugin/sqlserver/checker-qual-3.33.0.jar, file:/opt/trino_data/plugin/sqlserver/classmate-1.5.1.jar, file:/opt/trino_data/plugin/sqlserver/concurrent-234.jar, file:/opt/trino_data/plugin/sqlserver/configuration-234.jar, file:/opt/trino_data/plugin/sqlserver/discovery-234.jar, file:/opt/trino_data/plugin/sqlserver/error_prone_annotations-2.20.0.jar, file:/opt/trino_data/plugin/sqlserver/failsafe-3.3.1.jar, file:/opt/trino_data/plugin/sqlserver/failureaccess-1.0.1.jar, file:/opt/trino_data/plugin/sqlserver/geantyref-1.3.14.jar, file:/opt/trino_data/plugin/sqlserver/guava-32.0.0-jre.jar, file:/opt/trino_data/plugin/sqlserver/guice-7.0.0.jar, file:/opt/trino_data/plugin/sqlserver/hibernate-validator-8.0.0.Final.jar, file:/opt/trino_data/plugin/sqlserver/http-client-234.jar, file:/opt/trino_data/plugin/sqlserver/http2-client-11.0.15.jar, file:/opt/trino_data/plugin/sqlserver/http2-common-11.0.15.jar, file:/opt/trino_data/plugin/sqlserver/http2-hpack-11.0.15.jar, file:/opt/trino_data/plugin/sqlserver/http2-http-client-transport-11.0.15.jar, file:/opt/trino_data/plugin/sqlserver/j2objc-annotations-2.8.jar, file:/opt/trino_data/plugin/sqlserver/jackson-core-2.15.2.jar, file:/opt/trino_data/plugin/sqlserver/jackson-databind-2.15.2.jar, file:/opt/trino_data/plugin/sqlserver/jackson-datatype-guava-2.15.2.jar, file:/opt/trino_data/plugin/sqlserver/jackson-datatype-jdk8-2.15.2.jar, file:/opt/trino_data/plugin/sqlserver/jackson-datatype-joda-2.15.2.jar, file:/opt/trino_data/plugin/sqlserver/jackson-datatype-jsr310-2.15.2.jar, file:/opt/trino_data/plugin/sqlserver/jackson-module-parameter-names-2.15.2.jar, file:/opt/trino_data/plugin/sqlserver/jakarta.annotation-api-2.1.1.jar, file:/opt/trino_data/plugin/sqlserver/jakarta.inject-api-2.0.1.jar, file:/opt/trino_data/plugin/sqlserver/jakarta.validation-api-3.0.2.jar, file:/opt/trino_data/plugin/sqlserver/jboss-logging-3.4.1.Final.jar, file:/opt/trino_data/plugin/sqlserver/jcl-over-slf4j-2.0.7.jar, file:/opt/trino_data/plugin/sqlserver/jdbi3-core-3.39.1.jar, file:/opt/trino_data/plugin/sqlserver/jetty-alpn-client-11.0.15.jar, file:/opt/trino_data/plugin/sqlserver/jetty-alpn-java-client-11.0.15.jar, file:/opt/trino_data/plugin/sqlserver/jetty-client-11.0.15.jar, file:/opt/trino_data/plugin/sqlserver/jetty-http-11.0.15.jar, file:/opt/trino_data/plugin/sqlserver/jetty-io-11.0.15.jar, file:/opt/trino_data/plugin/sqlserver/jetty-util-11.0.15.jar, file:/opt/trino_data/plugin/sqlserver/jmx-234.jar, file:/opt/trino_data/plugin/sqlserver/jmxutils-1.23.jar, file:/opt/trino_data/plugin/sqlserver/joda-time-2.12.5.jar, file:/opt/trino_data/plugin/sqlserver/json-234.jar, file:/opt/trino_data/plugin/sqlserver/jsr305-3.0.2.jar, file:/opt/trino_data/plugin/sqlserver/log-234.jar, file:/opt/trino_data/plugin/sqlserver/log-manager-234.jar, file:/opt/trino_data/plugin/sqlserver/log4j-api-2.17.1.jar, file:/opt/trino_data/plugin/sqlserver/log4j-over-slf4j-2.0.7.jar, file:/opt/trino_data/plugin/sqlserver/log4j-to-slf4j-2.17.1.jar, file:/opt/trino_data/plugin/sqlserver/logback-core-1.4.7.jar, file:/opt/trino_data/plugin/sqlserver/modernizer-maven-annotations-2.6.0.jar, file:/opt/trino_data/plugin/sqlserver/mssql-jdbc-11.2.1.jre17.jar, file:/opt/trino_data/plugin/sqlserver/node-234.jar, file:/opt/trino_data/plugin/sqlserver/opentelemetry-semconv-1.26.0-alpha.jar, file:/opt/trino_data/plugin/sqlserver/security-234.jar, file:/opt/trino_data/plugin/sqlserver/slf4j-api-2.0.7.jar, file:/opt/trino_data/plugin/sqlserver/slf4j-jdk14-2.0.7.jar, file:/opt/trino_data/plugin/sqlserver/stats-234.jar, file:/opt/trino_data/plugin/sqlserver/trace-token-234.jar, file:/opt/trino_data/plugin/sqlserver/trino-base-jdbc-422.jar, file:/opt/trino_data/plugin/sqlserver/trino-cache-422.jar, file:/opt/trino_data/plugin/sqlserver/trino-matching-422.jar, file:/opt/trino_data/plugin/sqlserver/trino-plugin-toolkit-422.jar, file:/opt/trino_data/plugin/sqlserver/trino-sqlserver-422-services.jar, file:/opt/trino_data/plugin/sqlserver/trino-sqlserver-422.jar, file:/opt/trino_data/plugin/sqlserver/units-1.9.jar]

the spi class should be located in /opt/trino_data/plugin/sqlserver/trino-sqlserver-422.jar

$ jar -tf /opt/trino/plugin/sqlserver/trino-sqlserver-422.jar | grep SqlServerPlugin
io/trino/plugin/sqlserver/SqlServerPlugin.class

I tried to add some log in PluginManager's loadPlugin method, like this:

private void loadPlugin(String plugin, Supplier<PluginClassLoader> createClassLoader)
    // ...start code...
    try (ThreadContextClassLoader ignored = new ThreadContextClassLoader(pluginClassLoader)) {
        loadPlugin(pluginClassLoader);
        try {
            pluginClassLoader.loadClass("io.trino.plugin.sqlserver.SqlServerPlugin", true);
            log.info("loaded io.trino.plugin.sqlserver.SqlServerPlugin");
        }
        catch (Throwable e) {
            log.error("class not found: io.trino.plugin.sqlserver.SqlServerPlugin");
        }
    }
    // ...remain code...
}

...

and run again, resulted in 'class not found: io.trino.plugin.sqlserver'.

Could someone give a help. Thanks a lot.

[hashhar]

Can you check the ulimits when running under supervisord. My guess is the open files limit is low. Check the Trino docs for recommended values.