java.lang.OutOfMemoryError: unable to create native thread

[roei-datorios]

We recently started using the GCP pub/sub source connector (v1.2).
We have a test which repeatedly creates such a source connector, passes some data through it and deletes it.
When running this cycle long enough, Kafka connect stops functioning due to the error below. At this stage it has over 5k native threads running.
The issue does not reproduce if the source connector being created/deleted has the same name at each cycle, only if unique names are used.

Please note this looks very similar to GoogleCloudPlatform/pubsub#256 which was closed a while back.

ERROR WorkerSourceTask{id=s1-xhugzz-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-s1-xhugzz-0]
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.base/java.lang.Thread.start0(Native Method)
        at java.base/java.lang.Thread.start(Thread.java:798)
        at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
        at java.base/java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1583)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:346)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:632)
        at com.google.api.gax.rpc.Watchdog.start(Watchdog.java:94)
        at com.google.api.gax.rpc.Watchdog.create(Watchdog.java:82)
        at com.google.api.gax.rpc.InstantiatingWatchdogProvider.getWatchdog(InstantiatingWatchdogProvider.java:111)
        at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:223)
        at com.google.cloud.pubsub.v1.stub.GrpcSubscriberStub.create(GrpcSubscriberStub.java:287)
        at com.google.pubsub.kafka.source.CloudPubSubGRPCSubscriber.makeSubscriber(CloudPubSubGRPCSubscriber.java:115)
        at com.google.pubsub.kafka.source.CloudPubSubGRPCSubscriber.pull(CloudPubSubGRPCSubscriber.java:67)
        at com.google.pubsub.kafka.source.CloudPubSubRoundRobinSubscriber.pull(CloudPubSubRoundRobinSubscriber.java:60)
        at com.google.pubsub.kafka.source.AckBatchingSubscriber.pull(AckBatchingSubscriber.java:57)
        at com.google.pubsub.kafka.source.CloudPubSubSourceTask.poll(CloudPubSubSourceTask.java:187)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:273)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:240)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:182)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:231)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
INFO [Producer clientId=connector-producer-s1-xhugzz-0] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer) [task-thread-s1-xhugzz-0]