Getting ConcurrentModificationException stop the throttling in TM component

[SupunTharakaSandaruwan]

Description

Single GW node failure or adding a new GW component in distributed setup, stops the throttling decision publishing at TM node with ConcurrentModificationException exception.

[2023-11-12 14:23:43,862] ERROR - DeliveryTaskExceptionHandler Error occurred while processing task. Task id AMQP_Topic_throttledata_NODE:supuntharaka.local/192.1xx.0.1xx
java.util.ConcurrentModificationException: null
   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:911) ~[?:1.8.0_392-internal]
   at java.util.ArrayList$Itr.next(ArrayList.java:861) ~[?:1.8.0_392-internal]
   at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1044) ~[?:1.8.0_392-internal]
   at org.wso2.andes.kernel.NoLossBurstTopicMessageDeliveryImpl.deliverMessageToSubscriptions(NoLossBurstTopicMessageDeliveryImpl.java:71) ~[andes_3.3.11.jar:?]
   at org.wso2.andes.kernel.MessageFlusher.sendMessagesToSubscriptions(MessageFlusher.java:150) ~[andes_3.3.11.jar:?]
   at org.wso2.andes.kernel.slot.MessageDeliveryTask.sendMessagesToSubscriptions(MessageDeliveryTask.java:121) ~[andes_3.3.11.jar:?]
   at org.wso2.andes.kernel.slot.MessageDeliveryTask.call(MessageDeliveryTask.java:98) ~[andes_3.3.11.jar:?]
   at org.wso2.andes.kernel.slot.MessageDeliveryTask.call(MessageDeliveryTask.java:32) ~[andes_3.3.11.jar:?]
   at org.wso2.andes.task.TaskHolder.executeTask(TaskHolder.java:75) ~[andes_3.3.11.jar:?]
   at org.wso2.andes.task.TaskProcessor.call(TaskProcessor.java:85) [andes_3.3.11.jar:?]
   at org.wso2.andes.task.TaskProcessor.call(TaskProcessor.java:32) [andes_3.3.11.jar:?]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_392-internal]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_392-internal]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_392-internal]
   at java.lang.Thread.run(Thread.java:750) [?:1.8.0_392-internal]

Steps to Reproduce

• Setup a distributed pattern with multiple GWs and separate TM node using wso2am-4.0.0
• Published a few APIs to all the GWs.
• Subscribe it and invoke the APIs to get throttled out.
• During the API invocations, stop GW servers and Start
• The above concurrent modification exception will be thrown

Affected Component

APIM

Version

4.0.0

Environment Details (with versions)

No response

Relevant Log Output

No response

Related Issues

No response

Suggested Labels

No response