[APIM-4.1.0] Unable to invoke the deployed api through the one of the GW

[DaniduWijendra]

Description

We got an intermittent issue with some of the gateways that APIs are not getting deployed on some gateways and received the 404 error while invoking APIs. During the analysis, we noticed the following error trace on the broker logs.

WARN {org.wso2.andes.store.rdbms.RDBMSMessageStoreImpl} - Dropped message with ID: 107228407266803712 since queue: AMQP_Topic_notification_NODE:vm-madriddigitalsub-425-2.subnet08181351.vcn08181351.oraclevcn.com/10.0.0.170 does not exist org.h2.jdbc.JdbcSQLIntegrityConstraintViolationException: Referential integrity constraint violation: "CONSTRAINT_DCF: PUBLIC.MB_METADATA FOREIGN KEY(QUEUE_ID) REFERENCES PUBLIC.MB_QUEUE_MAPPING(QUEUE_ID) (419)"; SQL statement:
INSERT INTO MB_METADATA (MESSAGE_ID,QUEUE_ID,DLC_QUEUE_ID,MESSAGE_METADATA) VALUES ( ?,?,-1,? ) [23506-210]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:527)
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:496)
	at org.h2.message.DbException.get(DbException.java:227)
	at org.h2.message.DbException.get(DbException.java:203)
	at org.h2.constraint.ConstraintReferential.checkRowOwnTable(ConstraintReferential.java:311)
	at org.h2.constraint.ConstraintReferential.checkRow(ConstraintReferential.java:252)
	at org.h2.table.Table.fireConstraints(Table.java:1177)
	at org.h2.table.Table.fireAfterRow(Table.java:1195)
	at org.h2.command.dml.Insert.insertRows(Insert.java:188)
	at org.h2.command.dml.Insert.update(Insert.java:135)
	at org.h2.command.dml.DataChangeStatement.update(DataChangeStatement.java:61)
	at org.h2.command.CommandContainer.update(CommandContainer.java:174)
	at org.h2.command.Command.executeUpdate(Command.java:252)
	at org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:254)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:118)
	at com.sun.proxy.$Proxy52.execute(Unknown Source)
	at org.wso2.andes.store.rdbms.RDBMSMessageStoreImpl.storeMessage(RDBMSMessageStoreImpl.java:486)
	at org.wso2.andes.store.rdbms.RDBMSMessageStoreImpl.storeMessages(RDBMSMessageStoreImpl.java:411)
	at org.wso2.andes.store.FailureObservingMessageStore.storeMessages(FailureObservingMessageStore.java:123)
	at org.wso2.andes.kernel.MessagingEngine.messagesReceived(MessagingEngine.java:173)
	at org.wso2.andes.kernel.disruptor.inbound.MessageWriter.writeMessages(MessageWriter.java:100)
	at org.wso2.andes.kernel.disruptor.ConcurrentBatchEventHandler.onEvent(ConcurrentBatchEventHandler.java:137)
	at org.wso2.andes.kernel.disruptor.ConcurrentBatchEventHandler.onEvent(ConcurrentBatchEventHandler.java:40)
	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

After a detailed examination of database dumps, heap dumps, and system logs, we discovered inconsistencies in the data, specifically in the mapping between queue names and queue IDs. It appears that some necessary details are present in the system's memory but are absent in the database. This discrepancy causes constraint violation errors when we attempt to publish API creation details to the gateway nodes.

Steps to Reproduce

• Configure an APIM server with an external gateway.
• Create an API and deploy it on all the gateways.
• Try to invoke the API on each gateway.

Affected Component

APIM

Version

4.1.0

Environment Details (with versions)

No response

Relevant Log Output

No response

Related Issues

No response

Suggested Labels

No response

[shilmyhasan]

Public fix wso2/andes#1060