Dasb 615- Implement RabbitMQ plugin for Local Development #172

TeaSmith7 · 2024-08-26T13:47:07Z

This PR includes implementation of the RabbitMQ plugin for local development. The key changes include:

RabbitMQ plugin: setup and configuration of RabbitMQ for local runs as well as Docker Compose Configuration that setup a local RabbitMQ instance with the management UI accessible via port: 15672.
Implemented utils package with schema validation logic to be re-used across supported messaging systems, currently Azure Service Bus, RabbitMQ and eventually AWS SNS/SQS.
Updated README to include instructions and notes for RabbitMQ setup.

Added dependency build.gradle Added RabbitMQ service as dependency to Docker Compose with rabbitMQ management plugin. Introduced environment variables for AWS, Azure Service Bus, RabbitMQ to specify which system to use. Initial work for RabbitMQ plugin that connects to the RabbitMQ, and listens to the queue

updated Readme with environment variables configuration for Messaging Systems, as well as steps to send reports to rabbitMQ server. Improvements to RabbitMQ plugin: Added exception handling, improved Kdoc comments. Removed routingKey and ExchangeType from environment variables. Added utils package and created HelpersFunction for the purpose to re-use functions across asb, rabbitMQ and eventually aws sns/sqs plugins.

Application.kt- added cosmosModule back RabbitMQ.kt - updated logging RabbitMQProcessor.kt - updated logging and calling validateJsonSchema() from HelperFunctions.kt to proceed with report validation HelperFunctions.kt - moved functions for report validation logic to be re-used for all messaging systems

When disableValidation is True and the message is not valid JSON, the message is sent to the DLQ, and a return statement is added to prevent report creation.

…ect their purpose - Class: RabbitMQUtil() -> RabbitMQService() - File: HelperFunctions.kt -> SchemaValidationFunctions.kt

refactored service bus plugin to use schema validation related functions from utils package. Updated log messages and Kdoc comments

…Bus module the function is called when application goes down.

mkrystof

Some changes requested, please. Thanks for updating the readme too!

mkrystof · 2024-08-27T13:23:45Z

...ink-ktor/src/main/kotlin/gov/cdc/ocio/processingstatusapi/utils/SchemaValidationFunctions.kt

+
+
+
+object Helpers {


Let's encapsulate all of this in a class called SchemaValidation. Make the contents of the Helpers a companion object of the SchemaValidation class.

mkrystof · 2024-08-27T13:24:36Z

...ink-ktor/src/main/kotlin/gov/cdc/ocio/processingstatusapi/utils/SchemaValidationFunctions.kt

+    }
+}
+@Throws(BadRequestException::class)
+fun createReport(createReportMessage: CreateReportSBMessage) {


Would you please generate the missing function header comments?

mkrystof · 2024-08-27T13:24:51Z

...ink-ktor/src/main/kotlin/gov/cdc/ocio/processingstatusapi/utils/SchemaValidationFunctions.kt

+        false
+    }
+}
+fun sendToDeadLetter(


Missing comment header

mkrystof · 2024-08-27T13:30:13Z

pstatus-report-sink-ktor/src/main/kotlin/gov/cdc/ocio/processingstatusapi/Application.kt

@@ -52,7 +67,28 @@ fun main(args: Array<String>) {
 */
 fun Application.module() {
    configureRouting()


Would you please update the HealthQueryService to selectively check for the health of the messaging service? In other words, right now it is always checking ASB, but needs to check RabbitMQ, etc depending on the config.

mkrystof · 2024-08-27T13:32:36Z

pstatus-report-sink-ktor/src/main/kotlin/gov/cdc/ocio/processingstatusapi/Application.kt

+            }
+            MessageSystem.RABBITMQ.toString() -> {
+                single(createdAtStart = true) {
+                    RabbitMQService(environment.config, configurationPath = "rabbitMQ")


For clarity, similar to the ASB service configuration we may want to rename this to RabbitMQServiceConfiguration. Also, this will align with naming for AzureServiceBusConfiguration.

Update Kdoc comments for functions Create SchemaValidation class and move contents of Helpers object into a companion object Rename RabbitMQService to RabbitMQServiceConfiguration' Note: This PR does not include requested changes for HealthQueryService, will do that next.

Added msgType to Koin module for dependency injection. Added logic for getting Health of RabbitMQ messaging service. Updated HealthCheck.kt to retrieve the health status of the messaging service based on the msgType from the config.

removed irrelevant exceptions

`checkAndReplacedDeprecatedFields()`

mkrystof · 2024-08-29T23:45:43Z

...ink-ktor/src/main/kotlin/gov/cdc/ocio/processingstatusapi/utils/SchemaValidationFunctions.kt

+ * @param jsonString to be checked
+ * @return boolean true if its valid JSON otherwise false
+ */
+fun isJsonValid(jsonString: String): Boolean {


As a general good practice, we should strive to avoid writing functions that aren't part of a class. For example, the isJsonValid could be an Extension of String. See https://kotlinlang.org/docs/extensions.html.

mkrystof · 2024-08-29T23:46:21Z

...ink-ktor/src/main/kotlin/gov/cdc/ocio/processingstatusapi/utils/SchemaValidationFunctions.kt

+ * @param messageAsString message to be checked against depreciated fields.
+ * @return updated message if depreciated fields were found.
+ */
+fun checkAndReplaceDeprecatedFields(messageAsString: String): String {


This function might be best if made part of the Report class. It could be a companion object.

mkrystof · 2024-08-29T23:46:40Z

...ink-ktor/src/main/kotlin/gov/cdc/ocio/processingstatusapi/utils/SchemaValidationFunctions.kt

+ * @param message ReceivedMessage(from Azure Service Bus, RabbitMQ Queue or AWS SNS/SQS)
+ * @throws BadRequestException
+ */
+fun validateJsonSchema(message: String) {


Same sentiment as above - can we find a home for all these functions?

removed SchemaValidationFunctions.kt and merged functions into SchemaValidation.kt Updated RabbitMQProcessor() and ServiceBusProcessor() to use functions from SchemaValidation.kt Simplified loadSchemaFile() by removing the schemaDirectoryPath parameter, since the schema is retrieved using javaClass.classLoader.getResource() and folder name is constant schema.

TeaSmith7 · 2024-09-03T11:30:03Z

pstatus-report-sink-ktor/src/main/kotlin/gov/cdc/ocio/processingstatusapi/HealthCheck.kt

@@ -85,8 +102,12 @@ class HealthQueryService: KoinComponent {
    fun getHealth(): HealthCheck {


sonar cloud tags this function with brain overload, I agree it's too much going on for a single function. I will refactor with private functions for checking health.

mkrystof · 2024-09-03T15:38:16Z

...-report-sink-ktor/src/main/kotlin/gov/cdc/ocio/processingstatusapi/utils/SchemaValidation.kt

+import java.util.*
+import javax.activation.MimeType
+
+class SchemaValidation {


Would you please add a class comment header? Description should provide overview of the class - intent, usage, etc.

mkrystof · 2024-09-03T15:38:50Z

pstatus-report-sink-ktor/src/main/kotlin/gov/cdc/ocio/processingstatusapi/plugins/RabbitMQ.kt

+}
+
+
+class RabbitMQServiceConfiguration(config: ApplicationConfig, configurationPath: String? = null) {


Please add class comment header.

mkrystof · 2024-09-03T15:39:24Z

pstatus-report-sink-ktor/src/main/kotlin/gov/cdc/ocio/processingstatusapi/plugins/RabbitMQ.kt

+import org.apache.qpid.proton.TimeoutException
+import java.io.IOException
+
+object RMQConstants {


I think this would be better as a companion object of RabbitMQServiceConfiguration.

RabbitMQ.kt - moved constants to under RabbitMQServiceConfiguration class and added class comment header SchemaValidation.kt - added class comment header

mkrystof

LGTM!

sonarcloud · 2024-09-03T17:06:43Z

Quality Gate failed

Failed conditions
1 Security Hotspot

See analysis details on SonarCloud

TeaSmith7 added 10 commits August 16, 2024 17:18

Merge remote-tracking branch 'origin/develop' into DASB-615

3c1c6d7

Address the warning with unused value-parameter for service bus.

86d7027

When disableValidation is True and the message is not valid JSON, the message is sent to the DLQ, and a return statement is added to prevent report creation.

Renamed class and the file containing helper functions to better refl…

ff7e2eb

…ect their purpose - Class: RabbitMQUtil() -> RabbitMQService() - File: HelperFunctions.kt -> SchemaValidationFunctions.kt

Initial work for service bus plugin refactor

1784239

refactored service bus plugin to use schema validation related functions from utils package. Updated log messages and Kdoc comments

created private function cleanupResourcesAndUnsubscribe() for Service…

ac845fd

…Bus module the function is called when application goes down.

remove connection string to asb from logs

f0497a3

removed contentType as its not being used

a67d59b

TeaSmith7 requested review from mkrystof and manu-govind August 26, 2024 13:47

Commit missing gradle-wrapper.jar and related files

5578be9

mkrystof requested changes Aug 27, 2024

View reviewed changes

TeaSmith7 added 4 commits August 28, 2024 10:08

Update @throws annotation for isRabbitMQHealthy

2fe84c0

removed irrelevant exceptions

fixes a bug where I was not using message returned by

f09b333

`checkAndReplacedDeprecatedFields()`

mkrystof requested changes Aug 29, 2024

View reviewed changes

TeaSmith7 commented Sep 3, 2024

View reviewed changes

mkrystof requested changes Sep 3, 2024

View reviewed changes

Addresses requested changes for the PR

8bf608d

RabbitMQ.kt - moved constants to under RabbitMQServiceConfiguration class and added class comment header SchemaValidation.kt - added class comment header

mkrystof approved these changes Sep 3, 2024

View reviewed changes

Merge branch 'develop' into DASB-615

d699454

TeaSmith7 merged commit 432ef6a into develop Sep 3, 2024
0 of 2 checks passed

TeaSmith7 deleted the DASB-615 branch September 4, 2024 18:06

TeaSmith7 restored the DASB-615 branch September 4, 2024 18:07

TeaSmith7 deleted the DASB-615 branch September 4, 2024 18:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dasb 615- Implement RabbitMQ plugin for Local Development #172

Dasb 615- Implement RabbitMQ plugin for Local Development #172

TeaSmith7 commented Aug 26, 2024

mkrystof left a comment

mkrystof Aug 27, 2024

mkrystof Aug 27, 2024

mkrystof Aug 27, 2024

mkrystof Aug 27, 2024

mkrystof Aug 27, 2024

mkrystof Aug 29, 2024

mkrystof Aug 29, 2024

mkrystof Aug 29, 2024

TeaSmith7 Sep 3, 2024 •

edited

Loading

mkrystof Sep 3, 2024

mkrystof Sep 3, 2024

mkrystof Sep 3, 2024

mkrystof left a comment

sonarcloud bot commented Sep 3, 2024

		@@ -85,8 +102,12 @@ class HealthQueryService: KoinComponent {
		fun getHealth(): HealthCheck {

		}


		class RabbitMQServiceConfiguration(config: ApplicationConfig, configurationPath: String? = null) {

Dasb 615- Implement RabbitMQ plugin for Local Development #172

Dasb 615- Implement RabbitMQ plugin for Local Development #172

Conversation

TeaSmith7 commented Aug 26, 2024

mkrystof left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TeaSmith7 Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkrystof left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Sep 3, 2024

Quality Gate failed

TeaSmith7 Sep 3, 2024 •

edited

Loading