feat(sqs4ses): replace Kafka Connector Direct with Kafka to queue, then queue to email.

[benjaminpaige]

| Email processing pipeline lacks delay queue between Kafka and Lambda functions. | Introduced an SQS delay queue to improve fault tolerance and added monitoring for delays and errors. |

🔧 Changes

• Added an SQS delay queue between Kafka and Lambda for improved message handling.
• Updated processEmails.ts to handle both Kafka and SQS events:
  • Forward Kafka events to the delay queue.
  • Process delayed SQS messages for email generation and sending.
• Enhanced error handling in processEmails.ts:
  • Implemented error handling for failed SQS SendMessageCommand.
  • Added debug logging for easier troubleshooting.
• Updated email.ts stack:
  • Configured a new delay queue with a 2-minute delivery delay and required IAM permissions.
  • Added CloudWatch alarms for Lambda errors, throttling, and DLQ message count.
  • Improved alarm configurations and added SNS notifications for critical events.
• Documented key functions and stack components with JSDoc comments for clarity.

🚀 How to Test

1. Deploy the updated stack:
   • Ensure the environment variables (e.g., DELAY_QUEUE_URL) are set correctly in the Lambda configurations.
2. Test Kafka event processing:
   • Publish Kafka messages and verify that they are forwarded to the SQS delay queue.
3. Test SQS message processing:
   • Check if delayed messages from the queue are processed by the Lambda and trigger the email sending flow.
4. Simulate failure scenarios:
   • Introduce errors in the processing pipeline to validate error handling, DLQ behavior, and alarm triggers.
5. Review CloudWatch logs and alarms:
   • Confirm that appropriate metrics are being tracked and alarms are configured as expected.

📄 Additional Notes

• The processEmails.ts handler now logs detailed debug information for both Kafka and SQS events, aiding in identifying potential issues.
• New alarms monitor Lambda execution errors, throttling, and DLQ activity, ensuring visibility into critical operational metrics.
• This update is backward-compatible with the existing email processing logic.

[thwalker6]

Why are you deleting processemails? We have a lot of unit test that depend on that, and if this is a refactor you will need to update the test appropriately.

[thwalker6]

make sure we dont lose any coverage on this. A lot of test go through this handler

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	80.15%	3834 / 4783
🔵	Statements	79.95%	4189 / 5239
🔵	Functions	72.37%	1129 / 1560
🔵	Branches	67.32%	1889 / 2806

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
lib/lambda/delayedEmailProcessor.ts	12.5%	6.66%	25%	11.62%	29-226, 261-285, 303-337
lib/lambda/kafkaToSqs.ts	50%	16.66%	100%	50%	13-34
lib/libs/utils.ts	100%	100%	100%	100%
lib/libs/email/preview/InitialSubmissions/State/Temp_Extension.tsx	100%	100%	100%	100%

Generated in workflow #1742 for commit d7f1fb5 by the Vitest Coverage Report Action