Notifications System Architecture Proposal

[BogoCvetkov]

Motivation and context

Motivation

• Currently the notification capabilities of the platform are quite rudimental - they are limited to the essential things - account activation email, reset password etc.
• Now we want to improve our platform by levaraging more sophisticated notification such as:
  • allowing the users to subscribe to certain type of events on the platform - a campaign raises the target amount, a new campaign was opened, there was an update on a campaign status etc.
  • sending complimentary emails to an user based on some achievements -donating a sum above a certain treshold, being an active monthly donator, donating multiple times to the same campaign etc.
  • notifying the admins about events in the platform, that need their attention - bank donation was unrecognized, the bank-consent for accessing transaction data is expiring, a payment has failed etc.
• Those type of Notifications would greatly increase the engagement and commitment with the platform from the users. On the admin side it would also make management easier, by keeping the responsible parties informed in real-time about the problems that emerge.
• In order to implement all of this we need a specifically designed Architecture that would allow the system to:
  • work reliably - ensuring notifications are not getting lost and always get sent
  • allow for easy addition of new events and handlers for those event. It must also be easy to add new means of notification (e.g email, chat-apps, sms etc.)
  • scale - the system should behave equally well when the number of users and notifications starts growing

Context

Taking into consideration the above mentioned points here are the features of the notification system that have been analyzed when designing the architecture outlined in the next part:

• Having notifications sent based on some type of event that has occurred in the system - e.g a campaign has raised all of the money, a bank donation was not recognized etc.
  • users/admins must be able to un/subscribe to those events
• Having notifications that are being sent automatically by the system to the users when a certain condition/achievement was met - for example sending thank-you emails to users that donate 3 months in a row, or they donate above 100$. Could also be used for reset-pass, activation-emails and other essentials.
  • In contrast to the above mentioned notifications, those won't be available for subscription
• Having notifications that would be handled by a 3rd party marketing tool - The marketing team would set-up campaign pipelines in the respective tools (e.g MailChimp) which will do the heavylifting. In those cases the platform won't be resposible for directly communicating with an SMTP server, rather it will send the user_email to the marketing platform trough it's API.

Architecture Overview

DB Schema

To accommodate for the dynamic nature of the notification system, we would have a many-to-many relationship subscription table that maps the users to the events and an additional table for storing the templates for the notification messages. So we will have 3 new tables:

• Events Table

  • id (primary key)
  • name - name of the event
  • description
  • can_subscribe - as mentioned above, not all events will be able for explicit subscription(signup,register, etc.), so we need to distinguish them
  • type - user/admin or user + admin- determine for whom the events will be important -this field should make it easier on the FE to recognize the events

• Templates Table

  • id (primary key)
  • name - name of the template
  • description
  • content - the actual notification message
  • content_type - HTML, Plain etc.
    • We should think how the dynamic data( {{user.name}} etc.) would be determined in the templates message
  • event_id - (foreign key of the event)
  • type - system/ custom

• Subscriptions Table

  • id (primary key)
  • uid (foreign key of the user)
  • event_id (foreign key of the event)
  • template_id (foreign key of the template)
  • medium - Email, SMS, Viber,Discord etc.

Separating everything into it's own table allows for the following flexibility:

• Subscribing for a particular event to receive messages on different channels (email, SMS). Intergrating more channels will be quite easy with this approach
• For the same event and chanel multiple templates could be created, so that a particular group of users receive a different message than another group subscribed to the same event
• Flexibility regarding template and subscription management and creation

BE Implementation

Having the ability to send multiple and different types of notifications and doing so reliably and efficiently requires a more robust and scalable approach. This is why a simple background worker won't cut it. That's why the best approach here would be to use a battle tested Message Broker. The two best options (taking into considerations the needs of the platform) are:

• RabbitMQ - the most popular and proven Message Broker
  • Probably the better and most robust option. Since it's a dedicated message broker it has many vital features for ensuring message handling (retying failed jobs, recovering from restarts, pattern matching and routing, publisher confirmations, monitoring dashboard, low-latency etc. ).
• Redis - lightweight and easy to implement the pub/sub messaging pattern
  • It's simpler and thus a bit easier to configure (hypothetically). Since it's not a dedicated Message Queue it lacks a lot of the functionalities of RabbitMQ that are important, so it won't scale that well. It's a good option too but if we are not going to need Redis for anything else RabbitMQ is preferable

Advantages

1. Both of those options are battle-tested and free
2. NestJS provides an easy integration with the two
3. Setting them up won't be hard - we would use the official docker images
4. Using this approach will make sure that all notifications are handled and send and nothing is lost along the way - especially if we use RabbitMQ
5. As the platform grows it's almost guaranteed that we will be needing some sort of background workers for handling some additional tasks or maybe we would have some new microservices that need to communicate. Already having a Message Broker would make it really easy to spawn additional workers and queues for handling different tasks and managing how and when they are being handled.
6. Those provide some management on their resource usage, like for example how much data to keep in memory.

Actual Implementation

• Anywhere in the system where we need a notification to be send we would just have to emit an event to the Message Broker and leave the rest to the workers. Examples:
  • A user subscribes to recieve notifications when a campaign raises the target amount. In the donations route when the vault is filled, it would emit an event funding_raised and then everyone subscribed would be notified
  • The background task that imports IRIS transactions encounters an unrecognized donation it emits an event unrecognized_donation, where all subscribed admins will get notified that an action should be taken
  • A cron job that that check for certain conditions on equal intervals and emits events if they are met
• Email template generation and compilation could be done using Handlebars.js as a popular framework for this particular task

DevOps Considerations

1. We would have to manage 2 more containers in the K8s cluster
   • The Message Queue itself
   • A NestJS microservice that would handle all the consumers (workers), that would process the jobs in the queue
2. We should evaluate how much complexity would be added to the maintenance of the infrastructure and the system

FE Considerations

• We should generally discuss if the template creation will be on our side or if it will be offloaded to a marketing platform (MailChimp, MailGun, etc.). In the second case all we have to do is communicate with the marketing platform's API and in the case of the first approach, the following should be considered:

  1. We should think of how we're are going to define and create the HTML templates for the notifications. A WYSIWYG editor is probably the best way to visually create HTML .We already have a WYSIWYG editor implemented on the FE - React-Quill. It should be enough for defining and styling simple to moderately complex emails.
  2. The more tricky part would be defining the dynamic data(e.g placeholders) inside the templates - {{user.name}}, {{user.title}},{{campaign.name}},{{campaign.vault.amount}} etc. It's easy to define those when composing the template, the complex part would be defining which exact variables each template can use and how to monitor that only those variables are being used. Example:
     • Sending Notification to the admins that an unrecognized bank donation was detected would probably need - {{admin.name}}, {{transaction.id}}, {{transaction.sender}}
     • Sending Notification to an user that a campaign he is interested in has raised all the money it needs would need another set of variables - {{user.name}}, {{campaign.name}},{{campaign.vault.amount}}
  • So there should be an intuitive way to tell the person composing a template for a particular notification what {{placeholders}} he can/must define in it.
    • One approach I can think of is to pre-define the available placeholders for each template inside the DB in the Templates Table
    • Another approach would be to let the user freely choose any placeholder from a list of available placeholders. And then when compiling the template the system will recognize the required placeholders and provide the needed resources(user/campaign/vault data, etc.) to populate the real data. This approach will provide flexibility to create more engaging emails, but might also introduce more complexity in regards to development

UI/UX Considerations

1. How should the interface be organized so that it will be intuitive and easy for the admins to manage the the resources around the Notification system. Here are the things that should be managed:
   • Template creation - a section in the Admin dashboard for defining and editing the templates
     • When creating a template the user should be able to see the available {{placeholders}} he can put in the template
   • Subscription section - A place where all subscriptions can be viewed and managed
2. It should also be considered how the subscription to events would happen.

[igoychev]

Hi @BogoCvetkov and awesome work on describing the notifications!

All looks quite good, I just have a few comments to try make our life simpler:

• Regarding the Templates:
  • we need to check if we could rely on SendGrid template management so that the Marketing team could edit the Templates there, so that we don't need to implement Template UI on our side - https://docs.sendgrid.com/ui/sending-email/working-with-marketing-campaigns-email-designs
  • on the backend we will still need the association of which event will be using which template, so the template table will still be needed, just the content field will not be there
  • on the Admin UI we will still need to have section for associating template and available parameters as you desribed
• Regarding the Message Queue:
  • I wonder if it is possible to use Sendgrid API directly so that there is no need of adding RabbitMQ or REDIS on our side?
  • SendGrid should have already notification tracking mechanism that help us
• Regarding the subscription UI:
  • on the User Profile page we can show the list of possible subscriptions so the user can subscribe/unsubscribe as well as the link to it should be at the bottom of all mails
  • on the Admin UI we need to have a section for enabling/disabling the types of Notifications too

WIth this said, I think we can start the implementation as per your description and gradually extend to richer functionality.

[imilchev]

Very well described solution! Congrats on that one!

I am with @igoychev on this one and think we should stay away from adding even more software to our stack (it is already quite complicated). Reading through your notes I see that the idea is to have a generic eventing system that would allow us to send real-time updates in the UI, email notifications, marketing messages and whatever else we come up with later.

Based on the description it looks to me that we can go for a much simpler solution. What you are proposing is a valid way of implementing this but I think it's over-engineered for our use-case. Your solution is intended for a highly-available, high-throughput and distributed processing implementations. I think we have the need for neither of those:

• we don't have a large amount of events to need distributed storage and processing
• this is not a business-critical functionality. In case it is broken for a while, nobody would be affected by it

My proposal for solving this challenge is to use a simple table in Postgres. Actions in the platform can store the events in this table with all the necessary data. If we want to make it super generic we can use a JSONB column and store json objects directly.

We can then make a separate process (new deployment) that will be processing these events. I would say that we don't need to process anything in parallel. We just go top-to-bottom 1-by-1 and handle each event. This processor should store somewhere in the DB the last event that has been processed. In case the processor crashes or is re-deployed, it will read from the DB the last processed event ID and will start from the one after it.

We can make the processor's architecture generic by allowing the registration of multiple plugins that handle the different types of events. We can store the events in the DB forever (at least for now), which will allow us to do re-processing if needed.

[BogoCvetkov]

• I agree! Giving it a second thought this would indeed introduce unnecessary complexity - taking into consideration the scale and needs of the platform.
• Great argument about the business-critical functionality. I guess I was overindulging myself in the technical intricacies of the task rather than keeping a closer focus on the business and practical aspects of it.
• I'm also thinking about using a database table for the purpose of processing the events. Would definitely consider the option for storing data as JSONB.
• Considering what @igoychev commented above, we may just get away with a simple background worker (like the one we are already using to import bank donations) - which won't require spawning any separate processes(deployments). But in case we need a more generic solution that will handle other events in the future, I also do think that a separate deployment would be a more scalable approach.