NLU Rate Limits

[caiosba]

Description

As part of the ongoing development of the Natural Language Understanding (NLU) feature, it has been identified that the current implementation of the text similarity service lacks robust handling of concurrent requests. The absence of efficient concurrency management may lead to degraded performance and potential service disruptions when multiple requests are made simultaneously.

Issue

The primary issue is the lack of a concurrency control mechanism, resulting in potential race conditions and suboptimal response times during peak usage periods. It is imperative to enhance the system's capability to handle concurrent requests seamlessly to ensure a smooth and responsive user experience.

Objective

The goal of this ticket is to implement rate limits as a concurrent request handling mechanism for the NLU feature.

Changes

• Store messages even before they are parsed
• Keep a global NLU counter and increment/decrement it in a thread-safe way as NLU operations start and finish
• Global rate limit: do not process using NLU when the current number of NLU operations is greater than the limit defined in a configuration key
• User rate limit: do not process using NLU when the user who sent the message has sent more messages in the last seconds than the limit defined in a configuration key

[codeclimate]

Code Climate has analyzed commit 53b1498 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (100% is the threshold).

This pull request will bring the total coverage in the repository to 100.0% (0.0% change).

View more on Code Climate.