Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply Back-pressure to outgoing /bulkTransaction requests from SDK-Scheme-Adapter to Switch #3026

Closed
5 of 16 tasks
mdebarros opened this issue Nov 21, 2022 · 2 comments
Closed
5 of 16 tasks
Assignees
Labels
core-dev-squad oss-bulk Mojaloop bulk implementation and support oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it story

Comments

@mdebarros
Copy link
Member

mdebarros commented Nov 21, 2022

Goal:

As a Payer FSP

I want to ensure there is adequate Back-pressure when sending associated requests from a /bulkTransaction processing request

so that the Mojaloop Switch is able to process the individual Discovery requests, Bulk Requests for Quotes and Transfers in a consistent predictable manner.

Acceptance Criteria:

  • Characterization of the SDK-Scheme-Adapter utilising a Platformed-shared-bc Kafka Consumer library that supports Back-pressure when consuming messages from the Kafka Topic

Complexity: High

Uncertainty: Medium


Tasks:

  • Update Platformed-shared-bc Kafka Consumer library
  • Test Platformed-shared-bc Kafka Consumer library
  • Created a snapshot release for the Platformed-shared-bc Kafka Consumer library
  • Update SDK-Scheme-Adapter to utilise the modified Platformed-shared-bc Kafka Consumer library
  • Run Characterization locally to compare Pre and Post changes to the Platformed-shared-bc Kafka Consumer library
  • Determine if Characterization should be done remotely on a similiar platform to the previous Characterization effort.

Done

  • Acceptance Criteria pass
  • Designs are up-to date
  • Unit Tests pass
  • Integration Tests pass
  • Code Style & Coverage meets standards
  • Changes made to config (default.json) are broadcast to team and follow-up tasks added to update helm charts and other deployment config.
  • ...

Pull Requests:

Follow-up:

  • N/A

Dependencies:

  • N/A

Accountability:

@mdebarros mdebarros added core-dev-squad oss-bulk Mojaloop bulk implementation and support oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it story labels Nov 21, 2022
@mdebarros mdebarros self-assigned this Nov 21, 2022
@mdebarros
Copy link
Member Author

mdebarros commented Dec 1, 2022

Observations

  • Test with existing SDK-Scheme-Adapter v21.4.0 with Kafka Consumer consuming as many messages as possible with no lock-stop processing resulted in a processing time of 37768ms with some failures. (ref: 20221201-std-80-err-TTK-Assertion-Report-multi-2022-12-01T09 29 33.478Z.html)
  • Test with modified SDK-Scheme-Adapter v21.6.0-snapshot.0 with Kafka Consumer providing Back-pressure by consuming a single message at a time resulted in a bulkTransaction with 80 individual transfers taking 22585ms with no failures. (ref: 20221201-1msg-80-TTK-Assertion-Report-multi-2022-12-01T12 35 35.016Z.html)

Test-Reports.zip

Findings

  • The above preliminary results indicate that Back-pressure would improve the overal performance characteristcs of a bulkTransaction request through a live v14.1.0 Mojaloop deployment.

Recommendations

  • Finalise the Kafka Consumer changes with an official release and update the SDK-Scheme-Adapter to support those changes.
  • Re-run the characterization using a purpose build environment for Performance testing
  • Re-run the characterization using the previous methodology used for demo'ing the SDK's capabilities through a simulated TTK switch

Code Branches

@mdebarros
Copy link
Member Author

Refer to the following comment for next steps: #3026 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core-dev-squad oss-bulk Mojaloop bulk implementation and support oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it story
Projects
None yet
Development

No branches or pull requests

1 participant