Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kafka topic migration #966

Open
5 of 6 tasks
Theodus opened this issue Oct 22, 2024 · 0 comments
Open
5 of 6 tasks

kafka topic migration #966

Theodus opened this issue Oct 22, 2024 · 0 comments

Comments

@Theodus
Copy link
Member

Theodus commented Oct 22, 2024

This issue tracks the effort to migrate the kafka messages exported by the gateway. The goals are to reduce the message production cost for the gateway, and to shift much of the backward compatibility concerns onto titorelli.

  • Add titorelli support for consuming a new topic from the gateway, and optionally exporting the legacy gateway_client_query_results & gateway_indexer_attempts topics.
    • The new topic would be a single protobuf message, per client query, containing the necessary data points to reconstruct the legacy topics. It should also fulfill the needs of QoS Oracle V2.
  • Replace the gateway reports to produce the new topic.
  • Update tap-escrow-manager to use gateway_queries as realtime topic
  • Remove titorelli support for the old gateway_indexer_fees topic.
  • Update titorelli to produce aggregations for QoS Oracle V2
  • Update titorelli aggregations to support billing improvements
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant