Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-708] Indexer track e2e latency #1237

Merged
merged 12 commits into from
Mar 28, 2024

Conversation

jonfung-dydx
Copy link
Contributor

@jonfung-dydx jonfung-dydx commented Mar 25, 2024

vulcan -> socks. Thread through the kafka headers.

Socks, emit the e2e latency.
Vulcan, emit the e2e latency.

Copy link

linear bot commented Mar 25, 2024

Copy link
Contributor

coderabbitai bot commented Mar 25, 2024

Walkthrough

The recent updates across various services and packages aim to improve metrics, timing calculations, and message handling. Changes include introducing a new constant for stats sampling, refining timing metrics, enhancing message timestamp handling, and incorporating message headers in processing calls. Additionally, there is a significant refactoring related to time functionalities, introducing a utility to convert protobuf timestamps to JavaScript Date objects, with corresponding test adjustments.

Changes

File(s) Summary
.../socks/src/lib/message-forwarder.ts Introduced STATS_NO_SAMPLING, refined time metrics, added originalMessageTimestamp check.
.../vulcan/src/lib/on-message.ts, .../vulcan/__tests__/lib/on-message.test.ts Added message time tracking, modified handler.handleUpdate to include headers, updated test assertions.
.../vulcan/src/handlers/order-place-handler.ts, .../vulcan/__tests__/handlers/order-remove-handler.test.ts Updated methods to accept headers: IHeaders, modified test calls to include defaultKafkaHeaders.
.../v4-protos/src/index.ts, .../v4-protos/src/utils.ts Exported 'utils', added protoTimestampToDate function and time conversion constants.
.../ender/__tests__/helpers/indexer-proto-helpers.ts, .../ender/src/lib/helper.ts Refactored time-related functionalities, removed redundant imports.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit-tests for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit tests for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit tests.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • The JSON schema for the configuration file is available here.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 82c2dec and cbc7a99.
Files selected for processing (6)
  • indexer/services/socks/src/lib/message-forwarder.ts (3 hunks)
  • indexer/services/vulcan/src/handlers/handler.ts (2 hunks)
  • indexer/services/vulcan/src/handlers/order-place-handler.ts (7 hunks)
  • indexer/services/vulcan/src/handlers/order-remove-handler.ts (11 hunks)
  • indexer/services/vulcan/src/handlers/order-update-handler.ts (3 hunks)
  • indexer/services/vulcan/src/lib/on-message.ts (2 hunks)
Additional comments: 24
indexer/services/vulcan/src/handlers/handler.ts (3)
  • 7-7: The import of IHeaders from kafkajs is correctly added to support the new functionality of passing headers through the handler methods. This aligns with the PR's objective to enhance end-to-end latency tracking.
  • 19-19: The abstract method handle now correctly accepts an additional parameter headers of type IHeaders. This change is necessary for threading through headers in various components of the indexer services, as mentioned in the PR objectives.
  • 22-23: The handleUpdate method has been updated to accept and pass the headers parameter to the handle method. This ensures that headers are correctly propagated through the system, allowing for enhanced tracking of message time since received.
indexer/services/vulcan/src/lib/on-message.ts (3)
  • 65-75: The addition of tracking for message_time_since_received using the originalMessageTimestamp from the message headers is a significant improvement. It aligns with the PR's goal to enhance the precision of latency tracking within the indexer's infrastructure. However, ensure that the message_received_timestamp header is consistently set across all messages to avoid potential NaN results from Date.now() - Number(undefined).
  • 101-101: The conditional check for originalMessageTimestamp !== undefined before calculating and recording the timing statistics is a good practice. It ensures that only messages with a valid originalMessageTimestamp header contribute to the latency metrics.
  • 101-101: The use of message.headers ?? {} as a fallback when headers are null is a prudent measure to ensure that the handleUpdate method call does not fail due to missing headers. This change supports the PR's objective by allowing for the inclusion of message_received_timestamp from the headers in statistics, even when headers are not present.
indexer/services/socks/src/lib/message-forwarder.ts (3)
  • 6-6: The introduction of the STATS_NO_SAMPLING constant is a relevant addition for adjusting the calculation of time metrics. This constant likely serves to ensure that certain statistics are always recorded without sampling, which is crucial for accurate latency tracking.
  • 99-109: Adding a check for originalMessageTimestamp before updating stats is a critical enhancement. This ensures that latency metrics are only updated when the originalMessageTimestamp is available, contributing to more accurate and meaningful statistics.
  • 142-147: Refining timing calculations with distinct start times for different operations, as seen in the forwardMessage method, is a good practice. It allows for more granular tracking of operation durations, aligning with the PR's goal to enhance latency tracking.
indexer/services/vulcan/src/handlers/order-update-handler.ts (3)
  • 27-27: The modification to include IHeaders in the import statement is correctly implemented to support the new functionality of passing headers through the handle method. This change is essential for threading through headers in the order update handling process.
  • 54-54: The handle method now correctly accepts an additional headers parameter of type IHeaders. This is a necessary change to ensure that headers are threaded through the system, allowing for enhanced tracking of message time since received.
  • 174-176: Including the message_received_timestamp from the headers when sending a message is a crucial enhancement. It ensures that the latency tracking mechanism can accurately account for the time since the message was received, aligning with the PR's objectives.
indexer/services/vulcan/src/handlers/order-place-handler.ts (6)
  • 33-33: The inclusion of IHeaders in the import statement is correctly implemented to support the new functionality of passing headers through the handle and sendCachedOrderUpdate methods. This change is essential for threading through headers in the order placement handling process.
  • 54-54: The handle method now correctly accepts an additional headers parameter of type IHeaders. This is a necessary change to ensure that headers are threaded through the system, allowing for enhanced tracking of message time since received.
  • 139-139: The sendCachedOrderUpdate method now correctly takes headers: IHeaders as a parameter. This ensures that when cached order updates are sent, they include the necessary headers for latency tracking.
  • 148-150: Including the message_received_timestamp from the headers when creating a subaccount message is a crucial enhancement. It ensures that the latency tracking mechanism can accurately account for the time since the message was received, aligning with the PR's objectives.
  • 164-166: Including the message_received_timestamp from the headers when creating an orderbook message is a crucial enhancement. It ensures that the latency tracking mechanism can accurately account for the time since the message was received, aligning with the PR's objectives.
  • 347-349: Including the message_received_timestamp from the headers when sending a cached order update is a crucial enhancement. It ensures that the latency tracking mechanism can accurately account for the time since the message was received, aligning with the PR's objectives.
indexer/services/vulcan/src/handlers/order-remove-handler.ts (6)
  • 71-71: The method handle now accepts an additional parameter headers of type IHeaders. This change aligns with the PR's objective to enhance e2e latency tracking by threading through headers in various components. Ensure that all calls to this method have been updated accordingly to pass the headers parameter.
  • 134-134: The method handleStatefulOrderCancelation has been updated to include the headers parameter. This is consistent with the PR's goal of improving latency tracking. Verify that the headers are being used effectively within the method to track latency-related metrics.
  • 138-138: The method handleOrderRemoval now requires the headers parameter. This modification supports the PR's aim of enhancing latency measurement. Confirm that the inclusion of headers is leveraged for latency tracking purposes throughout the method's implementation.
  • 248-248: The call to updateOrderbook within handleStatefulOrderCancelation now passes the headers parameter. This change is part of the effort to thread headers through the system for improved latency tracking. Ensure that the updateOrderbook method utilizes the headers effectively for latency-related calculations or logging.
  • 333-333: Similarly, in handleOrderRemoval, the updateOrderbook method is called with the headers parameter. This adjustment is in line with the PR's objectives. Check that the headers are being used appropriately within updateOrderbook to enhance latency tracking.
  • 350-350: The updateOrderbook method now includes the headers parameter. This modification supports the overarching goal of the PR to improve latency tracking by threading headers through various components. Verify that the headers are utilized effectively within the method for latency measurement or logging.

@jonfung-dydx jonfung-dydx requested a review from dydxwill March 26, 2024 15:38
Copy link
Contributor

@dydxwill dydxwill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also add unit tests?

indexer/services/socks/src/lib/message-forwarder.ts Outdated Show resolved Hide resolved
@@ -86,7 +98,7 @@ export async function onMessage(message: KafkaMessage): Promise<void> {
const handler: Handler = new (getHandler(update))!(
getTransactionHashFromHeaders(message.headers),
);
await handler.handleUpdate(update);
await handler.handleUpdate(update, message.headers ?? {});

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we also emit a stat here comparing to message_received_timestamp ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already do that here in the same code block right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's before handling the message though

@@ -125,12 +139,12 @@ export class MessageForwarder {
return;
}

const start: number = Date.now();
const startForwardMessage: number = Date.now();
this.forwardMessage(messageToForward);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function seems to send out websocket messages in batches for certain websocket topics. Can we pipe the initial timestamp all the way to the sendMessage function?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean we should add a message_received_timestamp field to

export interface OutgoingMessage {
  type: OutgoingMessageType;
  connection_id: string;
  message_id: number;
}
```?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nvm, the batches happen at periodic intervals so not necessary to pipe it through.

Does it make more sense to have the stat be end - Number(message.headers?.message_received_timestamp) (after calling forwardMessage) rather than start - Number(message.headers?.message_received_timestamp)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between cbc7a99 and 57c2f4b.
Files selected for processing (2)
  • indexer/services/socks/src/lib/message-forwarder.ts (3 hunks)
  • indexer/services/vulcan/src/lib/on-message.ts (2 hunks)
Files skipped from review as they are similar to previous changes (2)
  • indexer/services/socks/src/lib/message-forwarder.ts
  • indexer/services/vulcan/src/lib/on-message.ts

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 57c2f4b and 42b236f.
Files selected for processing (2)
  • indexer/services/vulcan/src/handlers/order-place-handler.ts (7 hunks)
  • indexer/services/vulcan/src/lib/on-message.ts (2 hunks)
Files skipped from review as they are similar to previous changes (2)
  • indexer/services/vulcan/src/handlers/order-place-handler.ts
  • indexer/services/vulcan/src/lib/on-message.ts

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 1

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 42b236f and 01cfc7e.
Files selected for processing (6)
  • indexer/packages/v4-protos/src/index.ts (1 hunks)
  • indexer/packages/v4-protos/src/utils.ts (1 hunks)
  • indexer/services/ender/tests/helpers/indexer-proto-helpers.ts (2 hunks)
  • indexer/services/ender/src/lib/helper.ts (3 hunks)
  • indexer/services/vulcan/tests/handlers/order-remove-handler.test.ts (23 hunks)
  • indexer/services/vulcan/tests/helpers/constants.ts (1 hunks)
Additional comments: 24
indexer/packages/v4-protos/src/utils.ts (2)
  • 3-4: Constants MILLIS_IN_NANOS and SECONDS_IN_MILLIS are correctly defined and named, providing clear intent for time conversion purposes.
  • 5-11: The protoTimestampToDate function correctly converts a protobuf Timestamp to a JavaScript Date object. Consider adding a comment explaining the conversion process for future maintainability.
indexer/packages/v4-protos/src/index.ts (1)
  • 19-19: The addition of the export statement for the 'utils' module is correctly implemented, enabling other parts of the application to utilize the utilities defined in utils.ts.
indexer/services/ender/__tests__/helpers/indexer-proto-helpers.ts (1)
  • 53-53: The update in imports for protoTimestampToDate, moving it to @dydxprotocol-indexer/v4-protos, is a good practice. It centralizes utility functions, reducing code duplication and improving maintainability.
indexer/services/vulcan/__tests__/handlers/order-remove-handler.test.ts (20)
  • 8-10: The addition of defaultTime from ../helpers/constants is a good practice for ensuring consistent time values across tests. This helps in creating predictable and reproducible test outcomes.
  • 141-143: The introduction of defaultKafkaHeaders with message_received_timestamp derived from defaultTime is a solid approach to simulate realistic Kafka headers in test scenarios. This ensures that tests accurately reflect the handling of headers in production code.
  • 191-194: It's commendable to see thorough testing of failure scenarios, ensuring that the system behaves as expected when encountering invalid updates. This contributes significantly to the robustness of the order removal process.
  • 209-212: Testing for early return conditions, such as being unable to find an order in Redis, is crucial for ensuring the resilience of the system. This test case effectively verifies that the system can gracefully handle missing data scenarios.
  • 233-236: Similar to the previous comment, testing for the absence of a perpetual market and ensuring the system's graceful handling of such a scenario is vital for maintaining system integrity under various conditions.
  • 341-344: The comprehensive testing of successful order removal across different order types and conditions demonstrates a deep understanding of the system's requirements. This ensures that the order removal process is well-tested for a variety of scenarios.
  • 494-497: Testing for the BEST_EFFORT_CANCELED status in order removal scenarios is important for ensuring that the system can handle different cancellation statuses appropriately. This adds another layer of robustness to the order management system.
  • 631-634: Ensuring that the orderbook levels cache is not affected when an order is removed but was not resting on the book is a critical aspect of maintaining data integrity. This test case effectively validates that behavior.
  • 772-775: Testing for scenarios where the total filled quantity exceeds the order's quantums and ensuring that the orderbook level is not incorrectly increased is crucial for maintaining accurate market data. This test case effectively addresses that concern.
  • 1053-1056: Logging errors when an order cannot be found in Postgres during stateful cancellation is essential for debugging and operational monitoring. This test ensures that such errors are correctly logged, aiding in system maintenance.
  • 1104-1107: Sending subaccount websocket messages for orders not found in Redis during stateful cancellation scenarios is an important aspect of ensuring that clients are informed about order status changes. This test case validates that functionality.
  • 1207-1210: The removal of stateful orders that are not resting on the book and ensuring that the orderbook levels cache is not affected is crucial for data accuracy. This test case effectively verifies that the system behaves as expected in such scenarios.
  • 1327-1330: Successfully removing stateful orders that are resting on the book and correctly updating the orderbook levels cache is vital for maintaining market integrity. This test case thoroughly validates that process.
  • 1474-1477: Ensuring that the orderbook level is not incorrectly increased when the total filled quantity exceeds the order's quantums in stateful cancellation scenarios is important for data accuracy. This test case addresses that concern effectively.
  • 1599-1602: Successfully removing expired orders and ensuring that the orderbook levels cache is correctly updated is crucial for the integrity of market data. This test case effectively validates the handling of indexer-expired orders.
  • 1733-1736: Testing for the successful removal of fully filled expired orders without sending unnecessary websocket messages is important for system efficiency. This test case ensures that the system behaves as expected in such scenarios.
  • 1798-1801: Handling errors gracefully when the latest block cannot be found during the expiry verification of indexer-expired orders is crucial for system reliability. This test case ensures that such errors are correctly logged and handled.
  • 1868-1871: Properly handling scenarios where an order cannot be found during the expiry verification of indexer-expired orders is important for system integrity. This test case ensures that the system logs such occurrences and exits gracefully.
  • 1944-1947: Ensuring that long-term orders are not incorrectly processed during the expiry verification of indexer-expired orders is crucial for maintaining data accuracy. This test case validates that the system correctly identifies and logs such scenarios.
  • 2009-2012: Testing for scenarios where an order is not actually expired during the expiry verification of indexer-expired orders is important for avoiding unnecessary cancellations. This test case ensures that the system correctly identifies such cases and exits gracefully.

Comment on lines +10 to +12
export const defaultTime: Timestamp = {
seconds: Long.fromValue(Math.floor(defaultDateTime.toSeconds()), true),
nanos: (defaultDateTime.toMillis() % SECONDS_IN_MILLIS) * MILLIS_IN_NANOS,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The calculation of nanos in defaultTime might not be correct. It seems to multiply the remainder of milliseconds by MILLIS_IN_NANOS, which would result in an incorrect value for nanoseconds. The correct approach should be to take the remainder of milliseconds (after converting to seconds) and then multiply by MILLIS_IN_NANOS to convert it to nanoseconds.

-  nanos: (defaultDateTime.toMillis() % SECONDS_IN_MILLIS) * MILLIS_IN_NANOS,
+  nanos: (defaultDateTime.toMillis() % SECONDS_IN_MILLIS) * 1000,

Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change
export const defaultTime: Timestamp = {
seconds: Long.fromValue(Math.floor(defaultDateTime.toSeconds()), true),
nanos: (defaultDateTime.toMillis() % SECONDS_IN_MILLIS) * MILLIS_IN_NANOS,
export const defaultTime: Timestamp = {
seconds: Long.fromValue(Math.floor(defaultDateTime.toSeconds()), true),
nanos: (defaultDateTime.toMillis() % SECONDS_IN_MILLIS) * 1000,

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 1

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 01cfc7e and 2622bc8.
Files selected for processing (1)
  • indexer/services/vulcan/tests/lib/on-message.test.ts (2 hunks)

@@ -64,7 +64,7 @@ describe('onMessage', () => {
await onMessage(message);

expect(handler).toHaveBeenCalledTimes(1);
expect(handleUpdateMock).toHaveBeenCalledWith(update);
expect(handleUpdateMock).toHaveBeenCalledWith(update, message.headers ?? {});
Copy link
Contributor

@coderabbitai coderabbitai bot Mar 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The update to include message.headers ?? {} in the test assertion is a good practice, ensuring that the function is tested with the expected headers or a fallback to an empty object if headers are undefined. This aligns well with the PR objectives.

However, consider adding a specific test case to explicitly cover scenarios where message.headers is undefined, to ensure robustness.

Would you like assistance in crafting such a test case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no it is not the ?? operator sucks and promotes bad coding practices

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonfung-dydx, I understand your concerns regarding the nullish coalescing operator and its impact on coding practices. It's important to align with coding standards that best suit the project's needs and preferences.

As an alternative to using the ?? operator, you could explicitly check for undefined headers and provide a default value in such cases. This approach can maintain code clarity and robustness without relying on the ?? operator. Here's a modified version of the assertion:

expect(handleUpdateMock).toHaveBeenCalledWith(update, message.headers ? message.headers : {});

Or, if the project's style guide permits, using a default parameter in the function definition could also serve the purpose, ensuring headers always have a default value when not provided.

I hope this suggestion aligns better with your coding practices. If there's anything more I can help with, please let me know.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 2622bc8 and a755100.
Files selected for processing (1)
  • indexer/services/vulcan/tests/lib/on-message.test.ts (1 hunks)
Files skipped from review as they are similar to previous changes (1)
  • indexer/services/vulcan/tests/lib/on-message.test.ts

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between a755100 and d590723.
Files selected for processing (5)
  • indexer/services/vulcan/tests/handlers/order-place-handler.test.ts (3 hunks)
  • indexer/services/vulcan/tests/handlers/order-remove-handler.test.ts (24 hunks)
  • indexer/services/vulcan/tests/helpers/constants.ts (1 hunks)
  • indexer/services/vulcan/tests/helpers/helpers.ts (3 hunks)
  • indexer/services/vulcan/tests/helpers/websocket-helpers.ts (1 hunks)
Path instructions used (3)
indexer/services/vulcan/__tests__/helpers/websocket-helpers.ts (1)

/
Do not include a walkthrough. For all comments on line changes that are longer than 15 lines, add a prefix to the comment of what lines you are commenting on (for example: "The changes from lines 100 to 200...". Add your comment to be on the last 15 lines of the chunk rather than adding your comment to the entire chunk.

indexer/services/vulcan/__tests__/helpers/helpers.ts (1)

/
Do not include a walkthrough. For all comments on line changes that are longer than 15 lines, add a prefix to the comment of what lines you are commenting on (for example: "The changes from lines 100 to 200...". Add your comment to be on the last 15 lines of the chunk rather than adding your comment to the entire chunk.

indexer/services/vulcan/__tests__/handlers/order-place-handler.test.ts (1)

/
Do not include a walkthrough. For all comments on line changes that are longer than 15 lines, add a prefix to the comment of what lines you are commenting on (for example: "The changes from lines 100 to 200...". Add your comment to be on the last 15 lines of the chunk rather than adding your comment to the entire chunk.

Files skipped from review as they are similar to previous changes (2)
  • indexer/services/vulcan/tests/handlers/order-remove-handler.test.ts
  • indexer/services/vulcan/tests/helpers/constants.ts
Additional comments (4)
indexer/services/vulcan/__tests__/helpers/websocket-helpers.ts (1)
  • 8-18: The addition of expectedHeaders parameter and its validation against the headers in the ProducerRecord is correctly implemented and aligns with the PR's objective to enhance end-to-end latency tracking. Good job on ensuring thorough testing of the new functionality.
indexer/services/vulcan/__tests__/helpers/helpers.ts (2)
  • 27-27: The assignment of defaultKafkaHeaders to message.headers in handleInitialOrderPlace is correctly implemented, ensuring consistent headers for Kafka messages in tests. This aligns well with the PR's objective to enhance latency tracking.
  • 41-41: The assignment of defaultKafkaHeaders to message.headers in handleOrderUpdate is correctly implemented, ensuring consistent headers for Kafka messages in tests. This consistency is crucial for accurately testing the new latency tracking functionality.
indexer/services/vulcan/__tests__/handlers/order-place-handler.test.ts (1)
  • 193-204: > 📝 NOTE

This review was outside the diff hunks and was mapped to the diff hunk with the greatest overlap. Original lines [66-200]

The import and usage of defaultKafkaHeaders to set headers for Kafka messages in test setups are correctly implemented. This ensures consistency in message handling and aligns well with the PR's objective to enhance latency tracking. Good job on ensuring thorough testing of the new functionality.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between d590723 and 3bd29ce.
Files selected for processing (3)
  • indexer/services/ender/tests/helpers/indexer-proto-helpers.ts (2 hunks)
  • indexer/services/vulcan/tests/handlers/order-place-handler.test.ts (3 hunks)
  • indexer/services/vulcan/src/handlers/order-place-handler.ts (7 hunks)
Files skipped from review as they are similar to previous changes (3)
  • indexer/services/ender/tests/helpers/indexer-proto-helpers.ts
  • indexer/services/vulcan/tests/handlers/order-place-handler.test.ts
  • indexer/services/vulcan/src/handlers/order-place-handler.ts

@jonfung-dydx jonfung-dydx merged commit 60b94df into main Mar 28, 2024
18 checks passed
@jonfung-dydx jonfung-dydx deleted the jonfung/threadMessageTimingThroughIndexer branch March 28, 2024 20:32
jonfung-dydx added a commit that referenced this pull request Mar 29, 2024
jonfung-dydx added a commit that referenced this pull request Apr 1, 2024
jonfung-dydx added a commit that referenced this pull request Apr 2, 2024
* fwd through message times

* use the var i made

* post processing stat emission

* post-forwarding timestamp

* pass through event type from vulcan

* event type to stat emissions

* test fix function calls

* WIP WIP WIP

* fix tests

* unused import

* test that kafka messages are threaded
dydxwill added a commit that referenced this pull request Apr 3, 2024
* Add subaccountNumber to PerpetualPositionResponseObject (#1274)

Signed-off-by: Shrenuj Bansal <[email protected]>

* [CT-712] send updates for both normal order matches and liquidation (#1280)

* Remove volatile market (#1263)

* [SKI-21] Bump slinky version to v0.3.1 (#1275)

* Bump slinky version

* Add cp number method

* clean up socks logging (#1285)

* [CT-681] fix liquidated side and offsetting side for indexer delevera… (#1284)

* [CT-681] fix liquidated side and offsetting side for indexer deleveraging events

* fix test

* fix test

* fix lint

* [TRA-105] Add API for parent subaccount perpetual positions (#1282)


Signed-off-by: Shrenuj Bansal <[email protected]>

* Use sample rate with stream destroyed stats. (#1294)

* Revert "[CT-708] Indexer track e2e latency (#1237)" (#1292)

This reverts commit 60b94df.

* Fix swagger generation makefile command / regen swagger docs (#1299)

* pull dydx fork to generate swagger properly

* remove the print

* remove vault constants (#1293)

* Remove custom ping message from socks (#1301)

* Add subaccountNumber to the OrderResponseObject (#1296)

Signed-off-by: Shrenuj Bansal <[email protected]>

* sample more metrics (#1304)

* [OTE-256] Add upgrade handler to initialize OI during upgrade handler (#1302)

* Add upgrade handler to initialize OI during upgrade handler

* nits

* Fix lib.ErrorLogWithError: properly pass in args (#1306)

* fix broken tests (#1312)

* Explicitly close websockets on errors (#1290)

* Increase the number of allowed connections to 8000 (#1317)

* [TRA-104] Add parentSubaccountNumber API for orders (#1313)


Signed-off-by: Shrenuj Bansal <[email protected]>

* Improve Slinky logs to prevent unnecessary logs (#1289)

* [SKI-26]: Prevent funding index update with no oracle prices from (#1321)

halting indexer

* Skip equity tier limit check in PlaceShortTermOrder (#1318)

* Skip equity tier limit check in PlaceShortTermOrder

* remove tests

* Add comment

* fix lint (#1323)

---------

Signed-off-by: Shrenuj Bansal <[email protected]>
Co-authored-by: shrenujb <[email protected]>
Co-authored-by: jayy04 <[email protected]>
Co-authored-by: Eric Warehime <[email protected]>
Co-authored-by: vincentwschau <[email protected]>
Co-authored-by: Jonathan Fung <[email protected]>
Co-authored-by: Tian <[email protected]>
Co-authored-by: Teddy Ding <[email protected]>
Co-authored-by: roy-dydx <[email protected]>
Co-authored-by: Christopher-Li <[email protected]>
jonfung-dydx added a commit that referenced this pull request Apr 4, 2024
* [CT-708] Indexer track e2e latency (#1237)

* fwd through message times

* use the var i made

* post processing stat emission

* post-forwarding timestamp

* pass through event type from vulcan

* event type to stat emissions

* test fix function calls

* WIP WIP WIP

* fix tests

* unused import

* test that kafka messages are threaded

* pass through message headers verbatim

* test logs for on message

* short term order event types
mergify bot pushed a commit that referenced this pull request May 21, 2024
* [CT-708] Indexer track e2e latency (#1237)

* fwd through message times

* use the var i made

* post processing stat emission

* post-forwarding timestamp

* pass through event type from vulcan

* event type to stat emissions

* test fix function calls

* WIP WIP WIP

* fix tests

* unused import

* test that kafka messages are threaded

* pass through message headers verbatim

* test logs for on message

* short term order event types

(cherry picked from commit 4daa11d)

# Conflicts:
#	indexer/services/socks/src/lib/message-forwarder.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging this pull request may close these issues.

2 participants