Return server-side total bytes processed statistics as a header through query frontend

[madhu-reddy-peram]

What this PR does

This pull request adds bytes-processed statistics to the existing Server-Timing HTTP header in the query frontend. It provides information about the total bytes processed by Mimir for a query request that passes through the query frontend. This information will be useful for assessing user behaviour and applying rate limiting when users run expensive, long-range Mimir queries. It will help mitigate slow performance or occasional downtime in Mimir.

Which issue(s) this PR fixes or relates to

None

Checklist

• Tests updated.
• CHANGELOG.md updated

[CLAassistant]

All committers have signed the CLA.

[56quarters]

Thanks for this PR! Instead of adding another header for the query-frontend to send, please add this new value to the existing Server-Timing header with the key bytes_processed.

[madhu-reddy-peram]

Thanks for this PR! Instead of adding another header for the query-frontend to send, please add this new value to the existing Server-Timing header with the key bytes_processed.

@56quarters - Thank you for your feedback!
I chose to implement a new header for the X-Total-Bytes-Processed because the existing Server-Timing header primarily conveys timing information, and its keys are focused on duration. By introducing a dedicated header for bytes processed, clients can more easily access and handle this specific metric without needing to parse the Server-Timing header. I believe this approach enhances clarity and improves the overall user experience. However, I'm open to consolidating it into the existing header if you believe that would be more beneficial. Please confirm.

[56quarters]

Thanks for this PR! Instead of adding another header for the query-frontend to send, please add this new value to the existing Server-Timing header with the key bytes_processed.

@56quarters - Thank you for your feedback! I chose to implement a new header for the X-Total-Bytes-Processed because the existing Server-Timing header primarily conveys timing information, and its keys are focused on duration. By introducing a dedicated header for bytes processed, clients can more easily access and handle this specific metric without needing to parse the Server-Timing header. I believe this approach enhances clarity and improves the overall user experience. However, I'm open to consolidating it into the existing header if you believe that would be more beneficial. Please confirm.

I understand that parsing is simpler with a dedicated header but my preference is to minimize proliferation of custom headers that Mimir uses. I'd rather use the existing Server-Timing header. Thanks!

[madhu-reddy-peram]

Thanks for this PR! Instead of adding another header for the query-frontend to send, please add this new value to the existing Server-Timing header with the key bytes_processed.

@56quarters - Thank you for your feedback! I chose to implement a new header for the X-Total-Bytes-Processed because the existing Server-Timing header primarily conveys timing information, and its keys are focused on duration. By introducing a dedicated header for bytes processed, clients can more easily access and handle this specific metric without needing to parse the Server-Timing header. I believe this approach enhances clarity and improves the overall user experience. However, I'm open to consolidating it into the existing header if you believe that would be more beneficial. Please confirm.

I understand that parsing is simpler with a dedicated header but my preference is to minimize proliferation of custom headers that Mimir uses. I'd rather use the existing Server-Timing header. Thanks!

@56quarters - Thank you for the clarification! I’ve updated the PR to add the bytes processed stats into the existing Server-Timing header as requested. Let me know if there’s anything else you’d like me to address.

[56quarters]

Looks great, thank you!

[56quarters]

Looks like there are a few integration tests that need to be updated to work with the new key.

[madhu-reddy-peram]

Looks like there are a few integration tests that need to be updated to work with the new key.

Looking into these now.

[madhu-reddy-peram]

Looks like there are a few integration tests that need to be updated to work with the new key.

Looking into these now.

I've pushed a commit that should fix the failing tests.