Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bucket bytes download metric by filetype #5816

Closed
jlsherrill opened this issue Sep 19, 2024 · 8 comments
Closed

Bucket bytes download metric by filetype #5816

jlsherrill opened this issue Sep 19, 2024 · 8 comments
Labels

Comments

@jlsherrill
Copy link
Contributor

jlsherrill commented Sep 19, 2024

Is your feature request related to a problem? Please describe.
Currently the content type emits download metrics that feature bytes downloaded.

For the rpm plugin, we would like to know what is costing more bandwidth: metadata or rpms.

Describe the solution you'd like

One solution would be to bucket the metrics by filetype (using the extension of the file), so we can query just the rpms:

sum by(domain_name) (rate(pulp_artifacts_size_counter_Bytes_total{filetype="rpm"}[2m]) * 60)

Describe alternatives you've considered

Additional context

@dkliban
Copy link
Member

dkliban commented Sep 24, 2024

@jlsherrill What's the difference between this and #5815

@lubosmj
Copy link
Member

lubosmj commented Sep 24, 2024

This one is a feature request. It would be nice if we could determine which file types are being streamed through content-app and record this inside the metric. The other one is trying to fix/verify the current workflow of the existing metric.

Actually, I had in mind another issue. I think these are duplicates. We should close one of them.

@jlsherrill
Copy link
Contributor Author

This was for the existing metric related to download bytes (how much data was downloaded), #5815 is for a new metric to measure the number of files downloaded.

@dkliban
Copy link
Member

dkliban commented Oct 15, 2024

We will add an additional attribute to the existing metric. The attribute will describe the type of file served: rpm or metadata.

@lubosmj
Copy link
Member

lubosmj commented Oct 15, 2024

Nope. The attribute will describe the file extension.

It is plausible that we will extract this information from the header:

headers["Content-Type"] = content_type
.

@lubosmj
Copy link
Member

lubosmj commented Oct 17, 2024

We decided to record the file extension only when the HTTP response code is equal to 3XX or 2XX.

@lubosmj
Copy link
Member

lubosmj commented Oct 17, 2024

I have just realized that we do not emit the metric in case the caching is disabled... 🎱

@lubosmj
Copy link
Member

lubosmj commented Oct 28, 2024

We have integrated this into our pulp-service plugin. After careful consideration, we plan to remove the metric entirely and replace it by utilizing access logs (similarly to #5815 (comment)). @dkliban noted that it is possible to pass a header value (which is used by the current pulp_artifacts_size_counter_Bytes_total metric) into the logs, which would enable us to aggregate data based on a specific log attribute. Further investigation on our end is required. Therefore, this is not relevant to pulpcore anymore.

@lubosmj lubosmj closed this as completed Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants