Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

api, pipeline: support compressed measurements #561

Open
bassosimone opened this issue Feb 8, 2022 · 1 comment
Open

api, pipeline: support compressed measurements #561

bassosimone opened this issue Feb 8, 2022 · 1 comment
Assignees
Labels
enhancement New feature request or improvement to existing functionality ooni/api Issues related to https://github.com/ooni/api ooni/pipeline Issues related to https://github.com/ooni/pipeline optimization priority/medium Normal priority issue

Comments

@bassosimone
Copy link
Contributor

bassosimone commented Feb 8, 2022

This issue is about accepting compressed measurements. We'll close this issue when we've implemented accepting compressed measurements on the backend side.

This issue is a follow up to ooni/probe#1802.

We discussed this matter in today's backend meeting with @FedericoCeratto and @hellais. Here are notes from the meeting regarding what we think we'll need to do to make this happen:

  1. the probe needs to set a header to indicate the measurement is compressed;
  2. it seems python's gzip implementation could use a context manager to limit the used memory;
  3. the ooni/api does not touch the incoming measurement in any way and just stores it to disk where the upload and the fastpath will pick it up and, respectively, save it, process it and insert it into the fastpath table;
  4. so the ooni/api can just receive the compressed measurement and put it into a can;
  5. we need to use the filename as metadata to tell the rest of the backend the postcan is compressed;
  6. the fastpath needs to know how to handle compressed measurements.

See also ooni/probe#2014.

@bassosimone bassosimone added enhancement New feature request or improvement to existing functionality ooni/api Issues related to https://github.com/ooni/api priority/high Important issue that needs attention soon ooni/pipeline Issues related to https://github.com/ooni/pipeline optimization labels Feb 8, 2022
@FedericoCeratto FedericoCeratto added priority/medium Normal priority issue and removed priority/high Important issue that needs attention soon labels Nov 7, 2022
@hellais hellais moved this to Backlog in Roadmap Jan 8, 2025
@hellais hellais assigned hynnot and unassigned FedericoCeratto Jan 28, 2025
@hellais
Copy link
Member

hellais commented Jan 28, 2025

Add this as a child issue of: #931. We can maybe add support for this as part of that.

@hellais hellais removed this from Roadmap Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature request or improvement to existing functionality ooni/api Issues related to https://github.com/ooni/api ooni/pipeline Issues related to https://github.com/ooni/pipeline optimization priority/medium Normal priority issue
Projects
None yet
Development

No branches or pull requests

4 participants