As a Open Data Hub maintainer I want a quota to limit the amount of historical data that can be requested in a single call

[dulvui]

• meeting the 11th of April to discuss the limit to set
• Migrate our elaborations to use credentials and not been limited by the quota
• set the limit agreed

To prevent big queries that slow down or even bring our Mobility API down a quota that limits requests for more than a give time frame should receive a quota message, like we have for max requests per second.
To be fast, the implementation should be made to first introduce the limit and then implement the logic to allow bigger requests to certain users. So we implement the easy limit fast and the more complicated logic only if its really needed.

TODO:
@sseppi organize a dedicated meeting in the whole team, to discuss if this limits have any benefits or only through backs:

• PRO: Maybe against DoS attacks? @clezag thinks not.
• CONS: Data more difficult to access by students and others who would need the histroical data.

[clezag]

Implementation has already been done.
What's left is to define the actual time ranges and how to move into production (communication to clients etc.)

[clezag]

Before defining any policies, we will first analyze the API logs and figure out which kinds of users are currently making historical data requests, and which ranges they request

[clezag]

Limits have been disabled in test (can be configured by setting corresponding env variables in CI/CD)

[clezag]

Decisions made on 11.04.2024:

We will start with a quota limit of

• 5 days anonymous
• 100 days referer
• 1000 days basic
• open for advanced and premium

According to the lo statistics, there seems to be only one elaboration type 3rd party application that is expected to be impacted immediately by this quota (parking prediction by Chris Mair).

• Coordinate with Chris Mair about the parking prediction elaboration
• make all our internal elaborations use credentials
  • bluetooth traffic (no ninja request - direct sql)
  • env-a22-averages (already using credentials)
  • env-a22-nlc (already using credentials)
  • skyalps-gtfs (only uses station data)
  • parking-free (uses 4 weeks windows - referer implemented)
  • pollution v1 (already uses credentials)
  • pollution v2 (already uses credentials)
  • traffic a22 data quality (already uses credentials - might need to enable quota on them)
• Review data visualizations (webcomponents, databrowser, analytics) to handle the quota limit case properly
• Update documentation / website with new limits
• Notify customers about the change

[clezag]

Proposing production date: 11.09.
Do we need to notify customers beforehand?

[sseppi]

In september we foresee to send out the next Community Update mail, what about communicating the change in the community email and put everything into production just after the communication?

[clezag]

That's fine for me. I just needed a date to communicate to Chris, so we have an agreed deadline. We can postpone the actual release. Do you have a concrete date?

[sseppi]

@clezag we foresee to send out the community update the last week of september, probably the 24th or the 25th.

[rcavaliere]

@sseppi reminder for you: please notify Emily to include this point in the next community update.

[sseppi]

@rcavaliere Is alredy included in the text Emily presented at the last meeting.