Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor and maintain A22 database tables / indexes #10

Open
clezag opened this issue Feb 21, 2024 · 1 comment
Open

Refactor and maintain A22 database tables / indexes #10

clezag opened this issue Feb 21, 2024 · 1 comment

Comments

@clezag
Copy link
Member

clezag commented Feb 21, 2024

Database has steadily grown, to the point where we now have a 300+ GB traffic table, with multiple indexes of 100+ GB.

CPU load is high, correlating with busy day time and falling during nighttime.

There is already a similar issue open for mobility data: noi-techpark/bdp-core#270

We should try a few things to reign in the exploding database:

  • determine if we can delete old records or shift them to an archive table
  • partitioning
  • rebuild and adjust the already optimized partial index for timestamps > dec 2021
  • Remove the btree stationcode index that overlaps with the partial one
  • try brin indexes for timestamps
  • optimize column data types as most of them are needlessly large
@clezag
Copy link
Member Author

clezag commented Dec 20, 2024

During the last modifications to the a22 traffic connector, the full stationcode index was removed, and the partial index updated to oct 2024, freeing up a few 100GB of space, and lowering CPU load

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant