[Buildings] Automated download + changelog usage?

[JimShady]

Category Feedback

Hello,

Hoping for some advice please. We want to keep a copy of the Buildings dataset in our Data Lake. I have successfully managed to download the data as geoparquet and can write it into tables. I now have a couple of questions:

1. Is there some programmatical way that we can check for new releases and get the version if so? I'm thinking perhaps an API I can hit once a week to see if there is a new version or something? We'd use that to trigger the download pipeline.

2. Will we, each month, need to totally overwrite our data with the new release? Or can we somehow use the changelog and the ids of rows to selectively update our local copy with changes?

Appreciate any ideas or tips.

Thanks.

[RaczeQ]

Regarding automatic release checking, you can utilize this script: https://github.com/OvertureMaps/data/blob/main/utils/fetch-releases-from-s3.py and run it via cron each day (it is also run as a github workflow in this repository and uploaded to cloud - https://github.com/OvertureMaps/data/actions/runs/11136297775)
In theory, you can access it every day here: https://labs.overturemaps.org/data/releases.json

[JimShady]

Perfect, thanks.