Make the old and new API available at the same time from openstreetmap.org

[fwsmit]

If we want to make this change smooth there has to be a simple changeover procedure from the perspective of the tools. Phase III of the paper is very briefly described, but it is in my opinion, at least on a technical level, the most important phase. The paper talks about that tools need to be available to convert between data formats. That would be great, but I think it should be done by openstreetmap.org to make the change the least disruptive to data consumers.

What I think should happen is a long overlap period where two API's are available at the same time from openstreetmap.org. Any tools still using the old API shouldn't break. The new API should allow tools to slowly support the new data format, while keeping it behind a toggle. This old API should be available for a few years to allow for tools to slowly make the transition.

I don't know how feasible this is, but it would make it a whole lot less disruptive to tool makers. The behind the scenes data model change can be done at any time, given the old API stays working.

[simonpoole]

If we do away with way nodes I don't believe anything else than a read only "old API" would be tenable*, but as the API is actually "the editing API" providing that would not really serve any practical purpose.

Note that there are naturally parts of the API, for example all user related bits, that would not be impacted.

* "old" nodes would have to be synthesized.

[mmd-osm]

FWIW: in #18 I already talked at length about running two versions in parallel and keeping the old API working.

TL;DR version:
It will start with a read only new API and additional minutely diffs & planet files for the new data model. All translation work between data models happens automatically behind the scenes. There will be almost no changes to API 0.6 editors (we might no longer allow single object update endpoints), while adventurous data consumers can try out the new stuff already.

As part of the migration, the full history needs to be rewritten to match the new data model. This also has to happen behind the scenes without impacting today’s users.

We would need a few TB more disk space to keep both data model variants around at the same time. (Note: this isn’t part of the 2023 OWG budget!)

Everyone will have enough time to prepare for the day when the final switch to a read write new API happens, and the old API will be shut down.

[fwsmit]

From this it seems it's not entirely obvious if losses back-and-forth conversion is possible. One might consider only making the changes that can be losslesly converted.
https://github.com/osmlab/osm-data-model/blob/master/migration_plan.md?plain=1#L29

[mmd-osm]

Yes, exactly. 0.6 has all the relevant information to create 0.7. Once you upload 0.6 data, the update process would have to take care of updating the 0.7 database tables in the same transaction as the 0.6 db table update.

We need to be a bit careful not to create a huge number of new way versions by moving one node at a time. I'm sure there are other interesting special cases that need some more thought. Converting the full history is also quite tricky as I already wrote.

In general though, I believe this should be feasible.

[mmd-osm]

From this it seems it's not entirely obvious if losses back-and-forth conversion is possible.

Although I'm maintaining relevant parts of the API, I wasn't involved in writing this page back from 2018, or the current data model study. Probably the ideas don't match with what I'm talking about in #18. I'm only trying to come up with a proposal that's feasible from an API point of view with the least amount of disruption.

[fwsmit]

Is the conversion still possible after the 0.7 API is writable? In other words, do you need to have a complete V0.6 database for converting the 0.6 data and providing a 0.6 API? If we already know that 0.7 data cannot be converted to 0.6 data, then we should document that and make plans according to that. Because https://github.com/osmlab/osm-data-model/blob/master/migration_plan.md assumes that it's possible to convert the data losslesly (back and forth)

The tricky part is most likely the conversion of the existing data. The good news is that by the very requirement of lossless converters from the first step, this is guaranteed to be correctly solvable.

[mmd-osm]

I haven't really given much thought to backfilling an API 0.6 db once we enable write access to the API 0.7. As @simonpoole noted you would have to synthesize "old" nodes somehow. Because an API 0.7 way doesn't have the original node ids anymore, you need to match nodes by lat/lon values instead. Without a complete 0.6 db, you end up creating duplicate nodes, etc.

I'm sure this would add even more complexity to a fairly complex migration process. My preference would be to avoid this scenario, and turn off API 0.6, once API 0.7 enters write mode. As mentioned, there's no time pressure here.