Feature: Upgrade plugin in an existing pipeline

[lovromazgon]

Feature description

This could almost be categorized as a bug. Currently, Conduit is very strict about the changes that are allowed to be made on a pipeline, for instance, it is not allowed to update the connector plugin reference. The REST and gRPC APIs don't support updating the field at all, while in a pipeline configuration file a change of the plugin causes Conduit to delete the existing connector and create a new one with the new plugin. This results in the existing position being lost, so the connector will start from scratch and not resume from where it left off.

We initially implemented it this way, to prevent users from changing the plugin (e.g. from kafka to postgres) and then starting the pipeline with the new plugin using the state (position) from the old plugin. Because of that, we now don't have a clear path to going from one pinned version of the connector to another (e.g. from [email protected] to [email protected]) or from one plugin type to another (e.g. from builtin:kafka to standalone:kafka).

That said, we should also consider the use case of actually changing the entire plugin. Imagine having a community fork of the Postgres connector, let's call it postgres-community. Somebody running an existing pipeline using the postgres connector should ideally be able to switch to the postgres-community connector without losing the state (assuming the fork still stores the position in the same format).

If we allow users to update the plugin, it would probably be good to log a warning that the pipeline may not work correctly if the new connector is not compatible with the previous one. This would hopefully inform the user about what's going on in case the new connector actually fails to start. This could happen when the new connector contains a breaking change or the user moved between two unrelated connectors (e.g. from kafka to postgres).

Note

Note that everything written above about connectors applies to processors as well.