SQL custom query result set for source systems

[cavanaug]

Tell us about the problem you're trying to solve

Today we have some teams that use NiFi and while we are working to transition them to AirByte we have come across a use case that isnt supported. Specifically we have a few use cases where extraction from a foreign sql system is predicated on fully extracting data from a sql query. In essence a sql query is performed on the source system and the result set is desired to be populated into a target system. This is similar to the airbyte use case of replicating from a source system view, but in this situation the source system may not grant us the permissions to create/use a view.

Describe the solution you’d like

Im not sure if the solution is something where each of the source sql connector to allow for an option to pass a query or if a new sql connector should be created for this use case.

Ideally it would great if you could perform both an incremental load or a full load with this type of solution.

Describe the alternative you’ve considered or used

Today the teams are using NiFi with a sql connection to obtain a result set to push into a target sql system.

Are you willing to submit a PR?

We might be if we were given the recommended path for implementation.