Add SQS (and other cloud queue support) for Pipe

[Samrose-Ahmed]

Feature request

Load files directly from cloud object storage queue notifications like Amazon SQS + S3 object notifications (and eventually GCP/Azure/other cloud equivalents).

Is your feature request related to a problem? Please describe.

Load data from upstream sources that use object notifications to queue, quite common nowadays.

Describe the solution you'd like

Consume notifications from the queue and do pipe job as usual instead of listing files.

Describe alternatives you've considered

Currently list files is used but that is inefficient.

Additional context

E.g. for ref Snow pipe supports this.

[murphyatwork]

hey @Samrose-Ahmed, thanks for advice, this enhancement is on the roadmap, but not scheduled right now. if you're willing to contribute we can have a discussion

[Samrose-Ahmed]

Yes, i plan to work on this after a bit, will sync.

[Samrose-Ahmed]

Do you think it makes sense to just poll SQS on FE or need CN? A SQS Poll isnt very heavy. I think current FS list is done via broker.