Elastic Search Source #95

parikshitg · 2024-10-10T14:34:37Z

Description

This PR implements the ElasticSearch Source connector. It uses the Elasticsearch search API to pull data from multiple indexes.

Fixes ConduitIO/conduit#1578.

Quick checks:

I have followed the Code Guidelines.
There is no other pull request for the same update/change.
I have written unit tests.
I have made sure that the PR is of reasonable size and can be easily reviewed.

…asticsearch into feature/source

README.md

go.mod

source/config.go

source/position.go

source/worker.go

internal/elasticsearch/v8/search.go

parikshitg · 2024-10-16T10:36:24Z

I am working on using the search_after API as suggested #95 (comment).

…asticsearch into feature/source

parikshitg · 2024-10-17T18:27:43Z

Working on adding safety to the config as mentioned here. conduitio-labs/conduit-connector-redshift#97 (comment)

README.md

internal/elasticsearch/api/search.go

source/worker.go

…asticsearch into feature/source

README.md

internal/elasticsearch/v7/client.go

parikshitg · 2024-11-06T04:21:05Z

Thanks for your review @hariso, I will merge this request and work on other versions in a separate PR.

…asticsearch into feature/source

parikshitg added 27 commits October 1, 2024 09:24

boilerplate

ec62226

no more needed

1be548d

fix: config

5b88778

fix: updated config

13eb083

feat: search api

ce390a2

feat: position

7de7f7b

feat: open source

e978736

feat: updated client search, timeout handled

fdf6846

feat: worker definition

5bf7410

feat: source read

b2bb72c

feat: unmarshal position

adc8acf

feat: source ack

bfe0ca3

feat: source teardown

86b6070

feat: graceful shutdown of workers

1136568

feat: polling

61bc43c

fix: es client search definition

2099181

fix: linters

f0b894b

fix: cleanup

c1c0522

fix: add missing tag in config

a9be98f

fix: small fix in record metadata

d36199e

feat: updated position

5a91d3b

feat: cleanup

862243e

feat: added source in readme

ed485c4

fix: updated modules

e71b907

feat: posititon update and test

3bd6502

fix: linters and cleanup

f19eb7b

Merge branch 'main' of github.com:conduitio-labs/conduit-connector-el…

1b0fc75

…asticsearch into feature/source

hariso reviewed Oct 11, 2024

View reviewed changes

raulb assigned parikshitg Oct 15, 2024

parikshitg added 2 commits October 17, 2024 08:56

fix: some pr comments

ed87a32

feat: search after

4a5502a

parikshitg marked this pull request as ready for review October 17, 2024 18:19

parikshitg requested a review from a team as a code owner October 17, 2024 18:19

Merge branch 'main' of github.com:conduitio-labs/conduit-connector-el…

73a2000

…asticsearch into feature/source

parikshitg added 3 commits October 18, 2024 15:12

fix: config used map[string]type

633523f

fix: pr comment esapi modules updated

0d4d39f

fix: go.sum

0bc3124