Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connector: S3 java connector with Apache Iceberg format[Destination] #1237

Closed
1 task done
simonl2002 opened this issue Oct 11, 2023 · 0 comments
Closed
1 task done
Assignees
Labels
connector:destination Request for a destination connector

Comments

@simonl2002
Copy link
Member

simonl2002 commented Oct 11, 2023

Resource name

Apache Iceberg

Resource link

https://iceberg.apache.org/

Connector type

Destination

Expected connector behavior

Apache Iceberg isn't a datastore. It is a format for writing big analytics tables.
To achieve this, we'll write an S3 java conduit connector, and add the Apache iceberg format to that.

The official implementation of the format is in Java (see link below). There is a go implementation (https://github.com/apache/iceberg-go) but the first commit was 2023-08-23, so it is very very immature, So we'll go with Java.

Resources:
https://github.com/apache/iceberg

Steps:

@simonl2002 simonl2002 added the triage Needs to be triaged label Oct 11, 2023
@simonl2002 simonl2002 moved this from Triage to Todo in Conduit Main Oct 11, 2023
@simonl2002 simonl2002 added connector:destination Request for a destination connector and removed triage Needs to be triaged labels Oct 11, 2023
@maha-hajja maha-hajja self-assigned this Oct 12, 2023
@simonl2002 simonl2002 moved this from Todo to In Progress in Conduit Main Oct 20, 2023
@maha-hajja maha-hajja changed the title Connector: Apache Iceberg [Destination] Connector: S3 java connector with Apache Iceberg format[Destination] Oct 25, 2023
@maha-hajja maha-hajja moved this from In Progress to Done in Conduit Main Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
connector:destination Request for a destination connector
Projects
Archived in project
Development

No branches or pull requests

2 participants