Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(api)(frontend): source documents from s3 buckets #41

Merged
merged 8 commits into from
Apr 23, 2024

Conversation

mawandm
Copy link
Contributor

@mawandm mawandm commented Apr 22, 2024

This PR introduces ingesting documents from an s3 bucket.

Testing can be done using localstack by;

  1. Running docker-compose -f compose-dev.yml up
  2. Create a bucket with aws --endpoint-url=http://localhost:4566 s3 mb s3://nesis-test-bucket --region=us-east-1
  3. Upload documents with aws --endpoint-url="http://localhost:4566" s3 sync Documents/BBK/ s3://nesis-test-bucket
  4. Create a session with NESIS_TOKEN=$(curl -s -X POST -H "Content-Type: application/json" -k http://localhost:6000/v1/sessions -d '{"email":"[email protected]","password":"password"}' | jq -r '.token')
  5. Create a datasource with curl -d '{"type": "s3", "name": "boe-cloud-docs", "connection": {"user": "your_username","password": "your_password","endpoint": "http://localhost:4566","dataobjects": "my-test-bucket","region": "us-east-1"}}' -H "Content-Type: application/json" -H "Authorization: Bearer $NESIS_TOKEN" -X POST http://127.0.0.1:6000/v1/datasources
  6. Create a task with curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $NESIS_TOKEN" -k http://localhost:6000/v1/tasks -d '{"type":"ingest_datasource","parent_uuid":"<datasource-id>", "definition":{"datasource":{"id":"<datasource-id>"}}}', where <datasource-id> is the datasource id from 5.
  7. Task created in 6 shoudl start to run

Closes #22

@mawandm mawandm marked this pull request as draft April 22, 2024 13:26
@mawandm mawandm self-assigned this Apr 22, 2024
@mawandm mawandm changed the title feat(api): source documents from s3 buckets feat(api)(frontend): source documents from s3 buckets Apr 22, 2024
@mawandm mawandm marked this pull request as ready for review April 22, 2024 18:55
@mawandm mawandm requested a review from akizito April 22, 2024 18:55
@mawandm mawandm enabled auto-merge (squash) April 22, 2024 18:55
@mawandm mawandm marked this pull request as draft April 22, 2024 18:57
auto-merge was automatically disabled April 22, 2024 18:57

Pull request was converted to draft

@mawandm mawandm marked this pull request as ready for review April 22, 2024 20:03
@mawandm mawandm enabled auto-merge (squash) April 22, 2024 20:43
Copy link
Contributor

@akizito akizito left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good!

@mawandm mawandm merged commit 9407b88 into main Apr 23, 2024
6 checks passed
@mawandm mawandm deleted the 22-add-s3-datasource branch April 23, 2024 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Add s3 datasource
2 participants