feat: Preset.io source (Superset SaaS) #8188
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Deployment failed with the following error: |
github-actions
bot
added
the
ingestion
betodealmeida
force-pushed
the
preset_source
branch
from
0769851 to
15fe267
Compare
anshbansal
added
the
community-contribution
|
@mayurinehate could you take a look at this PR? |
mayurinehate
reviewed
Jul 6, 2023
There was a problem hiding this comment.
Hi @betodealmeida , thanks for the PR.
I'd like to share this guide on adding a metadata ingestion source - https://datahubproject.io/docs/metadata-ingestion/adding-source/ that'll help you through the process. Especially please refer steps 4,5 (making the new plugin discoverable) and 9 (adding new platform).
Also, I have a couple of initial comments on the PR -
- Let's move new source to separate file preset.py. (you said it !)
- For lint, I believe you'd need to inherit PresetConfig from SupersetConfig. If Preset Config does not need username/password - you can refractor common parts from existing SupersetConfig to BaseSupersetConfig and make SupersetConfig and PresetConfig inherit BaseSupersetConfig.
Lastly, excited to hear results from your end-to-end preset testing when you have them.
| self.config = config | ||
| self.report = StaleEntityRemovalSourceReport() | ||
|
|
||
| login_response = requests.post( |
There was a problem hiding this comment.
Can we refractor get token piece into separate get_access_token method in SupersetSource and only override create and get_access_token method in PresetSource ?
|
|
||
| pipeline.run() | ||
| pipeline.raise_from_status() | ||
| golden_file = "golden_test_ingest.json" |
There was a problem hiding this comment.
You'll need separate preset golden file as it will have platform preset in entity urns.
There was a problem hiding this comment.
This can help you generate golden file from scratch - https://datahubproject.io/docs/metadata-ingestion/developing/#updating-golden-test-files
Let me know if you need help testing end-to-end, I can get you a Preset workspace to run tests. Thank you! |
|
Thanks for the review, @mayurinehate! I'll address the comments in the next couple days! |
|
Hello @betodealmeida, would you have any idea when you might have time to work on it? thanks a lot |
mayurinehate
added
the
pending-submitter-response
|
@betodealmeida @mayurinehate, @hsheth2 is there still work to do on this? We just ran up against this requirement and would be willing to do the last mile work to get this in. |
|
I think i have a rebased version of this more or less sussed out. I'm hitting a snag testing it out in the local env. the ingest job fails to pull install the preset subpackage becuase its calling to the acryl pip package. It feels like I missed a step or im operating off of old docs. I can push that PR once I get the functionality ready. I've incorporated the suggestions on this PR. |
|
The class creation more or less mirrors this PR. I did also add the logo and all the registration stuff in the typescript files. I am using the I'm guessing there's maybe an env var or something im supposed to set that overrides this? subsequent runs faile with: |
|
@byndcivilization when developing on sources, you'll need to use ingestion from the CLI e.g. |
|
@coderabbitai full review |
shirshanka
added
the
on-deck
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Preset.io is a commercial SaaS provider for Apache Superset. This PR introduces a new ingestion source for Preset "workspaces", which are essentially Superset instances. The only difference is the authentication mechanism that needs to be done against an API gateway using a key/secret pair.
I implemented the source as a derived class of
SupersetSourcein the same file; let me know if it would be better to have it in a separate file.I added a new integration test, but I haven't tested it end-to-end yet. I'm in the process of setting up a local development to test it with a real workspace. But I thought it would be good to get early feedback in the PR since this is my first time contributing to DataHub.
Checklist