-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[REQUEST] Support Neo4j Aura API #2
Comments
I've transferred this issue to the neo4j specific repo. Overall, I like this suggestion and agree it does remove the barrier to entry. Here are some of my initial thoughts...
I think a lot of the nuance in this is here. I do have a few things I think we need to iron out.
Plugins in nodestream are weird. We often talk about plugins like the one you describe because its what builds an "ecosystem" the most, but tons of things are pluggable in nodestream - right down to what file formats are handled. I think notably for this proposal commands are pluggable as well.
I don't think we need to have a completely separate plugin for aura as it related to api connectivity. Since we've moved the neo4j code here, I think this is a reasonable to retain this logic here. But, I do have a possible approach that may resolve some of the concerns I have. Just throwing it out to see what we think. What if we plugged in some commands:
Or maybe...
...etc. Then these commands could take CLI arguments to provision the database how the user wants and wait for it to come up before a user even runs a pipeline. It could also then add a preconfigured target that sets the values correctly. I know the idea is super rough but hopefully its enough to get the concept across? Given these two suggestions, which do you think fits better with the aura product @aaronWass-neo? |
This is a good call out. My original thinking was to not store the credentials of the user database locally. We would just return the username and password for this new aura instance back to the user through the console. At this point they would be responsible to copy these credentials and setup a secure way to pass these back to create nodestream targets for this new database. I really like your idea of having the aura commands in nodestream and auto-generating the target in nodestream.yaml for the newly created Aura instance. It feels potentially unsafe to store the user/pass for this new instance here without the user being aware that we are saving the Aura password locally. One option could be to have an option to store that password or not. Using your suggested commands, you could do
which would create the target in nodestream.yaml with the user & password section blank. The user & password from the new Aura instance would be passed to the user via the console, and they would be responsible for determining if/how they want to add it to the target. or
which would create the target in nodestream.yaml with the user and password filled in
It is a best practice to specify which user database to connect to when connecting to Aura, or any Neo4j database. In Aura the user database is named 'neo4j'. Nodestream already sets this by default, so we don't need to worry about that. There aren't any other Aura specific settings to keep in mind here.
Definitely!
I really like this idea. I like Above I talked about the possibility of having two different options for this. One where we save the new aura instance password locally, and one where we just pass it to the user via the console. Do you like this idea? Other thoughts on the security aspect here? We should be able to use a lot of what has been done in aura-cli for the API interface. Automatically setting up the target for this newly created instance, should further streamline this aura ingestion process. |
Yeah, this is roughly my thinking. One difference was that we could configure |
Yea I think that's a good idea. |
I think we should be consistent with the current command noun/verb pattern so maybe |
Hmm interesting suggestion. There is some precedent though with the new nodestream migrations command like nodestream migrations make. I think my thinking originally was to namespace like concepts so it follows noun verb. This leaves some of the commands as they are today as "shorthand" like "nodestream run" being shorthand for "nodestream pipeline run" but it feels like your thinking about it the opposite way which makes perfect sense in retrospect. I'm not strongly for or against either way but it is obviously important to be consistent. I feel like I can cite examples both ways from other clis. |
Something else here is that the aura api can perform actions on different nouns... For example in aura-cli you can perform actions on tenants, instances or snapshots. So commands look like:
I don't think it is necessary to to have all of this functionality in nodestream, but remapping this might get confusing.
I don't have a strong preference one way or the other. Maybe the first option does flow more naturally. |
I added the first iteration of this on my fork here I went with
Running this with real arguments looks like this (environment variables saved for tenant id, aura client id and aura client secret):
This calls the Aura API and returns information about the newly created Aura instance:
|
I've started a feature branch for this so we can aggregate changes in this space before a release: https://github.com/nodestream-proj/nodestream-plugin-neo4j/tree/aura-integration |
With #20, we're going to land some basic commands support. I think before I am comfortable releasing it, I'd like to see the following:
These features can be added in some additional PRs to the |
Is your feature request related to a problem? Please describe.
It would be nice to be able to create a Neo4j Aura instance as part of a nodestream pipeline. This would further reduce the barrier to entry for newcomers to graph databases. Neo4j Aura (Neo4j's fully managed DBaaS) exposes an API that allows you to run CRUD operations on your Aura instances and tenants that you have access to.
Describe the solution you'd like
My current thinking would be to accept the Aura API parameters when defining a target in nodestream.yaml like this:
Then you could run a pipeline like this:
This would mean adding the logic for the Aura API to nodestream/databases/neo4j
Describe alternatives you've considered
An alternative approach could be to create a separate nodestream plugin that would handle the Aura API logic. My current understanding of nodestream plugins is that they are primarily for defining ingestion and schema modeling like in nodestream-plugin-akamai. I would love any feedback on if creating a separate plugin for this Aura API management would be a superior alternative.
The text was updated successfully, but these errors were encountered: