Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Environment userdev doesn't work for team members wanting to do dbt development #579

Open
aaronsteers opened this issue Feb 27, 2023 · 5 comments · May be fixed by #580
Open

Environment userdev doesn't work for team members wanting to do dbt development #579

aaronsteers opened this issue Feb 27, 2023 · 5 comments · May be fixed by #580

Comments

@aaronsteers
Copy link
Contributor

I tried to user userdev to just run meltano run dbt-snowflake:run and there were tons of issue that blocked me from:

  1. Testing existing dbt transformations.
  2. Adding additional dbt tranforms.

I'm opening a PR with fixes for the blockers I ran into...

@aaronsteers
Copy link
Contributor Author

aaronsteers commented Feb 27, 2023

The transform in cloud_ip_ranges.sql fails. I'm not sure yet why.

@aaronsteers aaronsteers linked a pull request Feb 27, 2023 that will close this issue
@aaronsteers
Copy link
Contributor Author

aaronsteers commented Feb 27, 2023

I've got most things working now in #580, but many build steps are slow and some haven't been able to complete yet.

I wanted to get the full project built before adding new transforms, but ran out of time. Might be worth a 1:1 @pnadolny13 if we can find time, or else I could bring to data office hours.

@pnadolny13
Copy link
Contributor

@aaronsteers I commented in the PR but can you share more of the exact issues you were running into? Its totally possible that things arent working as expected because I've mostly been the only one using this repo so far so I havent tested the onboarding too much.

@aaronsteers
Copy link
Contributor Author

aaronsteers commented Mar 1, 2023

Thanks, @pnadolny13. I think the PR piece-by-piece resolved my issues. Maybe one last thing that would be helpful would be a section under CONTRIBUTING.md that highlights how to run only a subset of the transforms - or conversely, how to exclude the most expensive ones.

I know there are a number of commands registered with different --select filters - I didn't see an obvious path for how to use the right one (or create a new one) for my use case, but admittedly this might also be a reflection of my stale dbt prowess. 😄

@pnadolny13
Copy link
Contributor

@aaronsteers yeah I can write up a better contributing section. It might end up being helpful to use tags for some of the larger tables so they can easily be excluded.

I know there are a number of commands registered with different --select filters - I didn't see an obvious path for how to use the right one (or create a new one) for my use case, but admittedly this might also be a reflection of my stale dbt prowess. 😄

I can add this to do docs but...when I run locally I usually use invoke and pass select filters as usual. I only create custom meltano command if I want to schedule a job, at that point I create a command and use it in a job/schedule.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants