Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrading version 0.15.1 → 0.16.0 cause "Runtime Error: Could not find profile named 'elementary" #1702

Open
phalanx-hk opened this issue Sep 11, 2024 · 10 comments
Open

Upgrading version 0.15.1 → 0.16.0 cause "Runtime Error: Could not find profile named 'elementary" #1702

phalanx-hk opened this issue Sep 11, 2024 · 10 comments
Assignees
@ofek1weiss
Labels
Bug Something isn't working Triage 👀

Comments

@phalanx-hk
Copy link

Describe the bug
Yesterday I upgrading elementary-data[bigquery] from 0.15.1 to 0.16.0, but Upgrading caused a Runtime Error.
I include below the contents of the files and logs that are considered necessary for troubleshooting.

To Reproduce
Steps to reproduce the behavior:

  1. setting dependencies and relevant yaml file as below
    pyproject.toml
"dbt-core >=1.8,<1.9",
"dbt-bigquery >=1.8,<1.9",
"elementary-data[bigquery] ==0.16.0",

profiles.yml

elementary:
  target: prod
  outputs:
    prod:
      dataset: dbt_elementary
      job_execution_timeout_seconds: 300
      job_retries: 1
      location: asia-northeast1
      method: oauth
      priority: interactive
      project: {project_id}
      threads: 4
      type: bigquery

packages.yml

packages:
  - package: elementary-data/elementary
    version: 0.16.0

dbt_project.yml

models:
  ...
  elementary:
  +schema: "dbt_elementary"
  +docs:
    show: false
flags:
  # Required from dbt 1.8 and above for certain Elementary features
  require_explicit_package_overrides_for_builtin_materializations: false
  1. run command
edr report --env prod --project-name {project_id}  --days-back=30
  1. see error
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-10 23:39:29 — INFO — Running with edr=0.16.0
2024-09-10 23:39:29 — INFO — Installing packages for edr internal dbt package...
2024-09-10 23:39:29 — INFO — Running dbt command --log-format json deps --project-dir {WORKING_DIR}/dbt/.venv/lib/python3.12/site-packages/elementary/monitor/dbt_project
{"data": {"exc": "Runtime Error\n  Could not find profile named 'elementary'"}, "info": {"category": "", "code": "Z002", "extra": {}, "invocation_id": "3ef899d9-d564-4170-a149-06641a6a5aac", "level": "error", "msg": "Encountered an error:\nRuntime Error\n  Could not find profile named 'elementary'", "name": "MainEncounteredError", "pid": 33465, "thread": "MainThread", "ts": "2024-09-10T14:39:31.495433Z"}}
2024-09-10 23:39:32 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Runtime Error
  Could not find profile named 'elementary'

Expected behavior
A report of the dbt project will be generated.

Environment (please complete the following information):

  • Elementary CLI (edr) version: 0.16.0
  • Elementary dbt package version: 0.16.0
  • dbt version you're using: 1.8.6
  • Data warehouse: bigquery
@phalanx-hk phalanx-hk added Bug Something isn't working Triage 👀 labels Sep 11, 2024
@geo909
Copy link
Contributor

geo909 commented Sep 16, 2024 •
edited
Loading

I have the exact same issue after upgrading from 0.15.1 to 0.16.1.

> dbt --version                                                                                                                                                         BI-811-dbt-sources-Fix-freshness-tests-and-alerting [3059a3f4] modified
Core:
  - installed: 1.8.6
  - latest:    1.8.6 - Up to date!

Plugins:
  - bigquery: 1.8.2 - Up to date!

Note that I also upgraded dbt from 1.7.x (dbt-core was 1.7.10 and dbt-bigquery was 1.7.6) to 1.8 at the same time.

@geo909
Copy link
Contributor

geo909 commented Sep 17, 2024 •
edited
Loading

Another tip, in case it helps.

I'm fixing the following:

# requirements.txt
dbt-core==1.8.6
dbt-bigquery==1.8.2

and

# packages.yml
packages
  - package: elementary-data/elementary
      version: 0.16.1
  1. With python package elementary-data==0.16.1 (same problem with 0.16.0):
> edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"                                                                                                                                       main [0e491059] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-17 10:19:10 — INFO — Running with edr=0.16.1
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "c48be6d8-6481-490d-890d-0baec2937684", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 24886, "thread": "MainThread", "ts": "2024-09-17T07:19:11.602498Z"}}
2024-09-17 10:19:12 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
2024-09-17 10:19:14 — INFO — Elementary's database and schema: '"bigquery-xxxx.user_gtzanakis_dbt"'
2024-09-17 10:19:14 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "43151044-9715-4608-b6f1-a1fa68f06067", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 24886, "thread": "MainThread", "ts": "2024-09-17T07:19:15.226170Z"}}
2024-09-17 10:19:15 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 177, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 114, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
  1. With python package elementary-data==0.15.1:
> edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"                                                                                                                                       main [0e491059] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.15.1, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-17 10:24:21 — INFO — Running with edr=0.15.1
2024-09-17 10:24:21 — INFO — edr internal dbt package is not up-to-date, updating it...
2024-09-17 10:24:21 — INFO — Running dbt --log-format json deps --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:24:25 — INFO — Running with dbt=1.8.6
2024-09-17 10:24:25 — INFO — Updating lock file in file path: /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project/package-lock.yml
2024-09-17 10:24:25 — INFO — Installing dbt-labs/dbt_utils
2024-09-17 10:24:25 — INFO — Installed from version 0.8.6
2024-09-17 10:24:25 — INFO — Updated version available: 1.3.0
2024-09-17 10:24:25 — INFO — Installing elementary-data/elementary
2024-09-17 10:24:25 — INFO — Installed from version 0.15.1
2024-09-17 10:24:25 — INFO — Updated version available: 0.16.1
2024-09-17 10:24:25 — INFO — 
2024-09-17 10:24:25 — INFO — Updates available for packages: ['dbt-labs/dbt_utils', 'elementary-data/elementary']                 
Update your versions in packages.yml, then run dbt deps
2024-09-17 10:24:44 — INFO — edr (0.15.1) and Elementary's dbt package (0.15.2) are compatible.
2024-09-17 10:24:49 — INFO — Elementary's database and schema: '"bigquery-xxxx.user_gtzanakis_dbt"'
2024-09-17 10:24:49 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:24:57 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 7, "invocations_per_test": 720}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:04 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:11 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:18 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:25 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_singular_tests", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:31 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 7, "exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:38 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_dbt_models_test_coverage", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:45 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_nodes_depends_on_nodes", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:51 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocation", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:58 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocations_data", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper

I don't know if it makes a difference but mentioning it just in case: I'm using pyenv virtualenv in ubuntu to handle my virtual environments, hopefully that's not the issue here.

@ofek1weiss
Copy link
Contributor

Hey,
@phalanx-hk , is your profiles.yml file located in your project directory or in the user directory?

  • if it is in the project directory, can you try adding --profiles-dir . to your command.

@geo909 from the logs you provided, i assume that in your profiles.yml file, i assume you define the following:

    keyfile: service-account-dbt.json

can you try setting the keyfile path to the absolute path of the file and see if this helps, something like

   keyfile: "/home/<user>/.../service-account-dbt.json

@ofek1weiss ofek1weiss self-assigned this Sep 24, 2024
@geo909
Copy link
Contributor

geo909 commented Sep 24, 2024

@ofek1weiss Still doesn't work:

ferryhopper:
  target: dev_user
  outputs:
    dev_user:
      dataset: user_{{ env_var('DBT_USER') }}_dbt
      job_execution_timeout_seconds: 600
      job_retries: 1
      keyfile: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"
      location: EU
      method: service-account
      priority: interactive
      project: bigquery-xxxxxx
      threads: 2
      type: bigquery

Output:

> edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"                                                                                                                                                                                                                                main [3a3a10e9] modified untracked
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-24 17:49:36 — INFO — Running with edr=0.16.0
2024-09-24 17:49:36 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Path 'ferryhopper' does not exist.
2024-09-24 17:49:36 — ERROR — Failed to parse Elementary's database and schema.
2024-09-24 17:49:36 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-24 17:49:36 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Path 'ferryhopper' does not exist.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Path 'ferryhopper' does not exist.

Tried this one too, just in case:

tzanakis@FHLT7FD7DL3:~/git/bi/dbt (*)
> edr report --project-dir "/home/tzanakis/git/bi/dbt/ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"                                                                                                                                                                                     main [3a3a10e9] untracked
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-24 17:53:23 — INFO — Running with edr=0.16.0
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 226007, "thread": "MainThread", "ts": "2024-09-24T14:53:25.257806Z"}}
2024-09-24 17:53:25 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
2024-09-24 17:53:28 — INFO — Elementary's database and schema: '"bigquery-xxxxxx.user_gtzanakis_dbt"'
2024-09-24 17:53:28 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 226007, "thread": "MainThread", "ts": "2024-09-24T14:53:28.741599Z"}}
2024-09-24 17:53:29 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.

Note that, if I change the virtual environment to switch back to 0.15.1 and change absolutely nothing else, both the above commands run without errors.

@phalanx-hk
Copy link
Author

@ofek1weiss
Thanks for your feedback!
I try adding --profiles-dir ., it works for me.

run command:
edr report --env prod --project-name {project_id} --days-back=30 --profiles-dir .

@ofek1weiss
Copy link
Contributor

@geo909 did you try both at the same time? (setting both the keyfile and the command flags to absolute paths)? or was it one at a time?

the reason i am asking is that in the logs for the second command, you can see the log No such file or directory: 'service-account-dbt.json', i would have expected to see No such file or directory: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"

@geo909
Copy link
Contributor

geo909 commented Sep 25, 2024

@geo909 did you try both at the same time? (setting both the keyfile and the command flags to absolute paths)? or was it one at a time?

the reason i am asking is that in the logs for the second command, you can see the log No such file or directory: 'service-account-dbt.json', i would have expected to see No such file or directory: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"

Oops.. Apologies, that was my mistake. I was trying the full path in the wrong profile.

So, the following combination finally worked!

  • keyfile: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"
  • edr report --project-dir "ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"

Played around with different expressions just in case, but relative paths expressed in different ways do not work, as expected probably. For example, the following:

  • keyfile: "./ferryhopper/service-account-dbt.json"
  • edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"

@ofek1weiss
Copy link
Contributor

@geo909 Firstly, that is great to hear :)
I'll try to figure out what change caused the keyfile issue, and see if it is reversible.
Regarding --profiles-dir, it should work with relative paths, so doing the following should work:

  • keyfile: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"
  • edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"
    can you please try that and let me know if it works/share logs if it does not? 🙏

@geo909
Copy link
Contributor

geo909 commented Sep 25, 2024 •
edited
Loading

@ofek1weiss

No, the last combination does not work. To recap:

Works

  • keyfile: "/home/tzanakis/git/bi/dbt/ferryhopper/service-account-dbt.json"
  • edr report --project-dir "ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"

Output:

> edr report --project-dir "ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"                                                                                                                                                        main [3a3a10e9] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-25 12:04:17 — INFO — Running with edr=0.16.0
2024-09-25 12:04:24 — INFO — edr (0.16.0) and Elementary's dbt package (0.16.1) are compatible.
2024-09-25 12:04:26 — INFO — Elementary's database and schema: '"bigquery-xxxxxx.user_gtzanakis_dbt"'
2024-09-25 12:04:26 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:32 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 7, "invocations_per_test": 720}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:35 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:39 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:42 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:46 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_singular_tests", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:49 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 7, "exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:54 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_dbt_models_test_coverage", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:58 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_nodes_depends_on_nodes", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:05:01 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocation", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:05:04 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocations_data", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper

Does NOT work

  • keyfile: "/home/tzanakis/git/bi/dbt/ferryhopper/service-account-dbt.json"
  • edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"

Output:

> edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"                                                                                                                                                                                main [3a3a10e9] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-25 12:06:18 — INFO — Running with edr=0.16.0
2024-09-25 12:06:18 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Path './ferryhopper' does not exist.
2024-09-25 12:06:18 — ERROR — Failed to parse Elementary's database and schema.
2024-09-25 12:06:18 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ./ferryhopper
2024-09-25 12:06:18 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Path './ferryhopper' does not exist.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Path './ferryhopper' does not exist.

Let me know if you need to me to try anything else!

@ofek1weiss
Copy link
Contributor

ohhhhhhh, you use elementary 0.16.0, in 0.16.1 it should work (it is the only difference between them)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ofek1weiss ofek1weiss

Labels
Bug Something isn't working Triage 👀
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@geo909 @phalanx-hk @ofek1weiss