Refactor: run DBT from devcontainer

[thekaveman]

Description

After working on #3468 and trying to get things running on a new local environment, I thought it would be nice to encode the dependencies and tools into the devcontainer configuration. This makes it much easier to get DBT running from a fresh environment, and is system/platform agnostic.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

How has this been tested?

See the README updates for steps to test/run this locally.

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

• No action required
• Actions required (specified below)

[angela-tran]

Starting a review of this. Gonna try opening up the devcontainer!

[angela-tran]

The devcontainer was created and opened for me 🎉 I think I'm now at the point of setting up my dbt profile (probably the command in postAttach.sh at line 13), and it seems to be hanging at a part where a browser is supposed to open for me to login:

Snippets of my output:

18:28:24  Running with dbt=1.5.1
18:28:24  Setting up your profile.
schema (usually your name; will be added as a prefix to schemas e.g. <schema>_mart_gtfs): angela
maximum_bytes_billed (the maximum number of bytes allowed per BigQuery query; default is 2 TB) [2000000000000]:
18:34:10  Profile calitp_warehouse written to /home/calitp/.dbt/profiles.yml using project's profile_template.yml and your supplied value
s. Run 'dbt debug' to validate the connection.
18:34:15  Running with dbt=1.5.1
18:34:15  dbt version: 1.5.1
18:34:15  python version: 3.9.20

...

18:34:18  1 check failed:
18:34:18  dbt was unable to connect to the specified database.
The database returned the following error:

  >Database Error
  Runtime Error

    dbt encountered an error while trying to read your profiles.yml file.

    Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/
external/set-up-adc for more information.


Check your database credentials and try again. For more information, visit:
https://docs.getdbt.com/docs/configure-your-profile


Welcome! This command will take you through the configuration of gcloud.

Your current configuration has been set to: [default]

You can skip diagnostics next time by using the following flag:
  gcloud init --skip-diagnostics

Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).

You must sign in to continue. Would you like to sign in (Y/n)?  Y

Your browser has been opened to visit:
...

Nothing happens here. 🤔

[angela-tran]

I tried copying the URL from the output and opening that in a browser manually, and that did show me a Google consent screen for Google Cloud SDK, but then when I finished going through the consent screens, it redirected me to a localhost:8085 URL. I guess this is not too surprising when you look at the parameters of the URL I pasted into my browser.

Screenshot of the localhost redirect

My devcontainer terminal is still waiting for me to log in. 🤔

Screenshot of devcontainer terminal


[thekaveman]

That second screenshot with the links in the console, can you click from there? That's what I had to do, and it worked after I logged in to my Google account on the browser.

[angela-tran]

That second screenshot with the links in the console, can you click from there? That's what I had to do, and it worked after I logged in to my Google account on the browser.

It doesn't let me click the full link, unfortunately. For some reason, it's getting cut off maybe by a line break or something. If it'd be helpful for me to get on a screenshare with you, let me know

[lalver1]

The devcontainer was created and it opened for me too 🙂. I did notice an error when postAttach.sh ran, but it didn't prevent the rest of the script from running 🤔:

[1722 ms] Start: Run in container: /bin/bash .devcontainer/postAttach.sh
pre-commit installed at .git/hooks/pre-commit
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pycqa/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pycqa/isort.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pycqa/bandit.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/home/calitp/.cache/pre-commit/repoz_qq3lxw/py_env-python3/bin/python', '-mpip', 'install', '.')
return code: 1
stdout:
    Processing /home/calitp/.cache/pre-commit/repoz_qq3lxw
      Preparing metadata (setup.py): started
      Preparing metadata (setup.py): finished with status 'done'
    Collecting GitPython>=1.0.1 (from bandit==0.0.0)
      Downloading GitPython-3.1.43-py3-none-any.whl.metadata (13 kB)
    Collecting PyYAML>=5.3.1 (from bandit==0.0.0)
      Using cached PyYAML-6.0.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (2.1 kB)
    Collecting stevedore>=1.20.0 (from bandit==0.0.0)
      Downloading stevedore-5.3.0-py3-none-any.whl.metadata (2.3 kB)
    Collecting rich (from bandit==0.0.0)
      Using cached rich-13.9.3-py3-none-any.whl.metadata (18 kB)
    Collecting gitdb<5,>=4.0.1 (from GitPython>=1.0.1->bandit==0.0.0)
      Downloading gitdb-4.0.11-py3-none-any.whl.metadata (1.2 kB)
    Collecting pbr>=2.0.0 (from stevedore>=1.20.0->bandit==0.0.0)
      Using cached pbr-6.1.0-py2.py3-none-any.whl.metadata (3.4 kB)
    Collecting markdown-it-py>=2.2.0 (from rich->bandit==0.0.0)
      Using cached markdown_it_py-3.0.0-py3-none-any.whl.metadata (6.9 kB)
    Collecting pygments<3.0.0,>=2.13.0 (from rich->bandit==0.0.0)
      Using cached pygments-2.18.0-py3-none-any.whl.metadata (2.5 kB)
    Collecting typing-extensions<5.0,>=4.0.0 (from rich->bandit==0.0.0)
      Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
    Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython>=1.0.1->bandit==0.0.0)
      Downloading smmap-5.0.1-py3-none-any.whl.metadata (4.3 kB)
    Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich->bandit==0.0.0)
      Using cached mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB)
    Downloading GitPython-3.1.43-py3-none-any.whl (207 kB)
    Using cached PyYAML-6.0.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (720 kB)
    Downloading stevedore-5.3.0-py3-none-any.whl (49 kB)
    Using cached rich-13.9.3-py3-none-any.whl (242 kB)
    Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)
    Using cached markdown_it_py-3.0.0-py3-none-any.whl (87 kB)
    Using cached pbr-6.1.0-py2.py3-none-any.whl (108 kB)
    Using cached pygments-2.18.0-py3-none-any.whl (1.2 MB)
    Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)
    Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
    Downloading smmap-5.0.1-py3-none-any.whl (24 kB)
    Building wheels for collected packages: bandit
      Building wheel for bandit (setup.py): started
      Building wheel for bandit (setup.py): finished with status 'done'
      Created wheel for bandit: filename=bandit-0.0.0-py3-none-any.whl size=119913 sha256=4507954499c0c4f40fbbd8c35cbfa4a920ad4494375a71ba501d3f0649f2fe1a
      Stored in directory: /tmp/pip-ephem-wheel-cache-8gp8q082/wheels/6a/4d/55/ae8cdabeab19e9f4ffc7f4dab60e605c1b4201fce4f3a9d5b4
    Successfully built bandit
    Installing collected packages: typing-extensions, smmap, PyYAML, pygments, pbr, mdurl, stevedore, markdown-it-py, gitdb, rich, GitPython, bandit
stderr:
    ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device: '/home/calitp/.cache/pre-commit/repoz_qq3lxw/py_env-python3/lib/python3.9/site-packages/pbr'
Check the log at /home/calitp/.cache/pre-commit/pre-commit.log
14:36:10  Running with dbt=1.5.1
14:36:10  Setting up your profile.

Then I was able to create my profile using my name as schema, use the default maximum_bytes_billed, and sign in with my @compiler.la email after going through the Google Cloud SDK consent screens. I then was able to select Pick cloud project to use.

[thekaveman]

@lalver1 I see in your output:

ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device

This seems strange? Did your Docker VM run out of virtual hard disk?

[lalver1]

Looks like the problem was with my Docker Engine @thekaveman. I cleaned up several dangling images I had and rebuilt the container and postAttach.sh ran fine with no errors 👍