No such file or directory: '/demo' regardless of input arguments for databricks labs dlt-meta onboard

[brojoe-1]

Hi, when running dlt-meta onboard, I was stuck on issue #51 that is not yet released, so I switched to your branch 51-dlt-meta-cli-onboard-command-issue-for-azure-databrickssdkerrorsplatformresourcealreadyexists

However, there are more issues there - regardless of input arguments for dlt-meta onboard, it tries to read from '/demo' that does not exist. Since I am on a mac, it is not possible for me to move the demo folder to root https://support.apple.com/en-gb/101400

I would assume that the 'dlt-meta onboard cmd' would try to read from folder specified in 'Provide onboarding files local directory', but that is not the case.

See my use of cmd and stacktrace:

sudo databricks labs dlt-meta onboard --debug --profile=test
15:50:54 INFO start pid=94725 version=0.223.1 args="databricks, labs, dlt-meta, onboard, --debug, --profile=test"
15:50:54 INFO Overriding login profile: test pid=94725
15:50:54 DEBUG Loading test profile from /Users/dev/.databrickscfg pid=94725 sdk=true
15:50:54 DEBUG Passing down environment variables: DATABRICKS_HOST, DATABRICKS_TOKEN pid=94725
15:50:54 DEBUG Forwarding subprocess: /Users/dev/.databricks/labs/dlt-meta/state/venv/bin/python3 /Users/dev/dlt-meta/src/cli.py {"command":"onboard","flags":{"log_level":"debug"},"output_type":""} pid=94725
15:50:54 DEBUG starting: /Users/dev/.databricks/labs/dlt-meta/state/venv/bin/python3 /Users/dev/dlt-meta/src/cli.py {"command":"onboard","flags":{"log_level":"debug"},"output_type":""} pid=94725
Provide onboarding file path (default: demo/conf/onboarding.template):
Provide onboarding files local directory (default: demo/):
Provide dbfs path (default: dbfs:/dlt-meta_cli_demo): dbfs:/dlt-meta_cli_demo_100
Provide databricks runtime version (default: 15.3.x-scala2.12):
Run onboarding with unity catalog enabled?
[0] False
[1] True
Enter a number between 0 and 1: 1
Provide unity catalog name: dlt_test
Provide dlt meta schema name (default: dlt_meta_dataflowspecs_f5eda329176c4678bd2cf2c499c270e2):
Provide dlt meta bronze layer schema name (default: dltmeta_bronze_2a2e2b8c465244e5a22743b4d0013eed):
Provide dlt meta silver layer schema name (default: dltmeta_silver_481fbe8078ef4267aec8966d4efb9ec1):
Provide dlt meta layer
[0] bronze
[1] bronze_silver
[2] silver
Enter a number between 0 and 2: 1
Provide bronze dataflow spec table name (default: bronze_dataflowspec):
Provide silver dataflow spec table name (default: silver_dataflowspec):
Overwrite dataflow spec?
[0] False
[1] True
Enter a number between 0 and 1: 1
Provide dataflow spec version (default: v1):
Provide environment name (default: prod): test
Provide import author name (default: dev): test_auth
Provide cloud provider name
[0] aws
[1] azure
[2] gcp
Enter a number between 0 and 2: 1
Update workspace/dbfs paths, unity catalog name, bronze/silver schema names in onboarding file?
[0] False
[1] True
Enter a number between 0 and 1: 1
Traceback (most recent call last):
File "/Users/dev/dlt-meta/src/cli.py", line 503, in
main(*sys.argv[1:])
File "/Users/dev/dlt-meta/src/cli.py", line 499, in main
MAPPINGcommand
File "/Users/dev/dlt-meta/src/cli.py", line 470, in onboard
dltmeta.onboard(cmd)
File "/Users/dev/dlt-meta/src/cli.py", line 160, in onboard
self._ws.dbfs.copy(cmd.onboarding_files_dir_path,
File "/Users/dev/.databricks/labs/dlt-meta/state/venv/lib/python3.9/site-packages/databricks/sdk/mixins/files.py", line 603, in copy
with child.open(read=True) as reader:
File "/Users/dev/.databricks/labs/dlt-meta/state/venv/lib/python3.9/site-packages/databricks/sdk/mixins/files.py", line 351, in open
return self._path.open(mode='wb' if overwrite else 'rb' if read else 'xb')
File "/Users/dev/.pyenv/versions/3.9.6/lib/python3.9/pathlib.py", line 1242, in open
return io.open(self, mode, buffering, encoding, errors, newline,
File "/Users/dev/.pyenv/versions/3.9.6/lib/python3.9/pathlib.py", line 1110, in _opener
return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/demo'
Error: exit status 1
15:52:15 ERROR failed execution pid=94725 exit_code=1 error="exit status 1"

thank you for any help.

[ravi-databricks]

@brojoe-1 , can you please test above error in branch: feature/v.0.0.8.
Here are steps:

 git clone https://github.com/databrickslabs/dlt-meta.git 

cd dlt-meta

 git checkout feature/v0.0.8

python -m venv .venv 

source .venv/bin/activate 

pip install databricks-sdk 

dlt_meta_home=$(pwd)

export PYTHONPATH=$dlt_meta_home

databricks labs dlt-meta onboard --debug --profile=test

[ravi-databricks]

fixed in feature/v0.0.8