fix: generate the bundle path including prefix/suffix if available #20
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
currently BRICKFLOW_WORKFLOW_SUFFIX does not work as expected.
if user deploys the workflow using suffix e.g. _suffix10
BRICKFLOW_WORKFLOW_SUFFIX=_suffix10 poetry run brickflow deploy --deploy-mode bundle --workflows-dir src/workflows -e local
then bundle root path would be - /Users/${workspace.current_user.userName}/.brickflow_bundles/test-project/local
if user deploys the same workflow using another suffix e.g. _suffix20
BRICKFLOW_WORKFLOW_SUFFIX=_suffix poetry run brickflow deploy --deploy-mode bundle --workflows-dir src/workflows -e local
then it will replace the earlier workflow because bundle root path will be generated again to be - /Users/${workspace.current_user.userName}/.brickflow_bundles/test-project/local
This fix will create dedicated bundle root path for each suffixes.
e.g _suffix10 -> /Users/${workspace.current_user.userName}/.brickflow_bundles/test-project/local/_suffix10
e.g _suffix20 -> /Users/${workspace.current_user.userName}/.brickflow_bundles/test-project/local/_suffix20
This way two different workflows will be deployed on databricks with their own lifecycle states.
Note: same concept applied for BRICKFLOW_WORKFLOW_PREFIX as well.