[Feature] Allow args flag to be passed to run-operation

[aranke]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

run-operation should support the args flag so we can run operations like so:

dbt run-operation foo --args 'v: {{ some_macro() }}'

Describe alternatives you've considered

N/A

Who will this benefit?

Anyone who wants to dynamically run different branches in a macro

{% macro foo(suffix_type=None) %}
{% if suffix_type = 'standard' %}
  {{ run_started_at.strftime('%d-%m-%Y') }}
{% elif suffix_type = 'alternate' %}
  {{ run_started_at.strftime('%Y%m%d') }}
{% endif %}
{% endmacro %}

dbt run-operation foo --args 'suffix_type: standard'
dbt run-operation foo --args 'suffix_type: alternate'

Are you interested in contributing this feature?

No response

Anything else?

No response

[dbeatty10]

Wanna give this a try?

macros/cowsay.sql

{% macro cowsay(message="hi") %}

{% do log(" ________________________________________", info=True) %}
{% do log("   " ~ message, info=True) %}
{% do log(" ----------------------------------------", info=True) %}
{% do log("        \   ^__^", info=True) %}
{% do log("         \  (oo)\_______", info=True) %}
{% do log("            (__)\       )", info=True) %}
{% do log("                ||----w |", info=True) %}
{% do log("                ||     ||", info=True) %}

{% endmacro %}

dbt run-operation cowsay --args '{message: "Arg, I be a pirate"}'

16:08:08   ________________________________________
16:08:08     Arg, I be a pirate
16:08:08   ----------------------------------------
16:08:08          \   ^__^
16:08:08           \  (oo)\_______
16:08:08              (__)\       )
16:08:08                  ||----w |
16:08:08                  ||     ||

[dbeatty10]

Since it looks like we already support this, I'm going to close this as "not planned". But just let me know if I overlooked something and we can re-open it.

[dbeatty10]

@aranke I overlooked one key piece in your original message 😅

👉 the part where {{ some_macro() }} is within the args you are passing.

I'm interpreting that the desired behavior is for the --vars argument to accept Jinja that is rendered as a pre-step and passed to the macro after it is rendered. (I recognize that it could be interpreted the exact opposite, so please let me know if so.)

While we don't plan on adopting that behavior within run-operation, there are a couple ways to achieve this today:

1. Push the Jinja logic into the macro (recommended)
2. Use interactive compilation to render your values as a pre-step

Option 1 - push the Jinja logic into the macro

Suppose you have a cowsay macro like described in #9862 (comment).

We'd recommend refactoring the cowsay macro so that it does not need --args to contain any Jinja. That way, it can be called like this:

dbt run-operation cowsay
dbt run-operation cowsay --args "{suffix_type: 'standard'}"
dbt run-operation cowsay --args "{suffix_type: 'alternate'}"

Toggle for an example.

{% macro cowsay(message="hi", suffix_type=none) %}

{% set timestamp = get_timestamp_suffix(suffix_type) %}

{% do log(" ________________________________________", info=True) %}
{% do log("   " ~ timestamp ~ " " ~ message          , info=True) %}
{% do log(" ----------------------------------------", info=True) %}
{% do log("        \   ^__^"                         , info=True) %}
{% do log("         \  (oo)\_______"                 , info=True) %}
{% do log("            (__)\       )"                , info=True) %}
{% do log("                ||----w |"                , info=True) %}
{% do log("                ||     ||"                , info=True) %}

{% endmacro %}


{% macro get_timestamp_suffix(suffix_type=none) %}
    {% if suffix_type == 'standard' %}
        {% do return(run_started_at.strftime('%d-%m-%Y')) %}
    {% elif suffix_type == 'alternate' %}
        {% do return(run_started_at.strftime('%Y%m%d')) %}
    {% else %}
        {% do return("") %}
    {% endif %}
{% endmacro %}

Option 2 - use interactive compilation

The following is a more complicated option. I'm not sure if there are valid use-cases for it or not, but providing it for the sake of completeness.

The general pattern is this:

1. Use interactive compilation to render your Jinja
2. Store the result in an environment variable
3. Pass the value of the environment variable as an argument

It is a multi-step process that is dependent upon the capabilities of the shell it is operating within (bash, zsh, fsh, PowerShell, etc.), so your mileage may vary.

Here's an example that works for me in zsh (and should work with bash as well) as long as jq is installed:

export message=$(dbt --log-format json compile --inline "{{ get_timestamp_suffix(suffix_type='standard') }}" --output text | jq -r 'select(.data.compiled != null) | .data.compiled')
dbt run-operation cowsay --args "{message: '$message'}"
unset message

If #9840 is implemented, then the command in the first step would be a bit more simple:

export message=$(dbt compile --inline "{{ get_timestamp_suffix(suffix_type='standard') }}" --quiet)
dbt run-operation cowsay --args "{message: '$message'}"
unset message