Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add batch property to model jinja fnction #6590

Merged
merged 11 commits into from
Dec 5, 2024
46 changes: 44 additions & 2 deletions website/docs/reference/dbt-jinja-functions/model.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@

<Tabs>

<TabItem value="cli" label="CLI">
<TabItem value="cli" label="Command line interface">

If you're using the CLI, use [log()](/reference/dbt-jinja-functions/log) to print the full contents:
If you're using the Command line interface (CLI), use [log()](/reference/dbt-jinja-functions/log) to print the full contents:
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

```jinja
{{ log(model, info=True) }}
Expand All @@ -42,6 +42,48 @@

</Tabs>

## Batch properties for microbatch models

From dbt Core v1.9, the model object includes a `batch` property (`model.batch`), which provides details about the current batch when executing an [incremental microbatch](/docs/build/incremental-microbatch) model. This property is only populated during the batch execution of a microbatch model.

Check warning on line 47 in website/docs/reference/dbt-jinja-functions/model.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/reference/dbt-jinja-functions/model.md#L47

[custom.Typos] Oops there's a typo -- did you really mean 'v1.9'?
Raw output
{"message": "[custom.Typos] Oops there's a typo -- did you really mean 'v1.9'? ", "location": {"path": "website/docs/reference/dbt-jinja-functions/model.md", "range": {"start": {"line": 47, "column": 15}}}, "severity": "WARNING"}
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

The following table describes the properties of the `batch` object. Note that dbt appends the property to the `model` and `batch` objects.

| Property | Description | Example |
| -------- | ----------- | ------- |
| `id` | The unique identifier for the batch within the context of the microbatch model. | `model.batch.id` |
| `event_time_start` | The start time of the batch's [`event_time`](/reference/resource-configs/event-time) filter (inclusive). | `model.batch.event_time_start` |
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@QMalcolm what do we mean by 'inclusive' and 'exclusive' here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mirnawong1 great question ❤️

Short answer

Inclusive means the value is included in the batch, exclusive means the value is excluded from the batch

Long answer

To think specifically about inclusivity/exclusivity means with regard to a batch, consider the batch filter statement

WHERE event_time >= event_time_start AND event_time < event_time_end

Things to note

  1. The comparison of event_time to event_time_start used >= with the = part of the operator including the event_time_start as part of the range. This means the usage of event_time_start is inclusive in the range selection
  2. In the comparison of event_time to event_time_end we used < meaning things up to, but not equal to, the event_time_end will be selected as part of the range. This means that event_time_end is exclusive in the range selection.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh got it, thanks so much for taking the time to clarify 🙏

| `event_time_end` | The end time of the batch's `event_time` filter (exclusive). | `model.batch.event_time_end` |

### Usage notes

`model.batch` is only available during the execution of a microbatch model batch. Outside of the microbatch execution, `model.batch` is `None`, and its sub-properties aren't accessible.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We sort of state this above. Maybe we could combine the two into a callout if it's important enough


#### Example: Safeguard access to batch properties
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

We recommend to always check if `model.batch` is populated before accessing its properties. Use an `if` statement to ensure safe access to batch properties:
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

```jinja
{% if model.batch %}
QMalcolm marked this conversation as resolved.
Show resolved Hide resolved
{{ log(model.batch.id) }} # Log the batch ID #
{{ log(model.batch.event_time_start) }} # Log the start time of the batch #
{{ log(model.batch.event_time_end) }} # Log the end time of the batch #
{% endif %}
```

In this example, the `if model.batch` statement makes sure that the code only runs during a batch execution. `log()` is used to print the `batch` properties for debugging.

#### Example: Log batch details
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

This is a practical example of how you might use `model.batch` in a microbatch model to log batch details for the `batch.id`:

```jinja
{% if model.batch %}
QMalcolm marked this conversation as resolved.
Show resolved Hide resolved
{{ log("Processing batch with ID: " ~ model.batch.id, info=True) }}
{{ log("Batch event time range: " ~ model.batch.event_time_start ~ " to " ~ model.batch.event_time_end, info=True) }}
{% endif %}
```
In this example, the `if model.batch` statement makes sure that the code only runs during a batch execution. `log()` is used to print the `batch` properties for debugging.

## Model structure and JSON schema

To view the structure of `models` and their definitions:
Expand Down
Loading