Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow dbt show and dbt compile to output JSON without extra logs #9958

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
+287 −195
Open

Allow dbt show and dbt compile to output JSON without extra logs #9958

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
920fe11
Allow `dbt show` and `dbt compile` to output JSON without extra logs
dbeatty10 Apr 16, 2024
0df6227
Add `quiet` attribute for ShowNode and CompiledNode messages
dbeatty10 Apr 16, 2024
c3c605d
Output of protoc compiler
dbeatty10 Apr 16, 2024
91e501e
Utilize the `quiet` attribute for ShowNode and CompiledNode
dbeatty10 Apr 16, 2024
15b94ce
Reuse the `dbt list` approach when the `--quiet` flag is used
dbeatty10 Apr 16, 2024
23149d3
Merge branch 'main' into dbeatty/9840-quiet-show
dbeatty10 Apr 16, 2024
0641ce2
Merge remote-tracking branch 'origin/main' into dbeatty/9840-quiet-show
dbeatty10 Apr 17, 2024
4c488e7
Merge remote-tracking branch 'origin/main' into dbeatty/9840-quiet-show
dbeatty10 Apr 25, 2024
456b5d5
Merge branch 'main' into dbeatty/9840-quiet-show
dbeatty10 Jul 24, 2024
f9b974d
Use PrintEvent to get to stdout even if the logger is set to ERROR
dbeatty10 Jul 24, 2024
f669e05
Functional tests for quiet compile
dbeatty10 Nov 4, 2024
5d1f216
Functional tests for quiet show
dbeatty10 Nov 4, 2024
6858a6d
Fire event same way regardless if LOG_FORMAT is json or not
dbeatty10 Nov 4, 2024
091575d
Merge branch 'main' into dbeatty/9840-quiet-show
dbeatty10 Nov 4, 2024
e825a8c
Switch back to firing ShowNode and CompiledNode events
dbeatty10 Nov 5, 2024
e804fc7
Make `--inline-direct` to be quiet-compatible
dbeatty10 Nov 5, 2024
1b1becf
Temporarily change to dev branch for dbt-common
dbeatty10 Nov 5, 2024
ee0d677
Remove extraneous newline
dbeatty10 Nov 5, 2024
df7aaee
Functional test for `--quiet` for `--inline-direct` flag
dbeatty10 Nov 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changes/unreleased/Features-20240416-123234.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: Allow `dbt show` and `dbt compile` to output JSON without extra logs
time: 2024-04-16T12:32:34.764394-06:00
custom:
Author: dbeatty10
Issue: "9840"
2 changes: 2 additions & 0 deletions core/dbt/events/core_types.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1571,6 +1571,7 @@ message ShowNode {
bool is_inline = 3;
string output_format = 4;
string unique_id = 5;
bool quiet = 6;
}

message ShowNodeMsg {
Expand All @@ -1585,6 +1586,7 @@ message CompiledNode {
bool is_inline = 3;
string output_format = 4;
string unique_id = 5;
bool quiet = 6;
}

message CompiledNodeMsg {
Expand Down
349 changes: 174 additions & 175 deletions core/dbt/events/core_types_pb2.py
View file
Open in desktop

Large diffs are not rendered by default.

8 changes: 6 additions & 2 deletions core/dbt/events/types.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1578,7 +1578,9 @@ def message(self) -> str:
{"node": self.node_name, "show": json.loads(self.preview)}, indent=2
)
else:
if self.is_inline:
if self.quiet:
return self.preview
elif self.is_inline:
return f"Previewing inline node:\n{self.preview}"
else:
return f"Previewing node '{self.node_name}':\n{self.preview}"
Expand All @@ -1595,7 +1597,9 @@ def message(self) -> str:
else:
return json.dumps({"node": self.node_name, "compiled": self.compiled}, indent=2)
else:
if self.is_inline:
if self.quiet:
return self.compiled
elif self.is_inline:
return f"Compiled inline node is:\n{self.compiled}"
else:
return f"Compiled node '{self.node_name}' is:\n{self.compiled}"
Expand Down
23 changes: 15 additions & 8 deletions core/dbt/task/compile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import threading

from dbt.flags import get_flags
from dbt.artifacts.schemas.run import RunStatus, RunResult
from dbt_common.events.base_types import EventLevel
from dbt_common.events.functions import fire_event
Expand Down Expand Up @@ -89,16 +90,22 @@ def task_end_messages(self, results):
matched_results = []

for result in matched_results:
fire_event(
CompiledNode(
node_name=result.node.name,
compiled=result.node.compiled_code,
is_inline=is_inline,
output_format=output_format,
unique_id=result.node.unique_id,
)

compiled_node_event = CompiledNode(
node_name=result.node.name,
compiled=result.node.compiled_code,
is_inline=is_inline,
output_format=output_format,
unique_id=result.node.unique_id,
quiet=get_flags().QUIET,
)

if get_flags().LOG_FORMAT == "json":
fire_event(compiled_node_event)
else:
# Cleaner to leave as print than to mutate the logger not to print timestamps.
print(compiled_node_event.message())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use the new PrintEvent being added to dbt-common.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! I'll check out dbt-labs/dbt-common#130 and give PrintEvent a try.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self: see #10131 for an example how to use PrintEvent.


def _runtime_initialize(self):
if getattr(self.args, "inline", None):
try:
Expand Down
22 changes: 14 additions & 8 deletions core/dbt/task/show.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
import threading
import time

from dbt.flags import get_flags
from dbt.context.providers import generate_runtime_model_context
from dbt.contracts.graph.nodes import SeedNode
from dbt.artifacts.schemas.run import RunResult, RunStatus
Expand Down Expand Up @@ -98,16 +99,21 @@ def task_end_messages(self, results):
if hasattr(result.node, "version") and result.node.version:
node_name += f".v{result.node.version}"

fire_event(
ShowNode(
node_name=node_name,
preview=output.getvalue(),
is_inline=is_inline,
output_format=self.args.output,
unique_id=result.node.unique_id,
)
show_node_event = ShowNode(
node_name=node_name,
preview=output.getvalue(),
is_inline=is_inline,
output_format=self.args.output,
unique_id=result.node.unique_id,
quiet=get_flags().QUIET,
)

if get_flags().LOG_FORMAT == "json":
fire_event(show_node_event)
else:
# Cleaner to leave as print than to mutate the logger not to print timestamps.
print(show_node_event.message())

def _handle_result(self, result):
super()._handle_result(result)

Expand Down
Loading