Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add adapter telemetry to snowplow event. #10859

Merged
merged 13 commits into from
Oct 28, 2024
Merged

Add adapter telemetry to snowplow event. #10859

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
7b11819
Add adapter telemetry to snowplow event.
VersusFacit Oct 16, 2024
74394f5
Temporary dev branch switch.
VersusFacit Oct 16, 2024
1c49168
Set tracking for overrideable adapter method.
VersusFacit Oct 18, 2024
8f1eabe
Do safer adapter ref.
VersusFacit Oct 18, 2024
7374c7f
Merge branch 'main' into ADAP-301/add-adapter-telemetry
VersusFacit Oct 18, 2024
bd45569
Improve comment.
VersusFacit Oct 18, 2024
11c7ba9
Code review comments.
VersusFacit Oct 22, 2024
ea7f908
Merge branch 'main' into ADAP-301/add-adapter-telemetry
VersusFacit Oct 22, 2024
d56914c
Don't call the asdict on a dict.
VersusFacit Oct 22, 2024
59367f3
Bump ci to pull in fix from base adapter.
VersusFacit Oct 23, 2024
666e139
Add unit tests for coverage.
VersusFacit Oct 23, 2024
556bc2e
Update field name from base adapter/schema change.
VersusFacit Oct 28, 2024
03dfd41
remove breakpoint.
VersusFacit Oct 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions core/dbt/task/run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import threading
import time
from copy import deepcopy
from dataclasses import asdict
from datetime import datetime
from typing import AbstractSet, Any, Dict, Iterable, List, Optional, Set, Tuple, Type

Expand Down Expand Up @@ -99,7 +100,7 @@
return status, message


def track_model_run(index, num_nodes, run_model_result):
def track_model_run(index, num_nodes, run_model_result, adapter=BaseAdapter):
VersusFacit marked this conversation as resolved.
Show resolved Hide resolved
if tracking.active_user is None:
raise DbtInternalError("cannot track model run with no active user")
invocation_id = get_invocation_id()
Expand All @@ -115,6 +116,12 @@
contract_enforced = False
versioned = False
incremental_strategy = None

# Each adapter returns a dataclass with a flexible dictionary for
# adapter-specific fields. Only the non-'adapter_details' fields
# are guaranteed cross adapter.
adapter_info = adapter.get_adapter_run_info(run_model_result.node.config)

Check warning on line 123 in core/dbt/task/run.py

View check run for this annotation

Codecov / codecov/patch

core/dbt/task/run.py#L123 

Added line #L123 was not covered by tests

tracking.track_model_run(
{
"invocation_id": invocation_id,
Expand All @@ -134,6 +141,7 @@
"contract_enforced": contract_enforced,
"access": access,
"versioned": versioned,
"adapter_info": asdict(adapter_info),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered that this should be something JSON safe, which a Python dict is 🫡

}
)

Expand Down Expand Up @@ -282,7 +290,7 @@
self.print_start_line()

def after_execute(self, result) -> None:
track_model_run(self.node_index, self.num_nodes, result)
track_model_run(self.node_index, self.num_nodes, result, adapter=self.adapter)

Check warning on line 293 in core/dbt/task/run.py

View check run for this annotation

Codecov / codecov/patch

core/dbt/task/run.py#L293 

Added line #L293 was not covered by tests
self.print_result_line(result)

def _build_run_model_result(self, model, context, elapsed_time: float = 0.0):
Expand Down
2 changes: 1 addition & 1 deletion core/dbt/tracking.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
RESOURCE_COUNTS = "iglu:com.dbt/resource_counts/jsonschema/1-0-1"
RPC_REQUEST_SPEC = "iglu:com.dbt/rpc_request/jsonschema/1-0-1"
RUNNABLE_TIMING = "iglu:com.dbt/runnable/jsonschema/1-0-0"
RUN_MODEL_SPEC = "iglu:com.dbt/run_model/jsonschema/1-0-4"
RUN_MODEL_SPEC = "iglu:com.dbt/run_model/jsonschema/1-0-5"
VersusFacit marked this conversation as resolved.
Show resolved Hide resolved
PLUGIN_GET_NODES = "iglu:com.dbt/plugin_get_nodes/jsonschema/1-0-0"

SNOWPLOW_TRACKER_VERSION = Version(snowplow_version)
Expand Down
2 changes: 1 addition & 1 deletion dev-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
git+https://github.com/dbt-labs/dbt-adapters.git@main
git+https://github.com/dbt-labs/dbt-adapters.git@ADAP-301/add-adapter-telemetry
git+https://github.com/dbt-labs/dbt-adapters.git@main#subdirectory=dbt-tests-adapter
git+https://github.com/dbt-labs/dbt-common.git@main
git+https://github.com/dbt-labs/dbt-postgres.git@main
Expand Down
Loading