Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics redesign #1389

Merged
merged 150 commits into from
Jan 10, 2025
Merged
Changes from 1 commit
Commits
Show all changes
150 commits
Select commit Hold shift + click to select a range
07d3a19
First draft of redesign.
Liraim Dec 3, 2024
368724f
Add examples
Liraim Dec 3, 2024
cf38a34
Add examples
Liraim Dec 3, 2024
371aa0d
Add FeatureScorers.
Liraim Dec 6, 2024
4953ef5
Add metric container, group by and support for llm_judge in scorers.
Liraim Dec 9, 2024
6ccffa6
Fix example.
Liraim Dec 9, 2024
7644cb1
Fix example.
Liraim Dec 9, 2024
ede9486
Add custom scorers.
Liraim Dec 9, 2024
c194b4b
Add first version of presets and report.
Liraim Dec 11, 2024
c350d95
Fix column name for FeatureScorer.
Liraim Dec 12, 2024
92bd2f5
Add implementations for column_max, column_mean, column_min and colum…
Liraim Dec 12, 2024
e142196
Support MetricPreset and MetricContainer in Reports.
Liraim Dec 13, 2024
ccab7fc
codegen scorers (#1394)
mike0sv Dec 13, 2024
e35ef8e
Add separate render for metrics and checks.
Liraim Dec 16, 2024
d451be5
Merge branch 'refs/heads/main' into feature/metrics-redesign
Liraim Dec 17, 2024
6a52347
rename scoreres->descriptors, add showcase.ipynb and add sql descriptor
mike0sv Dec 17, 2024
a61f7af
Collect checks in single list.
Liraim Dec 18, 2024
cca2ad3
Add example F1 metric (from legacy metrics) in new reports.
Liraim Dec 20, 2024
863f12a
Add metrics ROC AUC, f1, precision, recall.
Liraim Dec 22, 2024
e8556db
Add same render as in old metrics to RocAucMetric.
Liraim Dec 23, 2024
51af315
Rename checks to tests and reorganize code.
Liraim Dec 23, 2024
a4903b1
Bring back column_summary metrics.
Liraim Dec 23, 2024
5b765b0
Add quality by class preset.
Liraim Dec 24, 2024
cb3caa8
Remove tabs title and rename Checks to Tests.
Liraim Dec 24, 2024
f894998
support v2 metrics in v1 snapshots (#1398)
mike0sv Dec 24, 2024
81029c4
add mylabelvalue
mike0sv Dec 25, 2024
2ce0121
fix ipython import
mike0sv Dec 25, 2024
c831a51
fix
mike0sv Dec 25, 2024
a80e253
fix deduplication
mike0sv Dec 25, 2024
30e1e79
Add support for ByLabel tests.
Liraim Dec 25, 2024
7fca28e
Update column statistics metrics: min, max, mean,median, quantile and…
Liraim Dec 25, 2024
71ebb54
Fix group_by metrics and metric_workbench examples.
Liraim Dec 25, 2024
b9e04fa
make all reports
mike0sv Dec 26, 2024
79341e9
Add distribution widgets to metrics.
Liraim Dec 27, 2024
d5732b3
Merge remote-tracking branch 'origin/feature/metrics-redesign' into f…
mike0sv Dec 27, 2024
4ce3ea8
test widgets
mike0sv Dec 27, 2024
1368ba1
Add metrics InRange, OutRange, InList, OutList and fix some bugs.
Liraim Dec 27, 2024
5c123bc
add count value support
mike0sv Dec 27, 2024
f7c5e19
Move from evidently.v2 to evidently.future.
Liraim Dec 30, 2024
1312759
Rewrite tests.
Liraim Dec 30, 2024
788df14
Update examples.
Liraim Dec 30, 2024
4853eb1
Add dataset statistics metrics: RowCount, ColumnCount.
Liraim Dec 30, 2024
c5f07d6
Rename v2 to future fixes.
Liraim Dec 30, 2024
03ead97
Add example of dashboard with new metrics.
Liraim Dec 30, 2024
efe5fc6
Add dataset stats calculations on creation.
Liraim Dec 31, 2024
86f316d
Add value drift first version.
Liraim Dec 31, 2024
c6c32bf
Add ValueStats and DatasetStats.
Liraim Jan 1, 2025
2ae1833
Support for dict / json for snapshots.
Liraim Jan 1, 2025
8b2bda9
Support for legacy metrics.
Liraim Jan 2, 2025
bc3f164
Add empty rows and columns, constant columns metrics.
Liraim Jan 2, 2025
ada2b2b
Add MissingValuesCount metrics.
Liraim Jan 2, 2025
90ea710
update descriptors
mike0sv Jan 2, 2025
6355f18
fix init
mike0sv Jan 2, 2025
c2ce94d
fix descriptors showcase
mike0sv Jan 3, 2025
442e1db
Add metric config (#1403)
mike0sv Jan 3, 2025
197d092
Remove LegacyMetric and make dataset stats metrics simple single valu…
Liraim Jan 4, 2025
9511471
Update Classification metrics F1 by label, Precision by label, Recall…
Liraim Jan 4, 2025
c7ec559
Reorganize code with base types.
Liraim Jan 4, 2025
3c3b4d9
Add classification quality metrics: F1Score, Accuracy, Precision, Rec…
Liraim Jan 4, 2025
f8e0742
Add better example for custom descriptors.
Liraim Jan 4, 2025
dd9e401
First version of new DataDefinition.
Liraim Jan 5, 2025
1f793e4
Add ValueDrift example for text column.
Liraim Jan 6, 2025
386d2c1
Remove test results from metric config to calculations.
Liraim Jan 6, 2025
8135b9d
Fix errors after moving get_tests to calculations.
Liraim Jan 6, 2025
c08c122
renew registry
mike0sv Jan 6, 2025
294f93e
ensure registry import
mike0sv Jan 6, 2025
b2f9217
Add RocAuc metric.
Liraim Jan 6, 2025
1264cfe
Add LogLoss metric.
Liraim Jan 6, 2025
ff31bc1
Merge branch 'main' into feature/metrics-redesign
mike0sv Jan 6, 2025
06de356
Added a draft of a full list of descriptors, including non obvious ones
emeli-dral Jan 6, 2025
7cd182e
Add classification quality presets.
Liraim Jan 6, 2025
f24e2b5
Support add_descriptors in PandasDataset.
Liraim Jan 7, 2025
e89e4f9
Add ClassificationDummyQuality.
Liraim Jan 7, 2025
47250a0
updated descriptor list example
emeli-dral Jan 7, 2025
70563c6
Added custom callable examples
emeli-dral Jan 7, 2025
720ab08
Add Dataset.add_descriptors() method.
Liraim Jan 7, 2025
ff69ba1
Added more descriptor examples
emeli-dral Jan 7, 2025
373c691
Added more descriptor examples
emeli-dral Jan 7, 2025
32cd288
Added more descriptor examples
emeli-dral Jan 7, 2025
5079133
Fix pandas dataset creation without descriptors.
Liraim Jan 7, 2025
7b9614b
Add regression metrics.
Liraim Jan 8, 2025
36c31ec
add snapshot version
mike0sv Jan 8, 2025
d19454a
Add metric_id property for dashboards.
Liraim Jan 8, 2025
9e64f27
Fix typing in tests.
Liraim Jan 8, 2025
af193ba
Update regression metrics.
Liraim Jan 8, 2025
187bc40
Add RegressionQuality preset.
Liraim Jan 8, 2025
3cb83aa
Fix classification method and create column_mapping from data_definit…
Liraim Jan 8, 2025
0601485
fix DataDefinition and get_legacy_metric
DimaAmega Jan 8, 2025
802cf50
add new `bikes-v2`demo project
DimaAmega Jan 8, 2025
a349d2b
bit refactor: `list(DEMO_PROJECTS.keys())` -> `DEMO_PROJECTS_NAMES`
DimaAmega Jan 8, 2025
3189851
Added more descriptor examples
emeli-dral Jan 8, 2025
c64f32e
updated more descriptor example
emeli-dral Jan 8, 2025
60bc16a
fix CategoryCountCalculation calculation
emeli-dral Jan 8, 2025
b50eceb
add llm descriptors
mike0sv Jan 8, 2025
dbba5fc
add llm descriptors
mike0sv Jan 8, 2025
1ceedaf
add snapshot `version` into `@get("/{project_id:uuid}/{snapshot_id:uu…
DimaAmega Jan 8, 2025
07298ce
Feature/metrics redesign test configs (#1409)
mike0sv Jan 8, 2025
ab370d3
Fix tests method.
Liraim Jan 8, 2025
aa890b7
add aliases to registry
mike0sv Jan 8, 2025
751785d
added reviews demo project, generated ober future reports
emeli-dral Jan 8, 2025
279f96b
Add Regression dummy
Liraim Jan 8, 2025
25501a3
Add DataDrift preset and DriftedColumnsCount.
Liraim Jan 8, 2025
b09210b
Add TextEvals.
Liraim Jan 8, 2025
38d8f31
support `metrics` and `tests` tabs
DimaAmega Jan 9, 2025
64a07d1
commit UI assets
DimaAmega Jan 9, 2025
075962b
use metric v2 fingerprint in adapter
mike0sv Jan 9, 2025
ca1d817
Merge remote-tracking branch 'origin/feature/metrics-redesign' into f…
mike0sv Jan 9, 2025
bcbffa2
Make a copy of data_definition on Dataset creation.
Liraim Jan 9, 2025
d1b1c8e
fix descriptors alias
mike0sv Jan 9, 2025
097d2b2
Add data_definition property for datasets.
Liraim Jan 9, 2025
4e1610b
fix HF descriptors alias
mike0sv Jan 9, 2025
c3b9302
Merge remote-tracking branch 'origin/feature/metrics-redesign' into f…
mike0sv Jan 9, 2025
1600d13
Refactor TextEvals to include RowColumns only once.
Liraim Jan 9, 2025
de81bc3
Make a copy of data_definition on datset creation.
Liraim Jan 9, 2025
d171938
Do not throw exception if column not in column list.
Liraim Jan 9, 2025
0fddf73
Last attempt to copy data_definitions.
Liraim Jan 9, 2025
a766ade
added examples with LLMEvals
emeli-dral Jan 9, 2025
f81a41f
added llm judge descriptor example
emeli-dral Jan 9, 2025
4ac928c
added llm judge descriptor example
emeli-dral Jan 9, 2025
14f4487
added descriptor tutorial for users
emeli-dral Jan 9, 2025
711142d
added descriptor tutorial for users
emeli-dral Jan 9, 2025
39fc32f
add datadefinition backport
mike0sv Jan 9, 2025
d9e9f1f
Merge remote-tracking branch 'origin/feature/metrics-redesign' into f…
mike0sv Jan 9, 2025
d3c2bf1
support snapshotv2 in ws
mike0sv Jan 9, 2025
d84c97b
Add UniqueValueCount and new DistributionPanel.
Liraim Jan 10, 2025
c2a20d3
better `SnapshotWidgets` component
DimaAmega Jan 10, 2025
af3d668
Add target, prediction params in Regression.
Liraim Jan 10, 2025
d2e5ab0
revert using `version` on UI
DimaAmega Jan 10, 2025
e1c865d
add `reviews-v2` demo project
DimaAmega Jan 10, 2025
df1d96e
move version to metadata
mike0sv Jan 10, 2025
a1a062d
Merge remote-tracking branch 'origin/feature/metrics-redesign' into f…
mike0sv Jan 10, 2025
62880ff
move version to metadata
mike0sv Jan 10, 2025
22ef094
Add save_html, save_json methods.
Liraim Jan 10, 2025
1d6d1e5
Fix group_by metric.
Liraim Jan 10, 2025
ee1530e
tmp names in data definition
emeli-dral Jan 10, 2025
56027ba
minor rename in data definition
emeli-dral Jan 10, 2025
48a2919
fix test panels
mike0sv Jan 10, 2025
4773f5a
Merge remote-tracking branch 'origin/feature/metrics-redesign' into f…
mike0sv Jan 10, 2025
f2d9808
Fix mypy issues
Liraim Jan 10, 2025
b37bfa7
fix ui types
mike0sv Jan 10, 2025
fc9bc06
Fix mypy issues
Liraim Jan 10, 2025
23730bf
Merge remote-tracking branch 'origin/feature/metrics-redesign' into f…
mike0sv Jan 10, 2025
4d82521
Merge remote-tracking branch 'origin/feature/metrics-redesign' into f…
mike0sv Jan 10, 2025
e447c71
fix mypy
mike0sv Jan 10, 2025
4d46877
comment new v2 demo projects
DimaAmega Jan 10, 2025
dec649b
increase timeout
DimaAmega Jan 10, 2025
03884a6
Fix mypy issues
Liraim Jan 10, 2025
1fd7aa0
dvc: add new screenshots
DimaAmega Jan 10, 2025
629f9df
Fix tests.
Liraim Jan 10, 2025
4beb9e2
Fix tests.
Liraim Jan 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add snapshot version into `@get("/{project_id:uuid}/{snapshot_id:uu…
…id}/data")` handler and fix `bikes-v2` demo project
DimaAmega committed Jan 8, 2025
commit 1ceedaf6b0421fe966d230a4d7045e49f349320e
6 changes: 3 additions & 3 deletions src/evidently/ui/api/models.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import dataclasses
import datetime
from dataclasses import dataclass
from typing import Dict
@@ -104,10 +103,11 @@ class DashboardInfoModel:
widgets: List[BaseWidgetInfo]
min_timestamp: Optional[datetime.datetime] = None
max_timestamp: Optional[datetime.datetime] = None
version: str = "1"

@classmethod
def from_dashboard_info(cls, dashboard_info: DashboardInfo):
return cls(**dataclasses.asdict(dashboard_info))
def from_dashboard_info(cls, dashboard_info: DashboardInfo, version: str):
return cls(name=dashboard_info.name, widgets=dashboard_info.widgets, version=version)

@classmethod
async def from_project_with_time_range(
6 changes: 4 additions & 2 deletions src/evidently/ui/api/projects.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import asyncio
import datetime
import json
from dataclasses import asdict
@@ -203,8 +204,9 @@ async def get_snapshot_data(
snapshot_metadata: Annotated[SnapshotMetadata, Dependency()],
log_event: Callable,
) -> str:
info = DashboardInfoModel.from_dashboard_info(await snapshot_metadata.get_dashboard_info())
snapshot = await snapshot_metadata.load()
dashboard_info, snapshot = await asyncio.gather(snapshot_metadata.get_dashboard_info(), snapshot_metadata.load())
info = DashboardInfoModel.from_dashboard_info(dashboard_info=dashboard_info, version=snapshot.version)

log_event(
"get_snapshot_data",
snapshot_type="report" if snapshot.is_report else "test_suite",
1 change: 1 addition & 0 deletions src/evidently/ui/demo_projects/adult.py
Original file line number Diff line number Diff line change
@@ -97,6 +97,7 @@ def create_project(workspace: WorkspaceBase, name: str):
adult_demo_project = DemoProject(
name="Demo project - Adult",
create_data=create_data,
create_snapshot=None,
create_report=None,
create_project=create_project,
create_test_suite=create_test_suite,
6 changes: 6 additions & 0 deletions src/evidently/ui/demo_projects/base.py
Original file line number Diff line number Diff line change
@@ -9,6 +9,7 @@

from evidently import ColumnMapping
from evidently.report import Report
from evidently.suite.base_suite import Snapshot
from evidently.test_suite import TestSuite
from evidently.ui.base import Project
from evidently.ui.workspace import Workspace
@@ -26,6 +27,7 @@ class DemoProject:

create_data: Callable[[], DemoData]
create_report: Optional[Callable[[int, DemoData], Report]]
create_snapshot: Optional[Callable[[int, DemoData], Snapshot]]
create_test_suite: Optional[Callable[[int, DemoData], TestSuite]]
count: int

@@ -46,6 +48,10 @@ def create(self, workspace: Union[str, WorkspaceBase]):
data = self.create_data()

for i in range(0, self.count):
if self.create_snapshot is not None:
snapshot = self.create_snapshot(i, data)
ws.add_snapshot(project.id, snapshot)

if self.create_report is not None:
report = self.create_report(i, data)
ws.add_report(project.id, report)
1 change: 1 addition & 0 deletions src/evidently/ui/demo_projects/bikes.py
Original file line number Diff line number Diff line change
@@ -286,6 +286,7 @@ def create_project(workspace: WorkspaceBase, name: str):
bikes_demo_project = DemoProject(
name="Demo project - Bikes",
create_data=create_data,
create_snapshot=None,
create_report=create_report,
create_project=create_project,
create_test_suite=create_test_suite,
9 changes: 5 additions & 4 deletions src/evidently/ui/demo_projects/bikes_v2.py
Original file line number Diff line number Diff line change
@@ -30,7 +30,7 @@
from evidently.ui.workspace.base import WorkspaceBase


def create_report(i: int, data: Tuple[DataFrame, DataFrame, ColumnMapping]):
def create_snapshot(i: int, data: Tuple[DataFrame, DataFrame, ColumnMapping]):
current, reference, column_mapping = data

report = Report(
@@ -60,7 +60,7 @@ def create_report(i: int, data: Tuple[DataFrame, DataFrame, ColumnMapping]):

snapshot = report.run(dataset, None)

return snapshot_v2_to_v1(snapshot).as_report()
return snapshot_v2_to_v1(snapshot)


def noop():
@@ -204,8 +204,9 @@ def create_project(workspace: WorkspaceBase, name: str):
bikes_v2_demo_project = DemoProject(
name="Demo project - Bikes v2",
create_data=create_data,
create_report=create_report,
create_project=create_project,
create_snapshot=create_snapshot,
create_report=None,
create_test_suite=None,
create_project=create_project,
count=28,
)
1 change: 1 addition & 0 deletions src/evidently/ui/demo_projects/reviews.py
Original file line number Diff line number Diff line change
@@ -418,6 +418,7 @@ def create_project(workspace: WorkspaceBase, name: str):

reviews_demo_project = DemoProject(
name="Demo project - Reviews",
create_snapshot=None,
create_data=create_data,
create_report=create_report,
create_project=create_project,
1 change: 1 addition & 0 deletions src/evidently/ui/demo_projects/simple.py
Original file line number Diff line number Diff line change
@@ -66,6 +66,7 @@ def create_project(workspace: WorkspaceBase, name: str):

simple_demo_project = DemoProject(
name="Demo project - Simple",
create_snapshot=None,
create_data=create_data,
create_report=create_report,
create_project=create_project,