Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python: log experiment with html #1192

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

python: log experiment with html #1192

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 21 additions & 6 deletions python/langsmith/evaluation/_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""V2 Evaluation Interface."""

Check notice on line 1 in python/langsmith/evaluation/_runner.py

View workflow job for this annotation

GitHub Actions / benchmark

Benchmark results 

.........................................
create_5_000_run_trees: Mean +- std dev: 617 ms +- 41 ms
.........................................
create_10_000_run_trees: Mean +- std dev: 1.19 sec +- 0.05 sec
.........................................
create_20_000_run_trees: Mean +- std dev: 1.18 sec +- 0.06 sec
.........................................
dumps_class_nested_py_branch_and_leaf_200x400: Mean +- std dev: 707 us +- 24 us
.........................................
dumps_class_nested_py_leaf_50x100: Mean +- std dev: 25.4 ms +- 0.2 ms
.........................................
dumps_class_nested_py_leaf_100x200: Mean +- std dev: 104 ms +- 2 ms
.........................................
dumps_dataclass_nested_50x100: Mean +- std dev: 25.7 ms +- 0.3 ms
.........................................
WARNING: the benchmark result may be unstable
* the standard deviation (16.1 ms) is 24% of the mean (66.7 ms)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

dumps_pydantic_nested_50x100: Mean +- std dev: 66.7 ms +- 16.1 ms
.........................................
WARNING: the benchmark result may be unstable
* the standard deviation (30.5 ms) is 14% of the mean (220 ms)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

dumps_pydanticv1_nested_50x100: Mean +- std dev: 220 ms +- 30 ms

Check notice on line 1 in python/langsmith/evaluation/_runner.py

View workflow job for this annotation

GitHub Actions / benchmark

Comparison against main 

+------------------------------------+----------+------------------------+
| Benchmark                          | main     | changes                |
+====================================+==========+========================+
| dumps_pydantic_nested_50x100       | 71.7 ms  | 66.7 ms: 1.08x faster  |
+------------------------------------+----------+------------------------+
| create_20_000_run_trees            | 1.21 sec | 1.18 sec: 1.02x faster |
+------------------------------------+----------+------------------------+
| dumps_dataclass_nested_50x100      | 25.9 ms  | 25.7 ms: 1.01x faster  |
+------------------------------------+----------+------------------------+
| dumps_class_nested_py_leaf_50x100  | 25.5 ms  | 25.4 ms: 1.00x faster  |
+------------------------------------+----------+------------------------+
| dumps_class_nested_py_leaf_100x200 | 104 ms   | 104 ms: 1.01x slower   |
+------------------------------------+----------+------------------------+
| Geometric mean                     | (ref)    | 1.02x faster           |
+------------------------------------+----------+------------------------+

Benchmark hidden because not significant (4): dumps_pydanticv1_nested_50x100, create_5_000_run_trees, create_10_000_run_trees, dumps_class_nested_py_branch_and_leaf_200x400

from __future__ import annotations

Expand All @@ -7,12 +7,14 @@
import concurrent.futures as cf
import datetime
import functools
import importlib
import inspect
import itertools
import logging
import pathlib
import queue
import random
import sys
import textwrap
import threading
import uuid
Expand Down Expand Up @@ -94,6 +96,9 @@
],
]

_IS_INTERACTIVE_ENV = hasattr(sys, "ps2")
_IPYTHON_INSTALLED = importlib.util.find_spec("IPython")


def evaluate(
target: TARGET_T,
Expand Down Expand Up @@ -1113,15 +1118,25 @@
f"{base_url}/datasets/{dataset_id}/compare?"
f"selectedSessions={project.id}"
)
print( # noqa: T201
f"View the evaluation results for experiment: '{self.experiment_name}'"
f" at:\n{comparison_url}\n\n"
)
else:
# HACKHACK
comparison_url = None

if _IS_INTERACTIVE_ENV and _IPYTHON_INSTALLED and comparison_url:
from IPython.display import HTML, display # type: ignore[import-not-found]

display(
HTML(
f"RUNNING EXPERIMENT: <a href='{comparison_url}'>"
f"{self.experiment_name}</a>"
)
)
elif comparison_url:
print( # noqa: T201
"Starting evaluation of experiment: %s", self.experiment_name
f"RUNNING EXPERIMENT: {self.experiment_name}\nEXPERIMENT URL: "
f"{comparison_url}"
)
else:
print(f"RUNNING EXPERIMENT: {self.experiment_name}") # noqa: T201


class _ExperimentManager(_ExperimentManagerMixin):
Expand Down
Loading