Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: evidentlyai/evidently
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.4.37
Choose a base ref
...
head repository: evidentlyai/evidently
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref

Commits on Sep 6, 2024

  1. Copy the full SHA
    4d83af5 View commit details

Commits on Sep 7, 2024

  1. Copy the full SHA
    9d7990b View commit details

Commits on Sep 9, 2024

  1. UI: fix biome (#1291)

    * Update .pre-commit-config.yaml
    
    * Update README.md
    
    * Update ui.yml
    DimaAmega authored Sep 9, 2024
    Copy the full SHA
    5410270 View commit details
  2. Copy the full SHA
    29dca00 View commit details

Commits on Sep 10, 2024

  1. Update NavigationProgress.tsx z-index (#1295)

    * Update NavigationProgress.tsx
    
    * commit UI assets
    DimaAmega authored Sep 10, 2024
    Copy the full SHA
    cdd39cf View commit details

Commits on Sep 11, 2024

  1. Copy the full SHA
    dc562bc View commit details
  2. add base api model (#1298)

    mike0sv authored Sep 11, 2024
    Copy the full SHA
    5813560 View commit details

Commits on Sep 13, 2024

  1. Fixing minor typo (#1303)

    sapenov authored Sep 13, 2024
    Copy the full SHA
    05023a3 View commit details

Commits on Sep 15, 2024

  1. Copy the full SHA
    68f56f7 View commit details

Commits on Sep 18, 2024

  1. UUID7 (#1267)

    * UUID7
    
    * fixes and hopefully lint
    
    * fix mypy
    
    * fix tests
    
    * lint
    
    * validate new and old uuids
    
    * remove uuid7 requirement
    
    * remove more uuid4
    
    * serialization
    
    * change aliases to base uuid type
    
    * remove ser stuff
    
    * introduce new_id
    
    * fix
    
    * move
    mike0sv authored Sep 18, 2024
    Copy the full SHA
    40817e1 View commit details
  2. ALIES INVASION (#1301)

    * add alias_required
    
    * remove abc
    
    * fix tests
    
    * alias test WIP
    
    * alias test WIP
    
    * alias test WIP
    
    * alias test WIP
    
    * alias test WIP
    
    * alias test WIP
    
    * alias test WIP
    
    * alias test WIP
    
    * add aliases (#1302)
    
    * alias registries and tests
    
    * test classpaths exist
    
    * fix mypy
    
    * transitive aliases
    
    * fix lint
    
    * better error
    mike0sv authored Sep 18, 2024
    Copy the full SHA
    3a9e199 View commit details

Commits on Sep 23, 2024

  1. Copy the full SHA
    59c57d6 View commit details

Commits on Sep 25, 2024

  1. make ui/base async (#1307)

    mike0sv authored Sep 25, 2024
    Copy the full SHA
    9133cf6 View commit details
  2. Copy the full SHA
    55f49ad View commit details

Commits on Sep 26, 2024

  1. Copy the full SHA
    8472cbf View commit details

Commits on Sep 27, 2024

  1. Version 0.4.38

    emeli-dral committed Sep 27, 2024
    Copy the full SHA
    cd6cbfb View commit details

Commits on Oct 5, 2024

  1. Copy the full SHA
    42d836a View commit details

Commits on Oct 11, 2024

  1. run-snapshot-async (#1330)

    * add task_id to snapshot links
    ---------
    
    Co-authored-by: mike0sv <mike0sv@gmail.com>
    svetleo-evidently and mike0sv authored Oct 11, 2024
    Copy the full SHA
    1a3728d View commit details

Commits on Oct 14, 2024

  1. Copy the full SHA
    634568b View commit details

Commits on Oct 16, 2024

  1. Copy the full SHA
    adc7c51 View commit details
  2. Feature/dataset generator (#1340)

    * base
    
    * i am a banana
    
    * fix example
    
    * generate_dataset_from_docs
    
    * wip
    
    * wip
    
    * wip
    
    * wip
    
    * a draft code for a RAG dataset generation
    
    * generate_dataset_from_docs
    
    * generate_dataset_from_docs
    
    * generate_dataset_from_docs
    
    * readable version
    
    * wip
    
    * wip
    
    * wip
    
    * generate_dataset_from_docs
    
    * prompts
    
    * generate_dataset_from_docs
    
    * generate_dataset_from_docs
    
    * fixes
    
    * fixes
    
    * generate_dataset_from_docs
    
    * chunks count
    
    * async
    
    * generate_dataset_from_docs
    
    * generate_dataset_from_docs
    
    * generate_dataset_from_docs
    
    * generate_dataset_from_docs
    
    * generate_dataset_from_docs
    
    * WIP
    
    * fix
    
    * rename
    
    * fix import
    
    * move system prompts into user
    
    * generate_dataset_from_docs
    
    * generate_dataset_from_docs
    
    * generate_dataset_from_docs
    
    * generate_dataset_from_docs
    
    * prompt function signature
    
    * function signature validation
    
    * requirements
    
    * requirements
    
    * requirements
    
    * requirements
    
    * requirements
    
    * requirements
    
    * lil cleanup
    
    * mypy
    
    * move, add splitter
    
    * fix example and deps
    
    * rename
    
    * lint
    
    * lint
    
    * audit
    
    * type aliases and audit
    
    * type aliases and sudit and llm util refactor and stuff
    
    * fix import
    
    * pip audit
    
    * pip audit
    
    * reg
    
    * pip audit
    
    * remove
    
    ---------
    
    Co-authored-by: Svetlana Popova <svetleo@evidentlyai.com>
    Co-authored-by: Emeli Dral <emeli.dral@gmail.com>
    3 people authored Oct 16, 2024
    Copy the full SHA
    8a05265 View commit details
  3. Remove pip-audit (#1345)

    Liraim authored Oct 16, 2024
    Copy the full SHA
    5c933ea View commit details
  4. Add JSONSchemaMatch feature and descriptor (#1342)

    * Add JSONSchemaMatch() descriptor
    
    * Add feature for JSONSchemaMatch() descriptor
    
    * Tests for JSONSchemaMatch() descriptor
    
    * Update docs to include documentation for JSONSchemaMatch() descriptor
    
    * Reformat JSONSchemaMatch in feature registry
    
    ---------
    
    Co-authored-by: Trey Capps <treycapps@Treys-MacBook-Pro.local>
    trey-capps and Trey Capps authored Oct 16, 2024
    Copy the full SHA
    4bd1c13 View commit details

Commits on Oct 17, 2024

  1. IsValidJSON() descriptor (#1325)

    * isvalidjson descriptor
    
    * test cases
    
    * review comments
    
    * doc
    
    * review comments
    
    * review comments
    
    * review comments
    
    ---------
    
    Co-authored-by: Emeli Dral <emeli@evidentlyai.com>
    gagan-bhullar-tech and emeli-dral authored Oct 17, 2024
    Copy the full SHA
    9f2fbb8 View commit details
  2. Copy the full SHA
    aaf6423 View commit details
  3. Add WordMatch + WordNoMatch descriptor (#1334)

    * initial implementation
    
    * Initial unit tests
    
    * Update test_words_feature.py
    
    * clean up
    
    * Remove extra input data
    
    * Initial implementation
    
    * Update type alias
    
    * Add WordNoMatch to init
    
    * add display name parameter
    
    * Add test case from original issue
    
    * Fix incorrect aliases
    
    * update default display name
    
    * add column check to tests
    
    * fix failing code analysis
    
    * fix double quote syntax issue
    
    ---------
    
    Co-authored-by: Emeli Dral <emeli@evidentlyai.com>
    jon-bown and emeli-dral authored Oct 17, 2024
    Copy the full SHA
    0d02c62 View commit details

Commits on Oct 18, 2024

  1. use new wrapper/prompt api (#1346)

    * use new wrapper/prompt api
    
    * wip
    
    * mypy
    
    * mypy
    
    * oops
    
    * test run report with judge
    mike0sv authored Oct 18, 2024
    Copy the full SHA
    5f87168 View commit details

Commits on Oct 22, 2024

  1. Copy the full SHA
    b61d96f View commit details
  2. Copy the full SHA
    d3e21fb View commit details
  3. Copy the full SHA
    3106c3b View commit details
  4. Copy the full SHA
    082f131 View commit details
  5. Copy the full SHA
    edfa07a View commit details

Commits on Oct 23, 2024

  1. Copy the full SHA
    3c3b9fb View commit details
  2. Update mypy version and rules.

    Liraim committed Oct 23, 2024
    Copy the full SHA
    e361740 View commit details
  3. Copy the full SHA
    899c783 View commit details
  4. Copy the full SHA
    5628c8b View commit details
  5. Copy the full SHA
    53d3b7b View commit details
  6. Make MetricRenderer generic.

    Liraim committed Oct 23, 2024
    Copy the full SHA
    46a71b3 View commit details
  7. Copy the full SHA
    9ca4525 View commit details
  8. Copy the full SHA
    284ff4a View commit details
  9. Copy the full SHA
    483e743 View commit details
  10. Copy the full SHA
    e40e920 View commit details
  11. Copy the full SHA
    7b05262 View commit details
  12. Copy the full SHA
    9035eb2 View commit details
  13. Copy the full SHA
    35fbad6 View commit details
  14. Copy the full SHA
    4611385 View commit details
  15. Make Recsys tests generic.

    Liraim committed Oct 23, 2024
    Copy the full SHA
    a845a3a View commit details
  16. Copy the full SHA
    06a1d26 View commit details
  17. Copy the full SHA
    ef71527 View commit details
  18. Copy the full SHA
    fa90a1f View commit details
Showing 563 changed files with 26,228 additions and 9,123 deletions.
8 changes: 7 additions & 1 deletion .github/share-actions/ui-node-pnpm-install/action.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
name: UI install
inputs:
args:
type: string
description: ""
required: false
default: ""
runs:
using: "composite"
steps:
@@ -15,5 +21,5 @@ runs:

- name: 📥 Install node dependencies
working-directory: ui
run: pnpm i --frozen-lockfile --ignore-scripts
run: pnpm i --frozen-lockfile --ignore-scripts ${{ inputs.args }}
shell: bash
4 changes: 3 additions & 1 deletion .github/workflows/examples.yml
Original file line number Diff line number Diff line change
@@ -42,8 +42,10 @@ jobs:
if: matrix.minimal
run: pip install -r requirements.min.txt
- name: Prepare examples dependencies
run: pip install catboost sentence-transformers
run: pip install catboost sentence-transformers openai
- name: Export examples
run: jupyter nbconvert --to python examples/*/*.ipynb --output-dir example_scripts
- name: Run examples
run: python example_test.py
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY_GIT_DEV }}
15 changes: 8 additions & 7 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
@@ -98,6 +98,8 @@ jobs:
- changed_files
- prepare-cache-data
if: ${{ github.event.pull_request.draft == false && needs.changed_files.outputs.evidently_any_modified == 'true' }}
env:
EVIDENTLY_TEST_ENVIRONMENT: 1
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
@@ -115,22 +117,22 @@ jobs:
- name: Install minimal dependencies
run: pip install -r requirements.min.txt
- name: Install package
run: pip install -e .[dev,spark,fsspec]
- name: Run pip-audit
run: pip-audit --ignore-vuln PYSEC-2024-48 --ignore-vuln GHSA-jw8x-6495-233v --ignore-vuln GHSA-4hq2-rpgc-r8r7
run: pip install -e .[dev,spark,fsspec,llm]
- name: Run Tests
run: python -m pytest --durations=50
test:
# The type of runner that the job will run on
name: Test ${{ matrix.os }} with py${{ matrix.python }}
env:
EVIDENTLY_TEST_ENVIRONMENT: 1
needs:
- linter
- prepare-cache-data
runs-on: ${{ matrix.os }}
if: github.event.pull_request.draft == false
strategy:
matrix:
os: [ubuntu-22.04, windows-2022, macos-13]
os: [ubuntu-22.04, windows-2022, macos-14]
python: ["3.8", "3.9", "3.10", "3.11", "3.12"]
exclude:
- os: windows-latest
@@ -141,7 +143,6 @@ jobs:
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
architecture: "x64"
cache: "pip"
cache-dependency-path: setup.py
- uses: actions/cache@v4
@@ -155,7 +156,7 @@ jobs:
uses: ./.github/share-actions/get-bikes-dataset-cached

- name: Install package
run: pip install -e .[dev,spark,fsspec]
run: pip install -e .[dev,spark,fsspec,llm]
- name: Run Tests
run: python -m pytest --durations=50

@@ -173,7 +174,7 @@ jobs:
cache: "pip"
cache-dependency-path: setup.py
- name: Install dependencies
run: pip install -e ".[dev]"
run: pip install -e .
- name: Install wheel
run: pip install wheel
- name: Build package
31 changes: 15 additions & 16 deletions .github/workflows/ui.yml
Original file line number Diff line number Diff line change
@@ -57,8 +57,8 @@ jobs:
echo "One or more evidently_python file(s) has changed."
echo "List all the files that have changed: ${{ steps.changed-files.outputs.evidently_python_all_changed_and_modified_files }}"
ui-type-check:
name: UI type-check
ui-code-check:
name: UI code-check
runs-on: ubuntu-22.04
needs: changed_files
if: ${{ github.event.pull_request.draft == false && (needs.changed_files.outputs.ui_any_modified == 'true' || needs.changed_files.outputs.evidently_python_any_modified == 'true') }}
@@ -77,6 +77,10 @@ jobs:
working-directory: ui
run: pnpm type-check

- name: 🔬 Check code quality
working-directory: ui
run: pnpm code-check

ui-deps-analyze:
name: UI deps analyze
runs-on: ubuntu-22.04
@@ -147,26 +151,21 @@ jobs:
uses: ./.github/share-actions/ui-node-pnpm-install

- name: Install Playwright Browsers
working-directory: ui
run: pnpm dlx playwright@1.43.0 install --with-deps
working-directory: ui/service
run: pnpm exec playwright install --with-deps chromium

- uses: actions/cache@v3
id: cache-bikes-dataset
env:
cache-name: cache-bikes-dataset
with:
path: Bike-Sharing-Dataset.zip
key: cache-bikes-dataset
- name: Download test data
if: ${{ steps.cache-bikes-dataset.outputs.cache-hit != 'true' }}
run: curl -k https://archive.ics.uci.edu/static/public/275/bike+sharing+dataset.zip -o Bike-Sharing-Dataset.zip
- name: 🔍 Get bikes dataset cached
uses: ./.github/share-actions/get-bikes-dataset-cached

- name: Run UI
run: EXPERIMENTAL_DETERMINISTIC_UUID="true" evidently ui --port 8000 --workspace workspace-for-visual-testing --demo-projects all &
env:
EXPERIMENTAL_DETERMINISTIC_UUID: "true"
EVIDENTLY_TEST_ENVIRONMENT: 1
run: evidently ui --port 8000 --workspace workspace-for-visual-testing --demo-projects all &

- name: Wait UI to be ready to test
working-directory: ui/service
run: pnpm wait-on tcp:127.0.0.1:8000 -t 200000
run: pnpm wait-on tcp:127.0.0.1:8000 -t 6m

- name: Run Service Playwright tests
working-directory: ui/service
9 changes: 5 additions & 4 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -19,7 +19,8 @@ repos:
args: [--exit-non-zero-on-fix, --fix]
- id: ruff-format

# - repo: https://github.com/pre-commit/mirrors-prettier
# rev: "v3.0.3"
# hooks:
# - id: prettier
- repo: https://github.com/biomejs/pre-commit
rev: "v0.4.0"
hooks:
- id: biome-check
additional_dependencies: ["@biomejs/biome@1.8.3"]
53 changes: 53 additions & 0 deletions biome.jsonc
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
{
"files": {
// `endpoints.d.ts` is autogenerated
// `JsonParser.ts` copied from https://github.com/douglascrockford/JSON-js
"ignore": [
"ui/packages/evidently-ui-lib/src/**/JsonParser.ts",
"ui/packages/evidently-ui-lib/src/**/endpoints.d.ts",
"ui/packages/evidently-ui-lib/.tsc-dts/**"
],
"include": ["ui/*/src/**", "ui/packages/*/src/**"]
},
"formatter": {
"enabled": true,
"formatWithErrors": false,
"indentStyle": "space",
"indentWidth": 2,
"lineEnding": "lf",
"lineWidth": 100,
"attributePosition": "auto"
},
"css": {
"linter": {
"enabled": true
},
"formatter": {
"enabled": true
}
},
"javascript": {
"formatter": {
"arrowParentheses": "always",
"bracketSameLine": false,
"bracketSpacing": true,
"jsxQuoteStyle": "single",
"quoteStyle": "single",
"quoteProperties": "asNeeded",
"semicolons": "asNeeded",
"trailingCommas": "none"
}
},
"json": {
"formatter": {
"trailingCommas": "none"
}
},
"organizeImports": { "enabled": true },
"linter": {
"enabled": true,
"rules": {
"recommended": true
}
}
}
5 changes: 5 additions & 0 deletions docs/book/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@

{% hint style="info" %}
**You are looking at the old Evidently documentation**: Check the newer version [here](https://docs.evidentlyai.com/introduction).
{% endhint %}

Evidently helps evaluate, test, and monitor data and ML-powered systems.
* Predictive tasks: classification, regression, ranking, recommendations.
* Generative tasks: chatbots, RAGs, Q&A, summarization.
8 changes: 5 additions & 3 deletions docs/book/SUMMARY.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# Table of contents

* [New DOCS](https://docs.evidentlyai.com/)
* [What is Evidently?](README.md)
* [Get Started](get-started/README.md)
* [Evidently Cloud](get-started/quickstart-cloud.md)
@@ -18,7 +19,7 @@
* [Regression Performance](presets/reg-performance.md)
* [Classification Performance](presets/class-performance.md)
* [NoTargetPerformance](presets/no-target-performance.md)
* [Text Overview](presets/text-overview.md)
* [Text Evals](presets/text-overview.md)
* [Recommender System](presets/recsys.md)
* [Tutorials and Examples](examples/README.md)
* [All Tutorials](examples/examples.md)
@@ -80,11 +81,12 @@
* [Feature importance in data drift](customization/feature-importance.md)
* [Text evals with LLM-as-judge](customization/llm_as_a_judge.md)
* [Text evals with HuggingFace](customization/huggingface_descriptor.md)
* [Add a custom text descriptor](customization/add-custom-descriptor.md)
* [Add a custom drift method](customization/add-custom-drift-method.md)
* [Add a custom Metric or Test](customization/add-custom-metric-or-test.md)
* [Customize JSON output](customization/json-dict-output.md)
* [Show raw data in Reports](customization/report-data-aggregation.md)
* [Add text comments to Reports](customization/text-comments.md)
* [Add a custom drift method](customization/add-custom-drift-method.md)
* [Add a custom Metric or Test](customization/add-custom-metric-or-test.md)
* [Change color schema](customization/options-for-color-schema.md)
* [How-to guides](how-to-guides/README.md)

2 changes: 1 addition & 1 deletion docs/book/api-reference/evidently.calculations.md
Original file line number Diff line number Diff line change
@@ -277,7 +277,7 @@ Update dataset by predictions type:
set predicted_labels column by threshold
- (multy label classification) if predictions is a list and its length is greater than 2
- (multi label classification) if predictions is a list and its length is greater than 2
set predicted_labels from probability values in columns by prediction column
110 changes: 110 additions & 0 deletions docs/book/customization/add-custom-descriptor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
---
description: How to add custom text descriptors.
---

You can implement custom row-level evaluations for text data that you will later use just like any other descriptor across Metrics and Tests. You can implement descriptors that use a single column or two columns.

Note that if you want to use LLM-based evaluations, you can write custom prompts using [LLM judge templates](llm_as_a_judge.md).

# Code example

Refer to a How-to example:

{% embed url="https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/how_to_use_llm_judge_template.ipynb" %}

# Custom descriptors

Imports:

```python
from evidently.descriptors import CustomColumnEval, CustomPairColumnEval
```

## Single column descriptor

You can create a custom descriptor that will take a single column from your dataset and run a certain evaluation for each row.

**Implement your evaluation as a Python function**. It will take a pandas Series as input and return a transformed Series.

Here, the `is_empty_string_callable` function takes a column of strings and returns an "EMPTY" or "NON EMPTY" outcome for each.

```python
def is_empty_string_callable(val1):
return pd.Series(["EMPTY" if val == "" else "NON EMPTY" for val in val1], index=val1.index)
```

**Create a custom descriptor**. Create an example of `CustomColumnEval` class to wrap the evaluation logic into an object that you can later use to process specific dataset input.

```python
empty_string = CustomColumnEval(
func=is_empty_string_callable,
feature_type="cat",
display_name="Empty response"
)
```

Where:
* `func: Callable[[pd.Series], pd.Series]` is a function that returns a transformed pandas Series.
* `display_name: str` is the new descriptor's name that will appear in Reports and Test Suites.
* `feature_type` is the type of descriptor that the function returns (`cat` for categorical, `num` for numerical)

**Apply the new descriptor**. To create a Report with a new Descriptor, pass it as a `column_name` to the `ColumnSummaryMetric`. This will compute the new descriptor for all rows in the specified column and summarize its distribution:

```python
report = Report(metrics=[
ColumnSummaryMetric(column_name=empty_string.on("response")),
])
```

Run the Report on your `df` dataframe as usual:

```python
report.run(reference_data=None,
current_data=df)
```

## Double column descriptor

You can create a custom descriptor that will take two columns from your dataset and will run a certain evaluation for each row. (For example, for pairwise evaluators).

**Implement your evaluation as a Python function**. Here, the `exact_match_callable` function takes two columns and checks whether each pair of values is the same, returning "MATCH" if they are equal and "MISMATCH" if they are not.

```python
def exact_match_callable(val1, val2):
return pd.Series(["MATCH" if val else "MISMATCH" for val in val1 == val2])
```

**Create a custom descriptor**. Create an example of the `CustomPairColumnEval` class to wrap the evaluation logic into an object that you can later use to process two named columns in a dataset.

```python
exact_match = CustomPairColumnEval(
func=exact_match_callable,
first_column="response",
second_column="question",
feature_type="cat",
display_name="Exact match between response and question"
)
```

Where:

* `func: Callable[[pd.Series, pd.Series], pd.Series]` is a function that returns a transformed pandas Series after evaluating two columns.
* `first_column: str` is the name of the first column to be passed into the function.
* `second_column: str` is the name of the second column to be passed into the function.
* `display_name: str` is the new descriptor's name that will appear in Reports and Test Suites.
* `feature_type` is the type of descriptor that the function returns (`cat` for categorical, `num` for numerical).

**Apply the new descriptor**. To create a Report with a new Descriptor, pass it as a `column_name` to the ColumnSummaryMetric. This will compute the new descriptor for all rows in the dataset and summarize its distribution:

```python
report = Report(metrics=[
ColumnSummaryMetric(column_name=exact_match.as_column())
])
```

Run the Report on your `df` dataframe as usual:

```python
report.run(reference_data=None,
current_data=df)
```
Loading