Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] main from Azure:main #34

Open
wants to merge 9,287 commits into
base: main
Choose a base branch
from
Open

[pull] main from Azure:main #34

wants to merge 9,287 commits into from

Conversation

pull[bot]
Copy link

@pull pull bot commented Nov 12, 2021

See Commits and Changes for more details.


Created by pull[bot]

Can you help keep this open source service alive? 💖 Please sponsor : )

@pull pull bot added the ⤵️ pull label Nov 12, 2021
@google-cla
Copy link

google-cla bot commented Nov 12, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

28 similar comments
@google-cla
Copy link

google-cla bot commented Nov 12, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 12, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 12, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 12, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 13, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 15, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 15, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 15, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 15, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 15, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 15, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 15, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 16, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 16, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 16, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 16, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 16, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 16, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 16, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 16, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 16, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 16, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 16, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 17, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 17, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 17, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 17, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 17, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

swathipil and others added 30 commits March 21, 2025 11:58
* prompt + inst fro pylint

* update

* bump

* typo

* copilot typo
… by SDK owner) (#40040)

* code and test

* update testcases

---------

Co-authored-by: azure-sdk <PythonSdkPipelines>
Co-authored-by: ChenxiJiang333 <[email protected]>
Co-authored-by: azure-sdk <PythonSdkPipelines>
… owner) (#39786)

* code and test

* update testcases

* Revert version to 10.4.0

---------

Co-authored-by: azure-sdk <PythonSdkPipelines>
Co-authored-by: ChenxiJiang333 <[email protected]>
Co-authored-by: ChenxiJiang333 <[email protected]>
* march preview test

* fix for vector_store tests
Service Fabric managed identity should not allow a user to pass in a user-assigned identity
since Service Fabric will always use the cluster-configured identity.

Signed-off-by: Paul Van Eck <[email protected]>
* Update task_query_response.prompty

remove required keys

* Update task_simulate.prompty

* Update task_query_response.prompty

* Update task_simulate.prompty

* Fix the api_key needed

* Update for release

* Black fix for file

* Add original text in global context

* Update test

* Update the indirect attack simulator

* Black suggested fixes

* Update simulator prompty

* Update adversarial scenario enum to exclude XPIA

* Update changelog

* Black fixes

* Remove duplicate import

* Fix the mypy error

* Mypy please be happy

* Updates to non adv simulator

* accept context from assistant messages, exclude them when using them for conversation

* update changelog

* pylint fixes

* pylint fixes

* remove redundant quotes

* Fix typo

* pylint fix

* Update broken tests

* Include the grounding json in the manifest

* Fix typo

* Come on package

* Release 1.0.0b5

* Notice from Chang

* Remove adv_conv template parameters from the outputs

* Update chanagelog

* Experimental tags on adv scenarios

* Readme fix onbreaking change

* Add the category and both user and assistant context to the response of qr_json_lines

* Update changelog

* Rename _kwargs to _options

* _options as prefix

* update troubleshooting for simulator

* Rename according to suggestions

* Clean up readme

* more links

* Bugfix: zip_longest created null parameters

* Updated changelog

* zip does the job

* remove ununsed import

* Fix changelog merge

* Remove print statements

* Update the threshold to be individual parameters

* Remove higher is better as a public ref

---------

Co-authored-by: Nagkumar Arkalgud <[email protected]>
Co-authored-by: Nagkumar Arkalgud <[email protected]>
Co-authored-by: Nagkumar Arkalgud <[email protected]>
…e` (#40177)

* update pipeline

* format

* update for target release date

* update

* update

* update

* Update tools/azure-sdk-tools/packaging_tools/sdk_generator.py

Co-authored-by: Copilot <[email protected]>

---------

Co-authored-by: Copilot <[email protected]>
…DK owner) (#40152)

* code and test

* update testcases

* Update release date in CHANGELOG.md

---------

Co-authored-by: azure-sdk <PythonSdkPipelines>
Co-authored-by: ChenxiJiang333 <[email protected]>
Co-authored-by: ChenxiJiang333 <[email protected]>
Co-authored-by: azure-sdk <PythonSdkPipelines>
* Fix document debug info.

* update changelog

* update

* update
* use run id instead of name for storing files

* CL

* nit

* save new recordings
A port of the bare minimum batch engine code from the Promptflow repo. The focus was on expediency, rather than elegance and some items are still stubs or don't do much currently. A follow-up PR will integrate this new code so that it can be invoked as part of an evaluate call.

Things that still need to be done:
- Handle errors properly for OpenAI requests
- Handle "target" functions being passed
- Capture token usage from OpenAI requests (though this does not currently work right now)
- Handle logging in a cleaner way
- Determine what tracing (if any) is needed and either implement that code or remove it entirely
- Determine if we still need the saving to a local folder structure, whether this should be changed to save to a single file (optionally?), or remove that code outright
Added some details about the newer `SupportsTokenInfo` protocols into
the CLIENT_LIBRARY_DEVELOPER.md file.

Signed-off-by: Paul Van Eck <[email protected]>
* identify and fix the issue with no results on a matrix that is PRBatched. The platform filters eliminate all the jobs, which is why it only turned up on the second windows leg

---------

Co-authored-by: Scott Beddall <[email protected]>
* Tool Call Accuracy Evaluator (#40068)

* Tool Call Accuracy Evaluator

* Review comments

* Updating score key and output structure

* Tool Call Accuracy Evaluator

* Review comments

* Updating score key and output structure

* Updating prompt

* Renaming parameter

* Converter from AI Service threads/runs to evaluator-compatible schema (#40047)

* WIP AIAgentConverter

* Added the v1 of the converter

* Updated the AIAgentConverter with different output schemas.

* ruff format

* Update the top schema to have: query, response, tool_definitions

* "agentic" is not a recognized word, change the wording.

* System message always comes first in query with multiple runs.

* Add support for getting inputs from local files with run_ids.

* Export AIAgentConverter through azure.ai.evaluation, local read updates

* Use from ._models import

* Ruff format again.

* For ComputeInstance and AmlCompute update disableLocalAuth property based on ssh_public_access (#39934)

* add disableLocalAuth for computeInstance

* fix disableLocalAuthAuth issue for amlCompute

* update compute instance

* update recordings

* temp changes

* Revert "temp changes"

This reverts commit 64e3c38.

* update recordings

* fix tests

* Simplify the API by rolling up the static methods and hiding internals.

* Lock the ._converters._ai_services behind an import error.

* Print to install azure-ai-projects if we can't import AIAgentConverter

* By default, include all previous runs' tool calls and results.

* Don't crash if there is no content in historical thread messages.

* Parallelize the calls to get step_details for each run_id.

* Addressing PR comments.

* Use a single underscore to hide internal static members.

---------

Co-authored-by: Prashant Dhote <[email protected]>

* Adding intent_resolution_evaluator to prp/agent_evaluators branch (#40065)

* Add intent resolution evaluator

* updated intent_resolution evaluator logic

* Remove spurious print statements

* Address reviewers feedback

* add threshold key, update result to pass/fail rather than True/False

* Add example + remove repeated fields

* Harden check_score_is_valid function

* Add Task Adherence and Completeness  (#40098)

* Agentic Evaluator - Response Completeness

* Added Change Log for Response Completeness Agentic Evaluator

* Task Adherence Agentic Evaluator

* Add Task Adherence Evaluator to changelog

* fixing contracts for Completeness and Task Adherence Evaluators

* Enhancing Contract for Task Adherence and Response Completeness Agentic Evaluator

* update completeness implementation.

* update the completeness evaluator response to include threshold comparison.

* updating the implementation for completeness.

* updating the type for completeness score.

* updating the parsing logic for llm output of completeness.

* updating the response dict for completeness.

* Adding Task adherence

* Adding Task Adherence evaluator with samples

* Delete old files

* updating the exception for completeness evaluator.

* Changing docstring

* Adding changelog

* Use _result_key

* Add admonition

---------

Co-authored-by: Shiprajain01 <[email protected]>
Co-authored-by: ShipraJain01 <[email protected]>
Co-authored-by: Chandra Sekhar Gupta Aravapalli <[email protected]>

* Adding bug bash sample and instructions (#40125)

* Adding bug bash sample and instructions

* Updating instructions

* Update instructions.md

* Adding instructions and evaluator to agent evaluation sample

* Raising error when tool call not found

* add bug bash sample notebook for response completeness evaluator. (#40139)

* add bug bash sample notebook for response completeness evaluator.

* update the notebook for completeness.

---------

Co-authored-by: Chandra Sekhar Gupta Aravapalli <[email protected]>

* Sample specific for tool call accuracy evaluator (#40135)

* Update instructions.md

* Add IntentResolution evaluator bug bash notebook (#40144)

* Add intent resolution evaluator

* updated intent_resolution evaluator logic

* Remove spurious print statements

* Address reviewers feedback

* add threshold key, update result to pass/fail rather than True/False

* Add example + remove repeated fields

* Harden check_score_is_valid function

* Sample notebook to demo intent_resolution evaluator

* Add synthetic data and section on how to test data from disk

* Update instructions.md

* Improve task adherence prompt and add sample notebook for bugbash (#40146)

* For ComputeInstance and AmlCompute update disableLocalAuth property based on ssh_public_access (#39934)

* add disableLocalAuth for computeInstance

* fix disableLocalAuthAuth issue for amlCompute

* update compute instance

* update recordings

* temp changes

* Revert "temp changes"

This reverts commit 64e3c38.

* update recordings

* fix tests

* Add resource prefix for safe secret standard alerts (#40028)

Add the prefix to identify RGs that we are creating in our TME
tenant to identify them as potentially using local auth and violating
our safe secret standards.

Co-authored-by: Wes Haggard <[email protected]>

* Add examples to task_adherence prompt. Add Task Adherence sample notebook

* Undo changes to New-TestResources.ps1

* Add sample .env file

---------

Co-authored-by: Prashant Dhote <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>

* [AIAgentConverter] Added support for converting entire threads. (#40178)

* Implemented prepare_evaluation_data

* Add support for retrieving multiple threads into the same file.

* Parallelize thread preparing across threads.

* Set the maximum number of workers in thread pools to 10.

* Users/singankit/tool call accuracy evaluator tests (#40190)

* Raising error when tool call not found

* Adding unit tests for tool call accuracy evaluator

* Updating sample

* Fixng doc strings and moving sample to a different agent sample folder

* Fixing bug with raising exception

* Fixing rebase issue

* Removing cell outputs to fix spell check erros

* Fixing failing tests

* Fixing failing test

* Spon/update evals converter (#40204)

* Tool Call Accuracy Evaluator (#40068)

* Tool Call Accuracy Evaluator

* Review comments

* Updating score key and output structure

* Tool Call Accuracy Evaluator

* Review comments

* Updating score key and output structure

* Updating prompt

* Renaming parameter

* Converter from AI Service threads/runs to evaluator-compatible schema (#40047)

* WIP AIAgentConverter

* Added the v1 of the converter

* Updated the AIAgentConverter with different output schemas.

* ruff format

* Update the top schema to have: query, response, tool_definitions

* "agentic" is not a recognized word, change the wording.

* System message always comes first in query with multiple runs.

* Add support for getting inputs from local files with run_ids.

* Export AIAgentConverter through azure.ai.evaluation, local read updates

* Use from ._models import

* Ruff format again.

* For ComputeInstance and AmlCompute update disableLocalAuth property based on ssh_public_access (#39934)

* add disableLocalAuth for computeInstance

* fix disableLocalAuthAuth issue for amlCompute

* update compute instance

* update recordings

* temp changes

* Revert "temp changes"

This reverts commit 64e3c38.

* update recordings

* fix tests

* Simplify the API by rolling up the static methods and hiding internals.

* Lock the ._converters._ai_services behind an import error.

* Print to install azure-ai-projects if we can't import AIAgentConverter

* By default, include all previous runs' tool calls and results.

* Don't crash if there is no content in historical thread messages.

* Parallelize the calls to get step_details for each run_id.

* Addressing PR comments.

* Use a single underscore to hide internal static members.

---------

Co-authored-by: Prashant Dhote <[email protected]>

* Adding intent_resolution_evaluator to prp/agent_evaluators branch (#40065)

* Add intent resolution evaluator

* updated intent_resolution evaluator logic

* Remove spurious print statements

* Address reviewers feedback

* add threshold key, update result to pass/fail rather than True/False

* Add example + remove repeated fields

* Harden check_score_is_valid function

* Add Task Adherence and Completeness  (#40098)

* Agentic Evaluator - Response Completeness

* Added Change Log for Response Completeness Agentic Evaluator

* Task Adherence Agentic Evaluator

* Add Task Adherence Evaluator to changelog

* fixing contracts for Completeness and Task Adherence Evaluators

* Enhancing Contract for Task Adherence and Response Completeness Agentic Evaluator

* update completeness implementation.

* update the completeness evaluator response to include threshold comparison.

* updating the implementation for completeness.

* updating the type for completeness score.

* updating the parsing logic for llm output of completeness.

* updating the response dict for completeness.

* Adding Task adherence

* Adding Task Adherence evaluator with samples

* Delete old files

* updating the exception for completeness evaluator.

* Changing docstring

* Adding changelog

* Use _result_key

* Add admonition

---------

Co-authored-by: Shiprajain01 <[email protected]>
Co-authored-by: ShipraJain01 <[email protected]>
Co-authored-by: Chandra Sekhar Gupta Aravapalli <[email protected]>

* Adding bug bash sample and instructions (#40125)

* Adding bug bash sample and instructions

* Updating instructions

* Update instructions.md

* Adding instructions and evaluator to agent evaluation sample

* add bug bash sample notebook for response completeness evaluator. (#40139)

* add bug bash sample notebook for response completeness evaluator.

* update the notebook for completeness.

---------

Co-authored-by: Chandra Sekhar Gupta Aravapalli <[email protected]>

* Sample specific for tool call accuracy evaluator (#40135)

* Update instructions.md

* Add IntentResolution evaluator bug bash notebook (#40144)

* Add intent resolution evaluator

* updated intent_resolution evaluator logic

* Remove spurious print statements

* Address reviewers feedback

* add threshold key, update result to pass/fail rather than True/False

* Add example + remove repeated fields

* Harden check_score_is_valid function

* Sample notebook to demo intent_resolution evaluator

* Add synthetic data and section on how to test data from disk

* Update instructions.md

* Update _tool_call_accuracy.py

* Improve task adherence prompt and add sample notebook for bugbash (#40146)

* For ComputeInstance and AmlCompute update disableLocalAuth property based on ssh_public_access (#39934)

* add disableLocalAuth for computeInstance

* fix disableLocalAuthAuth issue for amlCompute

* update compute instance

* update recordings

* temp changes

* Revert "temp changes"

This reverts commit 64e3c38.

* update recordings

* fix tests

* Add resource prefix for safe secret standard alerts (#40028)

Add the prefix to identify RGs that we are creating in our TME
tenant to identify them as potentially using local auth and violating
our safe secret standards.

Co-authored-by: Wes Haggard <[email protected]>

* Add examples to task_adherence prompt. Add Task Adherence sample notebook

* Undo changes to New-TestResources.ps1

* Add sample .env file

---------

Co-authored-by: Prashant Dhote <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>

* [AIAgentConverter] Added support for converting entire threads. (#40178)

* Implemented prepare_evaluation_data

* Add support for retrieving multiple threads into the same file.

* Parallelize thread preparing across threads.

* Set the maximum number of workers in thread pools to 10.

* Users/singankit/tool call accuracy evaluator tests (#40190)

* Raising error when tool call not found

* Adding unit tests for tool call accuracy evaluator

* Updating sample

* update output of converter for tool calls

* add built-ins

* handle file search

* remove extra files

* revert

* revert

---------

Co-authored-by: Ankit Singhal <[email protected]>
Co-authored-by: Sandy <[email protected]>
Co-authored-by: Prashant Dhote <[email protected]>
Co-authored-by: Jose Santos <[email protected]>
Co-authored-by: ghyadav <[email protected]>
Co-authored-by: Shiprajain01 <[email protected]>
Co-authored-by: ShipraJain01 <[email protected]>
Co-authored-by: Chandra Sekhar Gupta Aravapalli <[email protected]>
Co-authored-by: Ankit Singhal <[email protected]>
Co-authored-by: Chandra Sekhar Gupta <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: spon <[email protected]>

* Updating tool call accuracy to use update tool call schema

* Update response completeness evaluator based on schema (#40214)

* add seed parameter for deterministic results and update completeness return type based on schema.

* add unit tests for response completeness.

* update completeness key to response completeness.

* fixing the unit test for updated completeness key.

* updating completeness to responsecompleteness evaluator.

* clearing output in response completeness sample notebook.

* clearing output in response completeness sample notebook.

---------

Co-authored-by: Chandra Sekhar Gupta Aravapalli <[email protected]>

* Spon/update evals converter (#40215)

* Tool Call Accuracy Evaluator (#40068)

* Tool Call Accuracy Evaluator

* Review comments

* Updating score key and output structure

* Tool Call Accuracy Evaluator

* Review comments

* Updating score key and output structure

* Updating prompt

* Renaming parameter

* Converter from AI Service threads/runs to evaluator-compatible schema (#40047)

* WIP AIAgentConverter

* Added the v1 of the converter

* Updated the AIAgentConverter with different output schemas.

* ruff format

* Update the top schema to have: query, response, tool_definitions

* "agentic" is not a recognized word, change the wording.

* System message always comes first in query with multiple runs.

* Add support for getting inputs from local files with run_ids.

* Export AIAgentConverter through azure.ai.evaluation, local read updates

* Use from ._models import

* Ruff format again.

* For ComputeInstance and AmlCompute update disableLocalAuth property based on ssh_public_access (#39934)

* add disableLocalAuth for computeInstance

* fix disableLocalAuthAuth issue for amlCompute

* update compute instance

* update recordings

* temp changes

* Revert "temp changes"

This reverts commit 64e3c38.

* update recordings

* fix tests

* Simplify the API by rolling up the static methods and hiding internals.

* Lock the ._converters._ai_services behind an import error.

* Print to install azure-ai-projects if we can't import AIAgentConverter

* By default, include all previous runs' tool calls and results.

* Don't crash if there is no content in historical thread messages.

* Parallelize the calls to get step_details for each run_id.

* Addressing PR comments.

* Use a single underscore to hide internal static members.

---------

Co-authored-by: Prashant Dhote <[email protected]>

* Adding intent_resolution_evaluator to prp/agent_evaluators branch (#40065)

* Add intent resolution evaluator

* updated intent_resolution evaluator logic

* Remove spurious print statements

* Address reviewers feedback

* add threshold key, update result to pass/fail rather than True/False

* Add example + remove repeated fields

* Harden check_score_is_valid function

* Add Task Adherence and Completeness  (#40098)

* Agentic Evaluator - Response Completeness

* Added Change Log for Response Completeness Agentic Evaluator

* Task Adherence Agentic Evaluator

* Add Task Adherence Evaluator to changelog

* fixing contracts for Completeness and Task Adherence Evaluators

* Enhancing Contract for Task Adherence and Response Completeness Agentic Evaluator

* update completeness implementation.

* update the completeness evaluator response to include threshold comparison.

* updating the implementation for completeness.

* updating the type for completeness score.

* updating the parsing logic for llm output of completeness.

* updating the response dict for completeness.

* Adding Task adherence

* Adding Task Adherence evaluator with samples

* Delete old files

* updating the exception for completeness evaluator.

* Changing docstring

* Adding changelog

* Use _result_key

* Add admonition

---------

Co-authored-by: Shiprajain01 <[email protected]>
Co-authored-by: ShipraJain01 <[email protected]>
Co-authored-by: Chandra Sekhar Gupta Aravapalli <[email protected]>

* Adding bug bash sample and instructions (#40125)

* Adding bug bash sample and instructions

* Updating instructions

* Update instructions.md

* Adding instructions and evaluator to agent evaluation sample

* add bug bash sample notebook for response completeness evaluator. (#40139)

* add bug bash sample notebook for response completeness evaluator.

* update the notebook for completeness.

---------

Co-authored-by: Chandra Sekhar Gupta Aravapalli <[email protected]>

* Sample specific for tool call accuracy evaluator (#40135)

* Update instructions.md

* Add IntentResolution evaluator bug bash notebook (#40144)

* Add intent resolution evaluator

* updated intent_resolution evaluator logic

* Remove spurious print statements

* Address reviewers feedback

* add threshold key, update result to pass/fail rather than True/False

* Add example + remove repeated fields

* Harden check_score_is_valid function

* Sample notebook to demo intent_resolution evaluator

* Add synthetic data and section on how to test data from disk

* Update instructions.md

* Update _tool_call_accuracy.py

* Improve task adherence prompt and add sample notebook for bugbash (#40146)

* For ComputeInstance and AmlCompute update disableLocalAuth property based on ssh_public_access (#39934)

* add disableLocalAuth for computeInstance

* fix disableLocalAuthAuth issue for amlCompute

* update compute instance

* update recordings

* temp changes

* Revert "temp changes"

This reverts commit 64e3c38.

* update recordings

* fix tests

* Add resource prefix for safe secret standard alerts (#40028)

Add the prefix to identify RGs that we are creating in our TME
tenant to identify them as potentially using local auth and violating
our safe secret standards.

Co-authored-by: Wes Haggard <[email protected]>

* Add examples to task_adherence prompt. Add Task Adherence sample notebook

* Undo changes to New-TestResources.ps1

* Add sample .env file

---------

Co-authored-by: Prashant Dhote <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>

* [AIAgentConverter] Added support for converting entire threads. (#40178)

* Implemented prepare_evaluation_data

* Add support for retrieving multiple threads into the same file.

* Parallelize thread preparing across threads.

* Set the maximum number of workers in thread pools to 10.

* Users/singankit/tool call accuracy evaluator tests (#40190)

* Raising error when tool call not found

* Adding unit tests for tool call accuracy evaluator

* Updating sample

* update output of converter for tool calls

* add built-ins

* handle file search

* remove extra files

* revert

* revert

* fix built-in tool parsing bug

* remove local debug

* Formatted and updated the converter to avoid built-in tool crashes.

* Added an experimental decorator to AIAgentConverter

* Update import path for experimental decorator

---------

Co-authored-by: Ankit Singhal <[email protected]>
Co-authored-by: Sandy <[email protected]>
Co-authored-by: Prashant Dhote <[email protected]>
Co-authored-by: Jose Santos <[email protected]>
Co-authored-by: ghyadav <[email protected]>
Co-authored-by: Shiprajain01 <[email protected]>
Co-authored-by: ShipraJain01 <[email protected]>
Co-authored-by: Chandra Sekhar Gupta Aravapalli <[email protected]>
Co-authored-by: Ankit Singhal <[email protected]>
Co-authored-by: Chandra Sekhar Gupta <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: spon <[email protected]>
Co-authored-by: Sandy Urazayev <[email protected]>

* Adding instructions file

* Updating default scores

* Fix test case for non-agent tool call

---------

Co-authored-by: Sandy <[email protected]>
Co-authored-by: Prashant Dhote <[email protected]>
Co-authored-by: Jose Santos <[email protected]>
Co-authored-by: ghyadav <[email protected]>
Co-authored-by: Shiprajain01 <[email protected]>
Co-authored-by: ShipraJain01 <[email protected]>
Co-authored-by: Chandra Sekhar Gupta Aravapalli <[email protected]>
Co-authored-by: Chandra Sekhar Gupta <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: stevepon <[email protected]>
Co-authored-by: spon <[email protected]>
Co-authored-by: Sandy Urazayev <[email protected]>
* added crud_container and crud_database

* got rid of duplicate tests

* split up test_crud and test_crud_async

---------

Co-authored-by: Andrew Mathew <[email protected]>
… by SDK owner) (#40089)

* code and test

* update testcases

---------

Co-authored-by: azure-sdk <PythonSdkPipelines>
Co-authored-by: ChenxiJiang333 <[email protected]>
#40039)

* code and test

* Update changelog for version 4.0.0b21

---------

Co-authored-by: azure-sdk <PythonSdkPipelines>
Co-authored-by: ChenxiJiang333 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.