Use "evaluation app" when the login is required in the "chat app"

[biancat821]

This issue is for a:

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [X] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

I would like to use the Evaluation app described in
https://learn.microsoft.com/en-us/azure/developer/python/get-started-app-chat-evaluations?tabs=github-codespaces

where the chat app has the optional login enabled, setting
azd env set AZURE_USE_AUTHENTICATION true

Which settings do I have to change to allow the Evaluations app accessing the chat app if the login is required?

Any log messages given by the failure

When I run the evaluation script with the Evaluations app as follows
python3 -m scripts evaluate --config=my_config.json --numquestions=14

I amn returned with the following error

[09:03:01] INFO     Running evaluation from config /workspaces/ai-rag-chat-evaluator/my_config.json                                                  evaluate.py:195
           INFO     Replaced prompt_template in config with contents of my_input/prompt_ignoresources.txt                                            evaluate.py:189
           INFO     Using Azure OpenAI Service with Azure Developer CLI Credential                                                               service_setup.py:18
[09:03:02] INFO     Running evaluation using data from /workspaces/ai-rag-chat-evaluator/my_input/qa.jsonl                                            evaluate.py:80
           INFO     Limiting evaluation to 14 questions                                                                                               evaluate.py:83
           INFO     Sending a test question to the target to ensure it is running...                                                                  evaluate.py:86
           ERROR    Failed to send a test question to the target due to error:                                                                        evaluate.py:98
                    Response from target https://app-backend-XXXXXXXXXXX.azurewebsites.net/chat is not valid JSON:                                                
                  Make sure that your configuration points at a chat endpoint that returns JSON.                                                                  
                                                                                                                                                              
           ERROR    Evaluation was terminated early due to an error ⬆  

Expected/desired behavior

The Evaluations app should work as when the optional login is disabled.

OS and Version?

I am using Windows 11 but everything is deployed on Azure.

azd version?

azd version 1.8.2 (commit 14600c7a54edac4f54397413f8638431f5c16327)

[biancat821]

I managed to solve this issue by adding a new method called get_token in the "evaluate" function that retreive a token that allows the access to the chat, with login enabled.

To use this new method you need:

• tenant_id: str,
• client_id: str,
• client_secret: str,

For client_id and client_secret check
https://learn.microsoft.com/en-us/answers/questions/834401/hi-i-want-my-client-id-and-client-secret-key

The client_secret key is visible only when it is created therefore it is likely that you need to add a new one, check
https://techcommunity.microsoft.com/t5/microsoft-defender-for-identity/app-secret-application-secret-azure-ad-azure-ad-app-secrets/m-p/3775325

The get_token method to retrieve the token

def get_token(tenant_id:str, client_id: str, client_secret: str):
    
    token_url = "https://login.microsoftonline.com/{tenant_id}/oauth2/token"

    data = {
        'grant_type': 'client_credentials',
        'client_id': client_id,
        'client_secret': client_secret,
        'resource': "api://" + client_id
    }

    # Make a POST request to the token endpoint
    response = requests.post(token_url.format(tenant_id=tenant_id), data=data)

    # Check if the request was successful
    if response.status_code == 200:
        # Extract the access token from the response
        access_token = response.json()['access_token']
        return access_token
    else:
        return -1

In the run_evaluation function before calling the send_question_to_target you have to add

...
logger.info("Getting token...")    
    try:
        token = get_token(tenant_id, client_id, client_secret)
    except Exception as e:
        logger.error("Failed to get token: \n%s", e)
        return False
...
...

Then you have to modify the two calls to the function send_question_to_target adding the retrieved token

target_data = send_question_to_target(
            "What information is in your knowledge base?", "So much", target_url, token, target_parameters, raise_error=True
        )

Finally you have to modify the send_question_to_target as follows:

def send_question_to_target(question: str, truth: str, target_url: str, access_token: str, parameters: dict = {}, raise_error=False):
    headers = {"Content-Type": "application/json",
               "Authorization": "Bearer " + access_token  # Include the access token in the Authorization header
               }
   ...
   ...

Here below you may find a script that test the retrieval of the token.
You have just to modify the tenand_id, client_id, client_secret.

import requests

# Your Azure AD tenant ID
tenant_id = your_tenant_id

# Your Azure AD application (client) ID
client_id = your_client_id

# Your Azure AD application (client) secret
client_secret = your_client_secret

# Azure AD OAuth 2.0 token endpoint
token_url = "https://login.microsoftonline.com/{tenant_id}/oauth2/token"

# Request body parameters
data = {
    'grant_type': 'client_credentials',
    'client_id': client_id,
    'client_secret': client_secret,
    'resource': "api://" + client_id
}

# Make a POST request to the token endpoint
response = requests.post(token_url.format(tenant_id=tenant_id), data=data)

# Check if the request was successful
if response.status_code == 200:
    # Extract the access token from the response
    access_token = response.json()['access_token']
    print("Access Token:", access_token)
else:
    print("Failed to retrieve access token. Status code:", response.status_code)

[pamelafox]

Thanks so much for sharing your approach! I may try to add that to the repo if more folks run into it. Very helpful!

[DuboisABB]

Thanks for the detailed instructions @biancat821. However, this doesn't work for me. I'm a bit confused by the client ID and secret. Two applications are actually created by the bicep files, the client app and server app. My understanding is that the server app is the one exposing an API. So which one should be used for client_id and client_secret in the code? I'm assuming that it's the server app. However, this doesn't work for me. I'm getting this error:

IDX10214: Audience validation failed.
Audiences: 'd9d64278-xxx-0b06169a315e'. Did not match:
validationParameters.ValidAudience: 'a76a1345-xxx-b39d2a46a015'
or validationParameters.ValidAudiences:
'api://d9d64278-xxx-0b06169a315e'."}

My env variables are as follows:
AZURE_CLIENT_APP_ID="a76a1345-xxx-b39d2a46a015"
AZURE_SERVER_APP_ID="d9d64278-xxx-0b06169a315e"

If I use the client id and secret instead of server, I get this (I configured "expose an API" for the client app to get this far)

Audiences: 'api://a76a1345-xxx-b39d2a46a015'. Did not match:
validationParameters.ValidAudience: 'a76a1345-xxx-b39d2a46a015'
or validationParameters.ValidAudiences:
'api://d9d64278-xxx-0b06169a315e'."}

So none of the options work. I'm a bit lost. This is related to #1935 as well.

[DuboisABB]

I found some relevant info about the api prefix which differs between version 1 and version 2, as indicated by accessTokenAcceptedVersion in the Manifest:
https://stackoverflow.com/questions/69747025/azure-ad-audience-is-invalid-due-to-api-being-appended

I changed accessTokenAcceptedVersion to null (in the server app registration) and it started "working" for me. By working I mean that it got further in the code, I'm not getting the Audiences error anymore. But the JSON response from the web site seems empty.