Issue with Magentic-One - Termination

[blockchainmiles]

Can anyone please state what the final correct model was here?:

Discussed in #4237

^{Originally posted by aantonellims November 16, 2024}
Hello team,

I'm currently testing the Magentic-One framework, but facing a weird behavior + an issue at the end as you can see below in the logs :

First, I can see that there is a warning message on the gpt4o model : "UserWarning: Resolved model mismatch: gpt-4o-2024-08-06 != gpt-4o-2024-05-13. Model mapping may be incorrect."

I've tried several setup with my environment such as

$env:CHAT_COMPLETION_KWARGS_JSON='{
>>     "api_version": "2024-10-01-preview",
>>     "azure_endpoint": "https://xxxx.openai.azure.com/",
>>     "model_capabilities": {
>>       "function_calling": true,
>>       "json_output": true,
>>       "vision": true
>>     },
>>     "azure_ad_token_provider": "DEFAULT",
>>     "model": "gpt-4o-2024-05-13" 
>>   }'

in the model name, I've tested different model name here : gpt-4o as it's my initial deployment name, gpt-4o-2024-08-06 and I also deployed a model in 2024-05-13 of gpt-4o named gpt-4o-2024-05-13)

Secondly, after receiving the output of my question, the Orchestrator throw an error as you can see below,
is there something I misconfigured ? or is it a bug ?

below the logs :

(autogenv4) PS C:\Users\aantonelli\source\repos\autogen_v04\autogen\python\packages\autogen-magentic-one>     python examples/example.py --logs_dir C:\Users\aantonelli\source\repos\autogen_v04\autogen\python\packages\autogen-magentic-one\examples\my_logs --hil_mode --save_screenshots
C:\Users\aantonelli\source\repos\autogenv4\Lib\site-packages\pydub\utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work
  warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)
[2024-11-16T18:23:48.222181], WebSurfer: Screenshot: screenshot_1731777827.png
User input ('exit' to quit):  what day are we and what's the current weather in paris today ? 

---------------------------------------------------------------------------
[2024-11-16T18:24:20.419974], UserProxy:

what day are we and what's the current weather in paris today ?
C:\Users\aantonelli\source\repos\autogen_v04\autogen\python\packages\autogen-magentic-one\src\autogen_magentic_one\agents\orchestrator.py:153: UserWarning: Resolved model mismatch: gpt-4o-2024-08-06 != gpt-4o-2024-05-13. Model mapping may be incorrect.
  response = await self._model_client.create(
C:\Users\aantonelli\source\repos\autogen_v04\autogen\python\packages\autogen-magentic-one\src\autogen_magentic_one\agents\orchestrator.py:166: UserWarning: Resolved model mismatch: gpt-4o-2024-08-06 != gpt-4o-2024-05-13. Model mapping may be incorrect.
  response = await self._model_client.create(

---------------------------------------------------------------------------
[2024-11-16T18:24:26.355872], Orchestrator (thought):

Initial plan:

We are working to address the following user request:

what day are we and what's the current weather in paris today ?


To answer this request we have assembled the following team:

WebSurfer: A helpful assistant with access to a web browser. Ask them to perform web searches, open pages, and interact with content (e.g., clicking links, scrolling the viewport, etc., filling in form fields, etc.) It can also summarize the entire page, or answer questions based on the content of the page. It can also be asked to sleep and wait for pages to load, in cases where the pages seem to be taking a while to load.
Coder: A helpful and general-purpose AI assistant that has strong language skills, Python skills, and Linux command line skills.
Executor: A agent for executing code
file_surfer: An agent that can handle local files.
UserProxy: The current user interacting with you.



Here is an initial fact sheet to consider:

1. GIVEN OR VERIFIED FACTS
   - None.

2. FACTS TO LOOK UP
   - The current date.
   - The current weather in Paris today.

3. FACTS TO DERIVE
   - None, all required information needs to be looked up.

4. EDUCATED GUESSES
   - None, as the requested information should be looked up for accuracy.


Here is the plan to follow as best as possible:

- Have WebSurfer look up the current date and the current weather in Paris today.
- Summarize the findings and report back.

This plan should be sufficient to address the original request accurately and efficiently.

C:\Users\aantonelli\source\repos\autogen_v04\autogen\python\packages\autogen-magentic-one\src\autogen_magentic_one\agents\orchestrator.py:217: UserWarning: Resolved model mismatch: gpt-4o-2024-08-06 != gpt-4o-2024-05-13. Model mapping may be incorrect.
  ledger_response = await self._model_client.create(

---------------------------------------------------------------------------
[2024-11-16T18:24:27.917624], Orchestrator (thought):

Updated Ledger:
{
  "is_request_satisfied": {
    "reason": "The user request has not yet been addressed.",
    "answer": false
  },
  "is_in_loop": {
    "reason": "We are not in a loop; this is a new request.",
    "answer": false
  },
  "is_progress_being_made": {
    "reason": "We are just starting to gather information.",
    "answer": true
  },
  "next_speaker": {
    "reason": "WebSurfer can find the current date and the weather in Paris.",
    "answer": "WebSurfer"
  },
  "instruction_or_question": {
    "reason": "WebSurfer can retrieve real-time data from the internet.",
    "answer": "Please look up the current date as well as the current weather in Paris today."
  }
}

---------------------------------------------------------------------------
[2024-11-16T18:24:27.940181], Orchestrator (-> WebSurfer):

Please look up the current date as well as the current weather in Paris today.

---------------------------------------------------------------------------
[2024-11-16T18:24:27.944983], Orchestrator (thought):

Next speaker WebSurfer
[2024-11-16T18:24:29.432745], WebSurfer: Screenshot: screenshot_som_1731777869.png
C:\Users\aantonelli\source\repos\autogen_v04\autogen\python\packages\autogen-magentic-one\src\autogen_magentic_one\agents\multimodal_web_surfer\multimodal_web_surfer.py:582: UserWarning: Resolved model mismatch: gpt-4o-2024-08-06 != gpt-4o-2024-05-13. Model mapping may be incorrect.
  response = await self._model_client.create(
[2024-11-16T18:24:35.392637], WebSurfer: input_text( {"reasoning": "Start by entering the search query to get current date and weather in Paris.", "input_field_id": 63, "text_value": "current date and weather in Paris today"} )
[2024-11-16T18:24:40.357951], WebSurfer: Screenshot: screenshot_1731777880.png
C:\Users\aantonelli\source\repos\autogen_v04\autogen\python\packages\autogen-magentic-one\src\autogen_magentic_one\agents\multimodal_web_surfer\multimodal_web_surfer.py:882: UserWarning: Resolved model mismatch: gpt-4o-2024-08-06 != gpt-4o-2024-05-13. Model mapping may be incorrect.
  response = await self._model_client.create(messages, cancellation_token=cancellation_token)

---------------------------------------------------------------------------
[2024-11-16T18:24:49.994837], WebSurfer:

I typed 'current date and weather in Paris today' into '0 caractères sur 2000'.

Here is a screenshot of [current date and weather in Paris today - Recherche](https://www.bing.com/search?q=current+date+and+weather+in+Paris+today&form=QBLH&sp=-1&lq=0&pq=&sc=0-0&qs=n&sk=&cvid=E7D24A2782794F76836BEAB4A6643E8C&ghsh=0&ghacc=0&ghpl=). The viewport shows 26% of the webpage, and is positioned at the top of the page.
The following metadata was extracted from the webpage:

{
    "meta_tags": {
        "referrer": "origin-when-cross-origin",
        "og:description": "La recherche intelligente de Bing facilite la recherche rapide et vous r\u00e9compense.",   
        "og:site_name": "Bing",
        "og:title": "current date and weather in Paris today - Bing",
        "og:url": "https://www.bing.com/search?q=current+date+and+weather+in+Paris+today&form=QBLH&sp=-1&lq=0&pq=&sc=0-0&qs=n&sk=&cvid=E7D24A2782794F76836BEAB4A6643E8C&ghsh=0&ghacc=0&ghpl=",
        "fb:app_id": "3732605936979161",
        "og:image": "http://www.bing.com/sa/simg/facebook_sharing_5.png",
        "og:type": "website",
        "og:image:width": "600",
        "og:image:height": "315"
    }
}

Automatic OCR of the page screenshot has detected the following text:

Sure, here is the transcribed text from the page:

**Search Bar:**
- current date and weather in Paris today

**Top Menu:**
- Microsoft Bing
- RECHERCHER
- COPILOT
- ACTUALITÉS
- IMAGES
- VIDÉOS
- CARTES
- SHOPPING
- PLUS
- À PROPOS DES RÉSULTATS DE RECHERCHE
- OUTILS
- Connexion
- 3
- Mobile

**Left Sidebar:**
- Indice UV
- Aucune prévision
- Légère brise
- 5 km/h, S
- Lever du soleil
- 08:02
- Coucher du soleil
- 17:08

**Main Content (Left Panel):**
- Environ 855 000 résultats
- Paris, Ville de Paris
- Mis à jour il y a quelques minutes
- 6°C
- 10°
- 4°
- Vent: 5 km/h
- Humidité: 74%
- Nuageux
- sam. 16, 18:24
- 19:00
- 22:00
- 1:00
- 4:00
- 7:00
- 10:00
- 13:00
- 16:00
- sam. 16
- dim. 17
- lun. 18
- mar. 19
- mer. 20
- jeu. 21
- ven. 22
- sam. 23
- dim. 24
- 10°
- 4°
- 11°
- 7°
- 12°
- 7°
- 12°
- 6°
- 12°
- 7°
- 12°
- 8°
- 12°
- 8°
- 11°
- 7°

**Right Panel Navigation Menu:**
- En savoir plus sur la météo pour Paris
  - MSN
  - The Weather Channel
  - Wunderground
  - Accuweather

**Right Panel:**
- Lancer la carte météorologique MSN
- Données météo historiques et moyennes
- Température
- Précipitations
- Neige
- MOY. MINIMALES
- MOY. MAXIMALES
- MIN. ENREG.
- MAX. ENREG.
- NOV.
- 6°
- 12°
- -7°(1998)
- 22°(2015)

**Footer:**
- Microsoft Bing
- Microsoft et ses fournisseurs tiers utilisent des cookies et des technologies similaires pour fournir, mettre à jour et améliorer ses services et annonces. Si vous l’acceptez, nous utiliserons ces données pour personnaliser les annonces et procéder à des analyses dans ce domaine.
- Vous pouvez sélectionner « Accepter » pour accepter ces utilisations, « Refuser » pour refuser ces utilisations ou cliquer sur « Plus d’options » pour passer en revue vos options. Vous pouvez modifier votre sélection sous « Gérer les Préférences de cookie » en bas de cette page. Déclaration de confidentialité
  - Accepter
  - Refuser
  - Plus d’options
<Image>

---------------------------------------------------------------------------
[2024-11-16T18:24:54.288162], Orchestrator (thought):

Updated Ledger:
{
  "is_request_satisfied": {
    "reason": "The current date and the weather details for Paris today have been found in the provided webpage screenshot.",
    "answer": true
  },
  "is_in_loop": {
    "reason": "We have successfully retrieved the required information.",
    "answer": false
  },
  "is_progress_being_made": {
    "reason": "The request has been fully addressed with the current date and weather information for Paris.",
    "answer": true
  },
  "next_speaker": {
    "reason": "The findings from the retrieved information should be summarized and communicated to the user.",        
    "answer": "UserProxy"
  },
  "instruction_or_question": {
    "reason": "The WebSurfer has found the necessary information, and we should now present it to the user to complete the request.",
    "answer": "The current date is November 16th and the weather in Paris is cloudy with a temperature of 6\u00b0C, a light breeze from the south at 5 km/h, and a 74% humidity level. Is there anything else you need assistance with?"      
  }
}

---------------------------------------------------------------------------
[2024-11-16T18:24:54.294797], Orchestrator (thought):

Request satisfied.
C:\Users\aantonelli\source\repos\autogen_v04\autogen\python\packages\autogen-magentic-one\src\autogen_magentic_one\agents\orchestrator.py:270: UserWarning: Resolved model mismatch: gpt-4o-2024-08-06 != gpt-4o-2024-05-13. Model mapping may be incorrect.
  response = await self._model_client.create(

---------------------------------------------------------------------------
[2024-11-16T18:24:56.907800], Orchestrator (final answer):


Today is Saturday, November 16. The current weather in Paris is 6°C (approximately 43°F) with light cloud cover. The wind is blowing at 5 km/h from the south, and the humidity is at 74%.

---------------------------------------------------------------------------
[2024-11-16T18:24:56.908801], Orchestrator (termination condition):

No agent selected.
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000020CA87D60C0>
Traceback (most recent call last):
  File "C:\Python311\Lib\asyncio\proactor_events.py", line 116, in __del__
    _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\proactor_events.py", line 80, in __repr__
    info.append(f'fd={self._sock.fileno()}')
                      ^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\windows_utils.py", line 102, in fileno
    raise ValueError("I/O operation on closed pipe")
ValueError: I/O operation on closed pipe
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x0000020CA87D49A0>
Traceback (most recent call last):
  File "C:\Python311\Lib\asyncio\base_subprocess.py", line 125, in __del__
    _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\base_subprocess.py", line 70, in __repr__
    info.append(f'stdin={stdin.pipe}')
                ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\proactor_events.py", line 80, in __repr__
    info.append(f'fd={self._sock.fileno()}')
                      ^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\windows_utils.py", line 102, in fileno
    raise ValueError("I/O operation on closed pipe")
ValueError: I/O operation on closed pipe
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000020CA87D60C0>
Traceback (most recent call last):
  File "C:\Python311\Lib\asyncio\proactor_events.py", line 116, in __del__
    _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\proactor_events.py", line 80, in __repr__
    info.append(f'fd={self._sock.fileno()}')
                      ^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\windows_utils.py", line 102, in fileno
    raise ValueError("I/O operation on closed pipe")
ValueError: I/O operation on closed pipe 

Thanks in advance for your help,

[skye0402]

Same issue with the tutorial program you published. I tried it on Windows 11 with Python 3.11.9 and 3.12.8 - same result. Autogen is all on 0.4.3

Running the example in WSL works if I use
xvfb-run --server-args="-screen 0 1280x960x24" python magentic01-log.py

Yes, you can manage this with Enable Cloud Integration Content in SAP Process Integration on SAP PI or PO - On-premises.
SAP Cloud-Platform Help
SAP PO Help - Cloud Integration Content
Perspective:
Architecture - Cloud / On-Premises and Hybrid:
<image>
---------- MagenticOneOrchestrator ----------
Yes, SAP Integration Suite supports on-premises deployment primarily through a hybrid integration model. According to an SAP Community blog post, you can deploy integration scenarios from SAP Cloud Platform Integration (SAP CPI) onto SAP Process Integration/Process Orchestration (SAP PI/PO) when operating in an on-premises environment. This means that while SAP Integration Suite is largely cloud-based, it offers integration capabilities that allow it to work seamlessly with on-premises SAP systems, forming a hybrid architecture. This approach enables you to leverage cloud-based integration scenarios while still maintaining components on-premises.
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000001B843D2BF60>
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\proactor_events.py", line 116, in __del__
    _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\proactor_events.py", line 80, in __repr__
    info.append(f'fd={self._sock.fileno()}')
                      ^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\windows_utils.py", line 102, in fileno
    raise ValueError("I/O operation on closed pipe")
ValueError: I/O operation on closed pipe
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x000001B843D2A840>
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_subprocess.py", line 125, in __del__
    _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_subprocess.py", line 70, in __repr__
    info.append(f'stdin={stdin.pipe}')
                ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\proactor_events.py", line 80, in __repr__
    info.append(f'fd={self._sock.fileno()}')
                      ^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\windows_utils.py", line 102, in fileno
    raise ValueError("I/O operation on closed pipe")
ValueError: I/O operation on closed pipe
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000001B843D2BF60>
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\proactor_events.py", line 116, in __del__
    _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\proactor_events.py", line 80, in __repr__
    info.append(f'fd={self._sock.fileno()}')
                      ^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\windows_utils.py", line 102, in fileno
    raise ValueError("I/O operation on closed pipe")
ValueError: I/O operation on closed pipe

Any hints?