feat(vertex-ai): function calling: convert markdown 'samples' to python #12811

Valeriy-Burlaka · 2024-12-02T15:21:05Z

Description

Convert Markdown 'samples' for Function Calling Guides & API Reference pages:

The entire "How to create a function calling application" section (a tutorial-style series of steps that together compile into a function-calling application)
The "Example syntax" section for the API Reference page.

Checklist

I have followed Sample Guidelines from AUTHORING_GUIDE.MD
Tests pass: nox -s py-3.9 && nox -s py-3.12
Lint pass: nox -s lint

snippet-bot · 2024-12-02T15:21:12Z

Here is the summary of changes.

You are about to add 9 region tags.

generative_ai/function_calling/example_syntax.py:26, tag generativeaionvertexai_function_calling_example_syntax
generative_ai/function_calling/function_calling_application.py:22, tag generativeaionvertexai_gemini_function_calling_app_setup
generative_ai/function_calling/function_calling_application.py:46, tag generativeaionvertexai_gemini_function_calling_app_declare_1
generative_ai/function_calling/function_calling_application.py:61, tag generativeaionvertexai_gemini_function_calling_app_declare_2
generative_ai/function_calling/function_calling_application.py:90, tag generativeaionvertexai_gemini_function_calling_app_declare_3
generative_ai/function_calling/function_calling_application.py:135, tag generativeaionvertexai_gemini_function_calling_app_prompt
generative_ai/function_calling/function_calling_application.py:145, tag generativeaionvertexai_gemini_function_calling_app_submit
generative_ai/function_calling/function_calling_application.py:169, tag generativeaionvertexai_gemini_function_calling_app_invoke
generative_ai/function_calling/function_calling_application.py:184, tag generativeaionvertexai_gemini_function_calling_app_generate

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

Valeriy-Burlaka · 2024-12-02T17:14:40Z

generative_ai/function_calling/basic_example.py

@@ -39,7 +39,7 @@ def generate_function_call() -> GenerationResponse:
    vertexai.init(project=PROJECT_ID, location="us-central1")

    # Initialize Gemini model
-    model = GenerativeModel("gemini-1.5-flash-002")
+    model = GenerativeModel("gemini-1.5-pro-002")


Gemini Flash is not working anymore with this code sample (see the log):

File "/workspace/generative_ai/function_calling/test_function_calling.py", line 31, in test_function_calling response = basic_example.generate_function_call() File "/workspace/generative_ai/function_calling/basic_example.py", line 75, in generate_function_call function_call = response.candidates[0].function_calls[0] IndexError: list index out of range

The actual response from the model is now:

candidates { content { role: "model" parts { text: "I cannot answer this question. The available function `get_current_weather` has no implementation." } } avg_logprobs: -0.17872051000595093 finish_reason: STOP }

Created a general issue to initiate the discussion about current CI process.

Thanks for reporting this, Valeriy-Burlaka. The Gemini Flash model appears to have been deprecated. The change to gemini-1.5-pro-002 in the pull request reflects the current, supported model. Your created issue is the appropriate place to track the broader CI implications.

Valeriy-Burlaka · 2024-12-02T18:54:21Z

generative_ai/function_calling/function_declaration_from_func.py

+def function_declaration_from_func():
+    # [START generativeaionvertexai_gemini_function_calling_declare_from_function]
+    # Define a function. Could be a local function or you can import the requests library to call an API
+    def multiply_numbers(numbers: List[int]) -> int:


This is almost a copy of the original function declaration.
I added the typings to fix the error 400 that was lurking in this doc sample:

... grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.INVALID_ARGUMENT details = "Unable to submit request because one or more function parameters didn't specify the schema type field. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling" debug_error_string = "UNKNOWN:Error received from peer ipv4:142.250.75.10:443 {created_time:"2024-12-02T15:20:19.676923+01:00", grpc_status:3, grpc_message:"Unable to submit request because one or more function parameters didn\'t specify the schema type field. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling"}" > ... google.api_core.exceptions.InvalidArgument: 400 Unable to submit request because one or more function parameters didn't specify the schema type field. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling

The unit-test utilizing a chat message is necessary to test correctness of the function declaration.

Valeriy-Burlaka · 2024-12-02T18:59:28Z

generative_ai/function_calling/test_function_calling.py

@@ -52,12 +55,14 @@ def test_function_calling_advanced_function_selection() -> None:
    )


+@pytest.mark.skip(reason="Blocked on b/... ")


Both samples that use OpenAI client are not working anymore (log):

File "/workspace/generative_ai/function_calling/test_function_calling.py", line 59, in test_function_calling_basic response = chat_function_calling_basic.generate_text() File "/workspace/generative_ai/function_calling/chat_function_calling_basic.py", line 39, in generate_text client = openai.OpenAI( File "/workspace/generative_ai/function_calling/.nox/py-3-9/lib/python3.9/site-packages/openai/_client.py", line 122, in __init__ super().__init__( File "/workspace/generative_ai/function_calling/.nox/py-3-9/lib/python3.9/site-packages/openai/_base_client.py", line 825, in __init__ self._client = http_client or SyncHttpxClientWrapper( File "/workspace/generative_ai/function_calling/.nox/py-3-9/lib/python3.9/site-packages/openai/_base_client.py", line 723, in __init__ super().__init__(**kwargs) TypeError: __init__() got an unexpected keyword argument 'proxies' - generated xml file: /workspace/generative_ai/function_calling/sponge_log.xml - =========================== short test summary info ============================ FAILED test_function_calling.py::test_function_calling_basic - TypeError: __i... FAILED test_function_calling.py::test_function_calling_config - TypeError: __... ========================= 2 failed, 5 passed in 14.85s =========================

Link to the doc section with OpenAI samples: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/function-calling#python-openai

Created a general issue to initiate the discussion about current CI process.

The error TypeError: __init__() got an unexpected keyword argument 'proxies' in the OpenAI client indicates a version mismatch or incompatibility with the openai library and the way it handles proxy settings. The proxies argument was likely added in a newer version. To resolve this, you should:

Check the openai library version: Ensure you're using a version compatible with your current environment. Check your requirements.txt and pyproject.toml (if used) for the specified version. If it's outdated, update it to the latest version that supports the proxies argument (or a version that doesn't require it if you're not using proxies).

Review proxy configuration: If you are using proxies, double-check your proxy settings in your environment variables or configuration files. Incorrectly formatted proxy settings can also cause this error. If you are not using proxies, remove any proxy settings from your code or environment.

Virtual environment consistency: Ensure your testing environment is consistent with your development environment. Use the same virtual environment or container for both to avoid version conflicts.

Dependency resolution: Try running pip install --upgrade pip and then pip install -r requirements.txt (or your equivalent package manager command) to ensure all dependencies are correctly resolved and updated.

The issue you've created is a good first step to track this problem. Consider adding more details about your environment (Python version, OpenAI library version, proxy configuration if applicable) to help diagnose the root cause.

Valeriy-Burlaka · 2024-12-02T19:32:56Z

generative_ai/function_calling/function_declaration_from_dict.py

+
+
+def function_declaration_from_dict() -> object:
+    # [START generativeaionvertexai_gemini_function_calling_declare_from_dict1]


@msampathkumar , I know we want only one pair of region tags per sample file, but just look at this doc section — we will have to add 8 new micro-files to convert all markdown used there 🤯 . Given that all markdown samples for "Step 1-5" sub-sections could basically compose into 1 coherent sample with different region tags, maybe we could make an exception from the "1 region tag" rule here? 🤔

Sure! We can do that. Let us catchup on this today.

@msampathkumar , please check the updated sample.
Following the idea of the "How to ..." section (creating an example app block by block), I compiled it into one code sample and added tests. As it runs together, it now lives together, in one file!

Valeriy-Burlaka · 2024-12-06T15:13:45Z

generative_ai/function_calling/test_function_calling.py

 @backoff.on_exception(backoff.expo, ResourceExhausted, max_time=10)
 def test_function_calling_basic() -> None:
    response = chat_function_calling_basic.generate_text()
    assert "get_current_weather" in response.choices[0].message.tool_calls[0].id


+@pytest.mark.skip(reason="Blocked on b/... ")


I will need help with creating a real ticket in Buganizer.

Valeriy-Burlaka · 2024-12-06T15:19:17Z

generative_ai/function_calling/test_function_calling.py

+    response = chat_session.send_message("What will be 1 multiplied by 2?")
+    assert response is not None
+
+    extract_sales_prompt = """


I am adding a test for each toolbox function created by this sample because of this reason. The toolbox is lazily evaluated, and it's not enough to simply create a function declaration to check that it's working.
Only when we trigger a "function call" response from the model with a target prompt it validates the declaration and can throw an error.

Valeriy-Burlaka · 2024-12-06T15:37:06Z

generative_ai/function_calling/function_calling_application.py

+PROJECT_ID = os.getenv("GOOGLE_CLOUD_PROJECT")
+
+
+def create_app() -> object:


Consolidates code samples for the https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling#how-works doc section into a single, testable script.

Each sub-step of this section presents a discrete code snippet (block) for the final application, demonstrating a single function calling concept at a time. This compilation creates a complete, maintainable implementation for DevRel purposes, reducing the amount of separate files we'd have if we keep each snippet in a separate file.
The original snippets and their documentation order are preserved through separate region tags, with (minor fixes) applied.

Valeriy-Burlaka · 2024-12-10T12:37:53Z

generative_ai/function_calling/example_syntax.py

+    # [START generativeaionvertexai_function_calling_example_syntax]
+    from vertexai.generative_models import FunctionDeclaration, GenerationConfig, GenerativeModel, Tool
+
+    gemini_model = GenerativeModel(


I tried to preserve the "spirit" of the original .md sample (i.e., a very short snippet that only shows how to declare a model with toolbox), while making it actually copyable, testable, and working. That's why I even moved the init call outside of the sample but added all necessary imports for proper functioning.

feat: convert markdown sample for function calling to python

ac8aaaa

Valeriy-Burlaka self-assigned this Dec 2, 2024

Valeriy-Burlaka requested review from a team as code owners December 2, 2024 15:21

Valeriy-Burlaka marked this pull request as draft December 2, 2024 15:21

product-auto-label bot added the samples Issues that are directly related to samples. label Dec 2, 2024

Valeriy-Burlaka added 2 commits December 2, 2024 16:22

fix region tag name

9134412

fix? basic example is no longer working with gemini-flash

b0788f5

Valeriy-Burlaka commented Dec 2, 2024

View reviewed changes

mark broken openai samples as skip

693f914

Valeriy-Burlaka commented Dec 2, 2024

View reviewed changes

fix test name

ade95c7

Valeriy-Burlaka commented Dec 2, 2024

View reviewed changes

Valeriy-Burlaka added 2 commits December 2, 2024 20:13

fix test name

d7f744b

fix checks; add experimental 2-tag sample

5b9e49b

Valeriy-Burlaka commented Dec 2, 2024

View reviewed changes

Valeriy-Burlaka changed the title ~~feat: convert markdown sample for function calling to python~~ feat(gen-ai): function calling: convert markdown declarations to python Dec 2, 2024

Valeriy-Burlaka added the api: vertex-ai Issues related to the Vertex AI API. label Dec 2, 2024

Valeriy-Burlaka changed the title ~~feat(gen-ai): function calling: convert markdown declarations to python~~ feat(vertex-ai): function calling: convert markdown declarations to python Dec 2, 2024

Valeriy-Burlaka changed the title ~~feat(vertex-ai): function calling: convert markdown declarations to python~~ feat(vertex-ai): function calling: convert markdown func declarations to python Dec 2, 2024

fix import order

2772747

Valeriy-Burlaka marked this pull request as ready for review December 2, 2024 19:43

Valeriy-Burlaka marked this pull request as draft December 2, 2024 19:43

fix import order

1f8c008

Valeriy-Burlaka marked this pull request as ready for review December 2, 2024 22:15

Valeriy-Burlaka marked this pull request as draft December 2, 2024 22:16

fix typo in region tag

6e34636

Valeriy-Burlaka marked this pull request as ready for review December 2, 2024 22:17

Valeriy-Burlaka marked this pull request as draft December 5, 2024 14:35

Valeriy-Burlaka added 3 commits December 6, 2024 12:27

Merge branch 'main' into vburlaka-vertexai-markdown-samples

9e73443

wip: re-work markdown 'how to create...' section into python

98df51c

update test prompt to execute the 'sales' func declaration

87aa2a4

Valeriy-Burlaka commented Dec 6, 2024

View reviewed changes

Valeriy-Burlaka marked this pull request as ready for review December 6, 2024 15:22

Valeriy-Burlaka requested a review from msampathkumar December 6, 2024 15:22

Valeriy-Burlaka commented Dec 6, 2024

View reviewed changes

Valeriy-Burlaka changed the title ~~feat(vertex-ai): function calling: convert markdown func declarations to python~~ feat(vertex-ai): function calling: convert markdown 'samples' to python Dec 10, 2024

feat(vertex-ai): function-calling api: 'example syntax'

26f725a

Valeriy-Burlaka mentioned this pull request Dec 10, 2024

feat(vertex-ai): function-calling api: convert 'example syntax' from markdown #12979

Closed

3 tasks

Valeriy-Burlaka commented Dec 10, 2024

View reviewed changes

This was referenced Dec 13, 2024

[generative_ai]: Function calling with OpenAI samples are not working anymore. #12996

Open

[generative_ai] Missed regressions in the code samples because of targeted test runs #12990

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(vertex-ai): function calling: convert markdown 'samples' to python #12811

feat(vertex-ai): function calling: convert markdown 'samples' to python #12811

Valeriy-Burlaka commented Dec 2, 2024 •

edited

Loading

snippet-bot bot commented Dec 2, 2024 •

edited

Loading

Valeriy-Burlaka Dec 2, 2024 •

edited

Loading

Valeriy-Burlaka Dec 13, 2024

code-review-assist bot Dec 13, 2024

Valeriy-Burlaka Dec 2, 2024 •

edited

Loading

Valeriy-Burlaka Dec 2, 2024 •

edited

Loading

Valeriy-Burlaka Dec 13, 2024

code-review-assist bot Dec 13, 2024

Valeriy-Burlaka Dec 2, 2024 •

edited

Loading

msampathkumar Dec 4, 2024

Valeriy-Burlaka Dec 6, 2024

Valeriy-Burlaka Dec 6, 2024

Valeriy-Burlaka Dec 6, 2024 •

edited

Loading

Valeriy-Burlaka Dec 6, 2024 •

edited

Loading

Valeriy-Burlaka Dec 10, 2024 •

edited

Loading

		@@ -52,12 +55,14 @@ def test_function_calling_advanced_function_selection() -> None:
		)


		@pytest.mark.skip(reason="Blocked on b/... ")



		def function_declaration_from_dict() -> object:
		# [START generativeaionvertexai_gemini_function_calling_declare_from_dict1]

		PROJECT_ID = os.getenv("GOOGLE_CLOUD_PROJECT")


		def create_app() -> object:

feat(vertex-ai): function calling: convert markdown 'samples' to python #12811

Are you sure you want to change the base?

feat(vertex-ai): function calling: convert markdown 'samples' to python #12811

Conversation

Valeriy-Burlaka commented Dec 2, 2024 • edited Loading

Description

Checklist

snippet-bot bot commented Dec 2, 2024 • edited Loading

Valeriy-Burlaka Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Valeriy-Burlaka Dec 13, 2024

Choose a reason for hiding this comment

code-review-assist bot Dec 13, 2024

Choose a reason for hiding this comment

Valeriy-Burlaka Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Valeriy-Burlaka Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Valeriy-Burlaka Dec 13, 2024

Choose a reason for hiding this comment

code-review-assist bot Dec 13, 2024

Choose a reason for hiding this comment

Valeriy-Burlaka Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

msampathkumar Dec 4, 2024

Choose a reason for hiding this comment

Valeriy-Burlaka Dec 6, 2024

Choose a reason for hiding this comment

Valeriy-Burlaka Dec 6, 2024

Choose a reason for hiding this comment

Valeriy-Burlaka Dec 6, 2024 • edited Loading

Choose a reason for hiding this comment

Valeriy-Burlaka Dec 6, 2024 • edited Loading

Choose a reason for hiding this comment

Valeriy-Burlaka Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Valeriy-Burlaka commented Dec 2, 2024 •

edited

Loading

snippet-bot bot commented Dec 2, 2024 •

edited

Loading

Valeriy-Burlaka Dec 2, 2024 •

edited

Loading

Valeriy-Burlaka Dec 2, 2024 •

edited

Loading

Valeriy-Burlaka Dec 2, 2024 •

edited

Loading

Valeriy-Burlaka Dec 2, 2024 •

edited

Loading

Valeriy-Burlaka Dec 6, 2024 •

edited

Loading

Valeriy-Burlaka Dec 6, 2024 •

edited

Loading

Valeriy-Burlaka Dec 10, 2024 •

edited

Loading