Limited to Azure OpenAI content filters

[yuichiromukaiyama]

Below is the request body to Azure OpenAI generated by python's FunctionCallingStepwise Planner.

When I sent a request to Azure OpenAI, it was caught by the content filter, as shown in the second example.

const body = {
  messages: [
    {
      role: "system",
      content:
        "Original request: Hello\n\nYou are in the process of helping the user fulfill this request using the following plan:\nHello! How can I assist you today? If you have a goal in mind and need a plan to achieve it, please provide me with the details, and I'll help you create a step-by-step plan using the functions available.\n\nThe user will ask you for help with each step.",
    },
    {
      role: "user",
      content:
        "Perform the next step of the plan if there is more work to do.When you have reached a final answer, use the UserInteraction-SendFinalAnswerfunction to communicate this back to the user.",
    },
  ],
  model: "gpt-4",
  frequency_penalty: 0.0,
  logit_bias: {},
  max_tokens: 256,
  n: 1,
  presence_penalty: 0.0,
  stream: false,
  temperature: 0.0,
  tool_choice: "auto",
  tools: [
    {
      type: "function",
      function: {
        name: "MathPlugin-Add",
        description: "Returns the Addition result of the values provided.",
        parameters: {
          type: "object",
          properties: {
            input: { description: "the first number to add", type: "number" },
            amount: { description: "the second number to add", type: "number" },
          },
          required: ["input", "amount"],
        },
      },
    },
    {
      type: "function",
      function: {
        name: "MathPlugin-Subtract",
        description: "Subtracts value to a value",
        parameters: {
          type: "object",
          properties: {
            input: { description: "the first number", type: "number" },
            amount: { description: "the number to subtract", type: "number" },
          },
          required: ["input", "amount"],
        },
      },
    },
    {
      type: "function",
      function: {
        name: "UserInteraction-SendFinalAnswer",
        description: "The final answer to return to the user",
        parameters: {
          type: "object",
          properties: {
            answer: { description: "The final answer", type: "string" },
          },
          required: ["answer"],
        },
      },
    },
    {
      type: "function",
      function: {
        name: "sequential_planner-create_plan",
        description: "Create a plan for the given goal",
        parameters: {
          type: "object",
          properties: {
            available_functions: {
              description:
                "A list of functions that can be used to generate the plan",
              type: "string",
            },
            goal: { description: "The goal to satisfy", type: "string" },
            name_delimiter: { description: "", type: "string" },
          },
          required: ["available_functions", "goal", "name_delimiter"],
        },
      },
    },
  ],
  top_p: 1.0,
};

{
  "error": {
    "message": "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766",
    "type": null,
    "param": "prompt",
    "code": "content_filter",
    "status": 400,
    "innererror": {
      "code": "ResponsibleAIPolicyViolation",
      "content_filter_result": {
        "hate": {
          "filtered": false,
          "severity": "safe"
        },
        "self_harm": {
          "filtered": true,
          "severity": "medium"
        },
        "sexual": {
          "filtered": false,
          "severity": "safe"
        },
        "violence": {
          "filtered": false,
          "severity": "safe"
        }
      }
    }
  }
}

Has anyone encountered a similar problem? The code for Function Calling Stepwise Planner used the example belowhttps://github.com/microsoft/semantic-kernel/blob/main/python/samples/kernel-syntax-examples/function_calling_stepwise_planner.py

[moonbox3]

Hi @yuichiromukaiyama, yes, I have been running into problems related to AzureOpenAI and random content filtering. I am working with a team right now to understand what is going on. What I noticed is that the AzureOpenAI model will either fail on a prompt that doesn't contain spaces (for example: This is a prompt.That would fail.As there is.No space between punctuation.) This is very odd, but it's what I have seen... And it's failing for the same self-harm that you show above. I will send this along to them as well, so they can have more data around it.

I have fixed this in the Python FunctionCallingStepwisePlanner -- you will see the user prompt has spaces now between the punctuation. This fix will be in the next release in the next day or so.

[yuichiromukaiyama]

Hi @moonbox3 ,Thank you so much for your reply! I'm glad to hear that this will be fixed in the next release.
I will definitely try it out as soon as v0.9.5 is released.