Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add example types to manifest #68

Open
wants to merge 1 commit into
base: development
Choose a base branch
from

Conversation

shiv810
Copy link

@shiv810 shiv810 commented Feb 10, 2025

Resolves #242

  • Adds Example type to the manifest type

@0x4007
Copy link
Member

0x4007 commented Feb 10, 2025

Any QA we can demo with?

Comment on lines +6 to +10
export const exampleCommandExecutionSchema = T.Object({
commandInvokation: T.String({ minLength: 1 }),
parameters: T.Optional(T.Record(T.String(), T.Any())),
expectedOutput: T.String({ minLength: 1 }),
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spelling mistake commandInvocation

is commandInvocation supposed to be the comment? example: @UbiquityOS can you change my address to 0x00
Parameters are the expected way to infer parameters from the comment, right?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{
  "examples": [
    {
      "commandInvocation": "@UbiquityOS what is my wallet address?",
      "parameters": {
        "repositoryOwner": "repoOwnerUsername",
        "repositoryName": "example-repo",
        "issueNumber": 42,
        "author": "user1"
      },
      "expectedOutput": "{function: 'command-query', parameters: {\"username\": \"user1\"}}"
    }
  ]
}

Yes,commandInvocation is supposed to be a commend, the parameters, are values that are given to the LLM, and the expectedOutput is what the LLM should output, in this case it would be a tool call.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I thought the expectedOutput was supposed to be the output from the plugin, and parameters are the tool call generated by the LLM.
Maybe we could change the name from parameters to issueContext, githubContext or just context, and expectedOutput to expectedToolCall, what do you think?

Is it important for expectedOutput to be a string instead of an object? Because it's easier to write an object than to deal with escape characters and it's more readable.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I thought the expectedOutput was supposed to be the output from the plugin, and parameters are the tool call generated by the LLM. Maybe we could change the name from parameters to issueContext, githubContext or just context, and expectedOutput to expectedToolCall, what do you think?

That sounds good. These terms would function more as guiding examples, which could be included in the prompt to assist with generating the tool calls.

Is it important for expectedOutput to be a string instead of an object? Because it's easier to write an object than to deal with escape characters and it's more readable.

No, it can definitely be an object instead

Copy link
Member

@0x4007 0x4007 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We follow the manifest standard so any property name that does not conform should be prefixed with ubiquity:

@shiv810
Copy link
Author

shiv810 commented Feb 11, 2025

Any QA we can demo with?

I am not sure how to QA this? Should I make some sample manifests?

@0x4007
Copy link
Member

0x4007 commented Feb 11, 2025

It would be ideal if you set up a kernel and we could interact with it with this upgraded behavior. Have you done this before?

@whilefoo
Copy link
Member

whilefoo commented Feb 11, 2025

We follow the manifest standard so any property name that does not conform should be prefixed with ubiquity:

We had a discussion with @gentlementlegen where we both agreed that following the standard so strictly is unnecessary for our use case as these are not browser extensions or PWAs. We are already breaking the standard because skipBotEvents, configuration and commands don't have the prefix.

@0x4007
Copy link
Member

0x4007 commented Feb 11, 2025

We follow the manifest standard so any property name that does not conform should be prefixed with ubiquity:

We had a discussion with @gentlementlegen where we both agreed that following the standard so strictly is unnecessary for our use case as these are not browser extensions or PWAs. We are already breaking the standard because skipBotEvents, configuration and commands don't have the prefix.

Its not hard to make the prefix. I don't see why not conforming to standards was the conclusion you guys came to, its not right.

@gentlementlegen
Copy link
Member

Because we can but I do not really see the point as we can conform to our own standards, we are not making a manifest for a web app, it just makes names in the manifest longer for no reason so far.

@0x4007
Copy link
Member

0x4007 commented Feb 12, 2025

Following this logic there's little reason to follow any standards. It doesn't make sense why you decide this to be an exception especially if the trade off is fixable with a find and replace.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve the tool calling
4 participants