PR: evaluate code review comments with OpenAI

[EresDev]

Resolves #45
Depends on #55
QA: EresDevOrg/ubiquibot-issues#15 (comment)

See exactly which type of review comments are being score by this PR. #45 (comment)

In the prompt,we ask the OpenAI to score review comments, which is not exactly the relevance score like issue comments. For review code comments it is score to improve the offered solution and the improvement in code quality. This made more sense to me. Because of this, the openai is strict in scoring here. Only code review comments with good details get a good score here. OpenAI gets issue specs, part of code change suggested, and the comment itself. It doesn't have more info. This can be improved in the future by including a single review entire conversation, or maybe the entire code file itself. However, it is a lot more work. What we have here I think is a good first try.

Edit:

Please note that OpenAI isn't that much strict in evaluating the code as I said above. It was mistake and it is fixed now. I think it is much better now. latest QA: EresDevOrg/ubiquibot-issues#15 (comment)

[gentlementlegen]

@EresDev Could you resolve conflicts so we can get this in please?

[EresDev]

Could you resolve conflicts so we can get this in please?

@gentlementlegen
I have resolved the conflicts. Here is the latest QA.

[0x4007]

Could you resolve conflicts so we can get this in please?

@gentlementlegen

I have resolved the conflicts. Here is the latest QA.

Honestly it's a bit difficult to understand results in detail especially from mobile but if you think that it looks as expected we can merge.

I'm a bit confused why many are scored zero

[gentlementlegen]

I'd be in favor to replace the - for relevances by 0 because to me it feels like no relevance was evaluated at all. I will test this on some previously closed tasks.

[0x4007]

Interesting point. If relevance evaluation was skipped due to the config, then - makes sense. If it evaluated to 0 then we should write 0. Let's do this.

[gentlementlegen]

@EresDev I gave it a try here and got a relevance of 1 for all pull-request commands. Perhaps the configuration should be changed as well?

[EresDev]

@EresDev I gave it a try here and got a relevance of 1 for all pull-request commands. Perhaps the configuration should be changed as well?

It appears to be running at some old commit and the latest code of this PR isn't being used.

[0x4007]

@EresDev I gave it a try here and got a relevance of 1 for all pull-request commands. Perhaps the configuration should be changed as well?

It appears to be running at some old commit and the latest code of this PR isn't being used.

@gentlementlegen change the package version to * to waste less time on housekeeping

[EresDev]

I'm a bit confused why many are scored zero

They appear very good to me. They are mostly comments I just wrote when I made a change to the bot and wanted to see its result. For example "updated config". Now this comment has no relevance to the original issue specifications and its relevance was scored 0 by OpenAI. I just wrote it in there to keep track of the QA.

[0x4007]

It seems fine although to be honest sometimes it's a bit difficult to tell what's going on from our results table, especially from mobile.

[gentlementlegen]

@EresDev I gave it a try here and got a relevance of 1 for all pull-request commands. Perhaps the configuration should be changed as well?

It appears to be running at some old commit and the latest code of this PR isn't being used.

@gentlementlegen change the package version to * to waste less time on housekeeping

Not sure how this is relevant to the problem, I just probably didn't merge / use latest commit properly when updating in my org.

[0x4007]

@EresDev Solve merge conflict and you can merge

[0x4007]

Looks like there's a ton of changes. Perhaps you should ensure it works by posting QA

[EresDev]

Looks like there's a ton of changes. Perhaps you should ensure it works by posting QA

Here is the latest QA. It looks good to me. If you find it good, you can merge it. @0x4007