Checklist items provide guides instead of analysis

[IlonaShishov]

Sometimes, items in the checklist provide a "How-to" guide or instructions on achieving a task, rather than fulfilling their intended purpose of scanning the code base and reporting actual findings based on the analysis.

An example of such an item is:
{ "input": "Inspect Cleanup Process: Analyze the cleanup process following the decompression of archives. Ensure that it does not follow relative paths that could lead to overwriting or deleting files outside the intended scope. This is crucial as the vulnerability exploits this specific behavior.", "response": "The cleanup process following the decompression of archives should be designed to prevent path traversal attacks. This can be achieved by ensuring that the decompression process does not follow relative paths that could lead to overwriting or deleting files outside the intended scope. The container image should use absolute paths and validate user input to prevent such attacks. However, without more specific information about the container image's implementation, it is difficult to provide a more detailed answer." },

CVE: CVE-2024-1485
Component: openshift4/ose-console
output.json

[ashsong-nv]

Hi Ilona, thank you for raising this issue! We discussed this as a team and we think that prompt tuning experiments for the checklist model may be able to improve this issue.

Before running specific checklist model prompt tuning experiments, we are concurrently evaluating llama 3.3 model as a general improvement to the pipeline, and so far we are seeing promising accuracy improvements when using it for checklist generation. Our plan is to work on prompt engineering for this issue after migrating to llama 3.3 to ensure the new prompt is customized to work with the new model.