PR to apply for E2E OLS evaluation framework for AAP chatbot

[justjais]

Description

PR to apply for E2E OLS evaluation framework for AAP chatbot

Type of change

• Refactor
• New feature
• Bug fix
• CVE fix
• Optimization
• Documentation Update
• Configuration Update
• Bump-up dependent library
• Bump-up library or tool used for development (does not change the final image)
• CI configuration change
• Konflux configuration change

Related Tickets & Documents

• Related Issue #
• Closes #

Checklist before requesting a review

• I have performed a self-review of my code.
• PR has passed all pre-merge test jobs.
• If it is a core feature, I have added thorough tests.

Testing

• Create a virtual environment, and install the necessary package via make install-deps, for this user need to clone ansible-chatbot-service repo.
• For running the E2E tests locally, user need to configure olsconfig.yaml and copy to parent directory, for AAP chatbot scenario sample olsconfig.yaml is under /scripts/evaluation/ folder.
• Once configured, user can run the evaluation framework over AAP complete set of QnA as under aap_doc_qna.parquet under /scripts/evaluation/eval_data/ folder, or else run over sample QnA defined under scripts/evaluation/eval_data/aap-sample.parquet file.
• To run the eval framework, run following cmd:

OPENAI_API_KEY=IGNORED python -m scripts.evaluation.driver --qna_pool_file /Users/sjaiswal/Sumit/wisdom/ansible-chatbot-service/scripts/evaluation/eval_data/aap-sample.parquet --eval_provider_model_id my_rhoai+granite3-8b --eval_metrics answer_relevancy answer_similarity_llm cos_score rougeL_precision --eval_modes ols_rag --judge_model granite3-8b --judge_provider my_rhoai --eval_query_ids qna1 qna2 qna3 qna4 qna5

[TamiTakamiya]

@justjais I have rebased ansible-chatbot-service to the latest upstream (road-core/service). Would you rebase to the current main branch? Sorry for causing extra work.

[TamiTakamiya]

@justjais I think we want to update those evaluation-related files whenever OLS updates their files. For that purpose, would you add following changes?

1. Copy all files under scripts/evaluation in the OLS repo even if they are not used. I think it is easier for us to have them when we import changes in the OLS repo.
2. If any additional files are required, give file names that include aap-
3. Create scripts/evaluation/README-aap.md to document what was changed/added from/to the OLS code.
4. Also document how to run the tool for Ansible chatbot in the same scripts/evaluation/README-aap.md file.