Setup the tracing infrastructure for agents

[Teneroy]

Description

The goal of this task is to set up LangFuse for tracing in the Kyma Companion backend. The task involves figuring out how to deploy LangFuse for testing, testing the deployment process on both a local machine and a Kubernetes cluster, and then integrating the LangFuse framework into the Kyma Companion backend. Additionally, the task includes configuring the ability to switch between debug and production modes with different levels of information storage.

Subtasks

1. LangFuse Deployment - Local Machine:

   • Research and figure out how to deploy LangFuse on a local machine.
   • Test the deployment process to ensure it works correctly.
   • Document the steps for deploying LangFuse on a local machine.

2. LangFuse Deployment - Kubernetes Cluster:

   • Research and figure out how to deploy LangFuse on a Kubernetes cluster.
   • Test the deployment process to ensure it works correctly.
   • Document the steps for deploying LangFuse on a Kubernetes cluster.

3. Integrate LangFuse into Kyma Companion Backend:

   • Integrate the LangFuse framework into the Kyma Companion backend.
   • Ensure that LangFuse can be enabled or disabled via configuration.

4. Debug and Production Mode Configuration:

   • Set up a mechanism to switch between debug mode and production mode.
   • In debug mode, configure LangFuse to store all available tracing information.
   • In production mode, configure LangFuse to store only token usage, time, and names of the traces.

5. Set Up Tracing for Code Verification:

   • Implement tracing for a specific part of the code to verify the functionality of LangFuse.
   • Ensure that the tracing works correctly both when deploying the Kyma Companion locally and on a Kubernetes cluster.
   • Verify that the functionality is consistent across both environments in debug and production modes.

6. Testing and Documentation:

   • Test the entire setup to ensure that LangFuse is correctly integrated and functioning as expected.
   • Document the entire process, including deployment steps, integration steps, and how to switch between modes.

Acceptance Criteria

• LangFuse is successfully deployed and documented for both local machine and Kubernetes cluster setups(LangFuse Helm charts).
• LangFuse is integrated into the Kyma Companion backend with a clear mechanism to switch between debug and production modes.
• Tracing is successfully set up and verified in both local and Kubernetes cluster environments.
• Functionality is consistent and validated across both environments in both debug and production modes.
• Documentation is complete and provides clear instructions for future reference.

[mfaizanse]

Create a small how-to guide on how to configure Langfuse properly, so we won't receive errors