Identify Key UI Metrics

[Gregory-Pereira]

Metrics discussion

Summary: lets think about what metrics we want to capture from the upstream perspective

Background

Relates to: #281

Our metrics infra will be entirely up soon. The deployments went through in #391 with some minor follow on fixes. From the UI side, we can leverage the default metrics in Umami with the implementation of their tracking codes in our TS codebase (#411). As a result, I would like to start a discussion on what custom pieces we hope to capture.

What Umami gives us out of the box

To see what Umami provides in depth please refer to their docs on the subject.

What metrics we would like to gather so far

Note: this will exclude some features that might take a while to implement!

Features excluded:
- auto-QNA generation and context chunking
- knowledge document propulating fields accuracy

Chat:

• how many requests are hitting the chatbot (per user, agregate, etc.)

Submissions (Primarily knowledge but skills where applicable):

• count / percentage of complemtions that get through each contribution stage (Author Information, Knowledge information)
• time per submission
• Author information:
  • accuracy prepopulating fields (If Applicable)
• knowledge information:
  • number of contributers per submission
• File Path Information:
  • Some way to track if users are using good document paths?
• Document Information:
  • How many manually enter documnets vs automatically upload documents
  • knowledge documents upload count, count per use
  • number of issues with md knowledge document upload, error count and type
• Attribution details:
  • accuracy prepopulating fields
  • how often users accesses the help and what fields they are having issues with
  • what licenseses the are being used

Stakeholders to engage

@vishnoianil
@nerdalert
@andybraren
@Misjohns
@aevo98765
@RobotSail
@toraponibm
And anyone else I may have missed.

[Misjohns]

Adding @pilhuhn to this conversation.

[vishnoianil]

I would like to add few more to the list

• How many time user use View Yaml and Attribution file for both knowledge and skill
• How many times user use Download Yaml and Attribution file for both knowledge and skill
• If user uses view or download option, does user still submit that skill or knowledge contribution.
• How many user use the custom add point feature and also attempt to chat with it.

[Misjohns]

@Gregory-Pereira I created this doc a while ago around the SME Contributor Journey. Please take a look and see if there's items we'd like to add here.
https://docs.google.com/document/d/1zyBaKJS3Icj28Ddwt_32MRWUS4uT5jI45ijErJJUiQA/edit?usp=sharing