[Resolver] Product Analytics in Openhands-Resolver?

[neubig]

Summary

It would be nice to see data on how the openhands-resolver is being used. Ideally we have a better idea of how users prefer using openhands-resolver and what types of tasks users are assigning to openhands-resolver.

Tracking Attributes

To begin with aggregates of the following values would be nice to see. This should also support users that are using the resolver on private repos but are willing to opt-in for tracking.

1. Invocation method (macro vs fix-me label)
2. If the issues/PR was successfully resolved
3. Resolver errors (agent didn't make changes, unexpectedly crashed, etc)

Technical Design

We would ask users to opt in by setting a repository variable openhands-resolver-opt-in which will be referenced when the workflow is trigger.

Since Posthog is already supported in Openhands proper we could implement something similar here, and track only if the opt-in variable is set by the user.

---

Moved from All-Hands-AI/openhands-resolver#332

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been stalled for over 30 days with no activity.