Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement the metrics module #24

Open
tkornuta-semiotic opened this issue Dec 19, 2022 · 6 comments
Open

Implement the metrics module #24

tkornuta-semiotic opened this issue Dec 19, 2022 · 6 comments
Assignees
@tumayt
Labels
🔧 Enhancement New feature or request

Comments

@tkornuta-semiotic
Copy link
Contributor

No description provided.

@tkornuta-semiotic tkornuta-semiotic changed the title test Test issue Dec 19, 2022
@tkornuta-semiotic
Copy link
Contributor Author

  1. Elementary metrics:
  • served queries
  • revenue (= served queries * price)
  1. Derivative metrics:
  • query ratio (= served queries / total queries)
  • regret (= customer budget - agent's revenue)
  • regret ratio (= agent's revenue / customer budget)

@tkornuta-semiotic tkornuta-semiotic added the 🔧 Enhancement New feature or request label Dec 19, 2022
@tkornuta-semiotic
Copy link
Contributor Author

  1. time to convergence
  • epsilon_1 = accepted difference between the customer budget b and agent's price p (mean)
  • epsilon_2 = defines how wide the gaussian can be (stddev)

agent converged IF |b - p| <= epsilon_1 AND stddev <= epsilon_2

@tkornuta-semiotic
Copy link
Contributor Author

note: think/discuss about the warm-up phase (dynamic budget and/or no queries)

@tkornuta-semiotic tkornuta-semiotic changed the title Test issue Implement the metrics module Dec 19, 2022
@tkornuta-semiotic
Copy link
Contributor Author

ideas: https://torchmetrics.readthedocs.io/en/stable/

@tkornuta-semiotic
Copy link
Contributor Author

https://snyk.io/advisor/python/jax-metrics

@tkornuta-semiotic
Copy link
Contributor Author

Price efficiency = Indexer price / customer budget

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tumayt tumayt

Labels
🔧 Enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

When branches are created from issues, their pull requests are automatically linked.

24-implement-the-metrics-module semiotic-ai/autoagora-agents
2 participants
@tkornuta-semiotic @tumayt