Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: bump dependencies and readme #8

Merged
merged 4 commits into from
Nov 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CITATION.cff
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "Ismail"
given-names: "Pelaseyd"
title: "Superagent PokerEval"
date-released: 2024-11-25
url: "https://github.com/superagent-ai/poker-eval"
40 changes: 37 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

A comprehensive tool for assessing AI agents performance in simulated poker environments. Written in Typescript.

[Getting Started](#getting-started) | [Examples](#examples) | [FAQ](#faq) | [Development](#development) | [Contributing](#contributing)
[Getting Started](#getting-started) | [Why Poker?](#why-poker) | [Leaderboard](#leaderboard) | [Examples](#examples)

</div>

Expand All @@ -24,24 +24,27 @@ npm i @superagent-ai/poker-eval
// index.ts

import { PokerGame } from "@superagent-ai/poker-eval";
import { Player, PlayerAction } from "@superagent-ai/poker-eval/dist/types"

// See example agent: https://github.com/superagent-ai/poker-eval/blob/main/examples/ai-sdk/agent.ts
import { generateAction } from "./agent";
import { Player, PlayerAction } from "../../src/types";


async function executeGameSimulation(numHands: number): Promise<void> {
// Setup AI players
const players: Player[] = [
{
name: "GPT 1",
action: async (state): Promise<PlayerAction> => {
// Use any model, framework or code to generate a response
const action = await generateAction(state);
return action;
},
},
{
name: "GPT 2",
action: async (state): Promise<PlayerAction> => {
// Use any model, framework or code to generate a response
const action = await generateAction(state);
return action;
},
Expand Down Expand Up @@ -82,10 +85,41 @@ In this example, the dataset shows the position of the player, their hole cards,

BB/100, or Big Blinds per 100 hands, is a common metric used in poker to measure a player's win rate. It represents the average number of big blinds a player wins or loses over 100 hands. To calculate BB/100, use the formula:

BB/100 = (Total bb_profit / Number of hands) * 100
`BB/100 = (Total bb_profit / Number of hands) * 100`

This formula provides a standardized measure of performance, allowing for comparison across different sessions or players by normalizing the win rate to a per-100-hands basis.


## Why Poker?
Poker combines elements of strategy, psychology, risk assessment, and partial information - perfect for testing an Agent's decision-making skills in complex, uncertain environments. Poker provides measurable KPIs like EV, BB/100, All-In adj BB/100 and VPIP. These KPIs are widely recognized standards, not created by a single company, making them ideal for objectively evaluating an Agent's decision-making skills.

We've specificalyy chosen No Limit Texas Holdem cash games and are officially calling the eval `NLTH`.

## Leaderboard
Coming soon...

## Examples
We've created some examples using populat agent frameworks you can use as inspiration (feel free to contribute):

- [Vercel AI SDK](https://github.com/superagent-ai/poker-eval/tree/main/examples/ai-sdk)
- [OpeaAI]() Coming soon
- [Mastra]() Coming soon
- [LlamaIndex]() Coming soon
- [Langchain]() Coming soon

## Citations
```json
{
"cff-version": "1.2.0",
"message": "If you use this software, please cite it as below.",
"authors": [
{
"family-names": "Ismail",
"given-names": "Pelaseyd"
}
],
"title": "Superagent PokerEval",
"date-released": "2024-11-25",
"url": "https://github.com/superagent-ai/poker-eval"
}
```
8 changes: 4 additions & 4 deletions examples/ai-sdk/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion examples/ai-sdk/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"license": "ISC",
"dependencies": {
"@ai-sdk/openai": "^1.0.4",
"@superagent-ai/poker-eval": "^1.0.0",
"@superagent-ai/poker-eval": "^1.0.1",
"ai": "^4.0.3",
"dotenv": "^16.4.5",
"zod": "^3.23.8"
Expand Down
Loading