Models: openai/gpt-4o-mini-2024-07-18 openai/gpt-4o-2024-08-06 openai/gpt-4-turbo-2024-04-09 (3d, no reason to test 4d) openai/o1-mini-2024-09-12 (3d, costly)
anthropic/claude-3-haiku-20240307 (3d, poor perf) anthropic/claude-3-5-haiku-20241022
- anthropic/claude-3-5-sonnet-20240620
- anthropic/claude-3-5-sonnet-20241022
openrouter/google/gemini-flash-1.5 (gemini-1.5-flash-002)
=== openrouter/meta-llama/llama-3.1-8b-instruct openrouter/meta-llama/llama-3.1-70b-instruct openrouter/qwen/qwen-2.5-coder-32b-instruct
openai/gpt-4o-mini-2024-07-18: 283k cached + 221k uncached + 68k output = $0.1 openai/gpt-4o-2024-08-06: 174k cached + 241k uncached + 56k output = $1.38 openai/gpt-4-turbo-2024-04-09: UNKNOWN = $6.65 openai/o1-mini-2024-09-12: 0k cached + 335k uncached + 1345k output = $17.15
anthropic/claude-3-haiku-20240307: 492k input + 46k output = $0.18
openai/gpt-4o-mini-2024-07-18: 451k cached + 429k uncached + 100k output = $0.15 openai/gpt-4o-2024-08-06: 553k cached + 287k uncached + 87k = $2.29 (25 games) openai/o1-mini-2024-09-12: 0k cached + 584k uncached + 1815k output = $23.54
anthropic/claude-3-5-haiku-20241022: 969k input + 90k output = $1.42
$5.2 sonnet 3.5 (openrouter)