CHANGELOG.md

Changelog

0.9.0 (2025-12-23)

Features

• Add prompt caching (#135) (a1b1988)
• Per-turn evaluations for multi-turn conversations (#149) (23fad93)

Bug Fixes

• Per-turn evaluation render (#150) (a40a253)

Chores

• Sort eval results by model name and sample ID for consistent ordering (#148) (eae891a)

0.8.1 (2025-12-18)

Chores

• Add Gemini 3 Flash to leaderboard (#146) (bd05b8e)

0.8.0 (2025-12-16)

Features

• Add kimi-k2-thinking (#125) (a5ca7e7)
• Add skills suite for leaderboard (#128) (d252523)
• Remove instance level pass/fail metrics (#127) (b7e91b6)

Bug Fixes

• Fix visualization status bar + mark empty submissions as errors (#126) (ffe75d2)
• Google loads google.svg (#134) (877538c)
• Leaderboard tooltip, spacing (#129) (993683f)
• Migrate to 1.0 SDK (#143) (08a2854)
• update index.html (2dda09f)

Chores

• Add leaderboard favicon (#138) (313f380)
• Add updates tab to leaderboard (#137) (44b0255)
• GPT 5.2 leaderboard evals (#139) (00894cb)
• GPT 5.2 xhigh leaderboard evals (#141) (7f7c4cb)
• GPT5.1, Gemini 3, Opus 4.5 updates (#133) (de1f4ca)
• Improve empty submission check (#132) (2463a5b)
• Multiple benchmarks on leaderboard (#124) (39cb733)
• Track costs during runs (#130) (dc980a1)
• Update leaderboard logs (#142) (e34e0cd)
• Update leaderboard with Deepseek v3.2 and Mistral large 3 (#136) (01a17cb)

0.7.0 (2025-11-04)

Features

• Evaluate multiple models with letta-code (#113) (f98933b)
• Support multiple graders in gates with weighted average and logical combinations (#117) (d0d0add)

Bug Fixes

• Fix per sample pass is None (#120) (f416487)

Documentation

• Add examples for multiple metric gates (#118) (739e4de)
• Add reference to multi grader gate example in top level README (#119) (7ad732d)
• Clean up README of Claude references (#114) (a41f9d0)

0.6.1 (2025-10-30)

Bug Fixes

• Letta code session_id to agent_id (#112) (3ad7c0c)
• patch site (24cfa0b)

Chores

• fix lints (#111) (e3431b8)
• fix styling (7dd2818)

0.6.0 (2025-10-29)

Features

• add eval website (#104) (2daaf0c)
• Fix workflow (#105) (62207bd)
• Support letta code as builtin target (#101) (fe1ae2f)

Bug Fixes

• Fix kwargs in run function (#97) (c0f64b0)
• Remove duplicate gpt 4.1 results (#95) (2b9092c)
• Update Sonnet 4.5 cost (#96) (3b7a39d)

Refactors

• Add extra vars to Sample (#100) (9ce87cd)
• Make target spec a discriminated union (#103) (762b520)
• Refactor AgentTarget to LettaAgentTarget (#98) (952d6f8)
• Refactor Target to AbstractAgentTarget (#99) (bb632e0)

Documentation

• Add letta code example to READMEs (#102) (2fe0f68)
• adjust (2b16cd3)
• patch svg, update site (64c3491)

Chores

• Add leaderboard results yaml and script (#93) (e24fb74)
• fix style (#108) (575be53)
• relative routes bruh (#109) (d68ee95)
• whoops (#107) (53fc4eb)

0.5.0 (2025-10-23)

Features

• Add agent_id to visualization (#91) (2f77348)
• Add agent-as-judge support for rubric grading (#77) (ae4878e)
• Add summary tables on suite finish for all display types (rich, simple) (#92) (19e1e1c)
• Support anthropic models as grader (#83) (c38cf1f)
• Support default Letta judge agent with new letta_judge grader kind (#86) (b4bfd6c)

Bug Fixes

• Add defensive check for run_id from streaming chunk (#75) (7d34884)
• Add pre-fill trick for Anthropic json output (#84) (2a4fd4a)
• Fix retry logic for failing agent (#74) (ecd5d5a)
• Fix typo in chunks appending (#82) (d888474)
• OpenRouter for Kimi (#76) (2cd6192)
• Print out chunks on run_id error (#81) (8b71a64)

Performance Improvements

• Retry on letta server failures (#72) (acaf5a5)

Refactors

• Flatten package imports for easier pip usage (#89) (24fd61a)
• Rename rubric to model_judge (#87) (0047d3f)
• Use Pydantic discriminated union for GraderSpec types (#88) (bb21f1b)

Documentation

• Add perma-link for UI screenshot (#79) (00dfb38)
• Fix typo in README (#80) (a5e0497)
• Write multi-turn memory example (#90) (a683727)

0.4.1 (2025-10-21)

Features

• Add ruff check --fix . to README (#65) (066408b)

Bug Fixes

• Use rubric grader for file system task (#63) (03ef537)

Documentation

• Add example for tool output extractor (#66) (523d498)
• Add extra instructions for cloud usage (#71) (a8d3e5b)
• Write high level README (#69) (68be3da)

Chores

• Update preview image (#70) (81d5b41)

0.4.0 (2025-10-21)

Features

• Add multi-run (#57) (23ead2d)
• Add rubric vars (#58) (29f775a)
• Support csv loading (#60) (28bf491)

Bug Fixes

• Update filesystem dataset (#62) (634bfc6)

Chores

• Replace deepseek and GLM models (#61) (3f0bf79)

0.3.2 (2025-10-20)

Features

• Add visualization library and simple visualization configurations (#55) (57d483d)

0.3.1 (2025-10-20)

Bug Fixes

• Package python files in build (#53) (d5e2682)

0.3.0 (2025-10-20)

Features

• Add filesystem benchmark generator (#48) (c2be72a)
• Add max samples to display (#47) (c821171)
• Add memory block built-in extractor (#51) (d171a10)
• Remove hardcoding metric to accuracy (#45) (62f41c4)
• Support passing in handles instead of just model configs (#41) (927e70a)
• Use gpt-5-mini as rubric grader model (#46) (4a4a0ff)

Bug Fixes

• Cannot access local variable 'stream' error (#33) (a989a16)
• Expunge send_message and disable tool rules (7775596)
• Fix streaming bug returns partial results (8d3e3d8)
• Model, status and metric columns after evaluation completes (#34) (b47c508)
• Update leaderboard task suites (#35) (436ce6f)

Refactors

• Support passing in token, base_url, and project_id programmatically (#36) (1e3780a)

Documentation

• Add README for memory block extraction (#52) (49ccf25)

Chores

• Configurable retries and timeout (9ee97b6)
• Report average metrics across attempted and total samples (#50) (f2b4f7a)
• Separate files for headers and summary (#49) (9f3dbcc)
• Update examples to use letta_v1_agent (#31) (62a6ab6)
• Update model configs (91afcb8)

0.2.0 (2025-10-15)

Features

• Add builtin tool output/arguments extractors (5133966)
• add core memory update benchmark (#21) (e0261de)
• add filesystem eval (#24) (55b902d)
• add letta leaderboard (#8) (ae68e22)
• Add model configs and multi-model runners (cf6a707)
• Add programmatic agent creation (fd820a2)
• Add support for re-grading cached evaluation trajectories (#19) (8feab64)
• Clean up results.json schema (#18) (ec72e2b)
• Flatten directories further (759638c)
• Implement decorator based custom functions (e53ee19)
• Refactor to use TaskGroups (9bc810b)
• Support custom extractors/Python tool evaluators (fc1b2f7)
• support multiple metrics (#27) (b0fa023)
• Support relative paths for custom graders (6f3cda3)
• Support streaming for stability (ef18ef6)
• update together configs (#20) (79d7890)

Chores

• Prepare repo for Pypi publishing (#28) (6ad92b2)
• Rename ideal to ground truth (9367c05)