Python 4.1.0

New to deepeval? Get started here.

✨ New Feature

• Add AgentLoopDetectionMetric for deterministic loop detection in agent traces. (#2645 by @Jeel3011)
• Add a deterministic ToolPermissionMetric for metrics evaluation. (#2826 by @gh-raju)

⚡ Improvement

• Add an AI-generated draft body for release notes generation. (#2861 by @A-Vamshi)
• Fix typos in documentation and code comments. (#2865 by @maxtaran2010)

🐛 Bug Fix

• Fix MCP metrics to use tools_called and relax type strictness. (#2864 by @A-Vamshi)
• Fix Ragas integration to call capture_metric_type with the updated telemetry signature. (#2568 by @sachinML)
• Fix task completion to pass tools_called_formatted into the extract_goal_and_outcome template. (#2808 by @ppcvote)
• Fix synthesizer crash by avoiding an UnboundLocalError when a document yields zero chunks. (#2849 by @hassaanch23)
• Fix dotted attribute access to deepeval.evaluate.configs. (#2851 by @ErenAta16)

New contributors: @maxtaran2010, @Jeel3011, @gh-raju, @sachinML, @ppcvote, @hassaanch23, @ErenAta16

A huge thank you to everyone who contributed to this release ❤️

Opus 4.8: Day 0 Support

New Feature

• Add support for the claude-opus-4-8 model preset, including multimodal and structured output capabilities with updated pricing metadata. (#2698) (Vamshi Adimalla)

🎉 New Decision Graph Logic for Granular Simulation Control

New Features

• Add a simulation graph API to control how user turns are generated during conversation simulation. ConversationSimulator now accepts simulation_graph, and controller is deprecated in favor of stopping_controller with a warning for legacy usage. (#2678) (Jeffrey Ip)
• Add support for retrieval_context entries as RetrievedContextData with context and source, enabling contextual precision to group retrieved chunks by source. This also normalizes mixed inputs to plain context strings when preparing datasets and API payloads. (#2669) (Vamshi Adimalla)
• Add multimodal trace support by converting LangChain image/PDF content blocks into placeholders and including the referenced attachments when exporting traces. Multimodal detection now recognizes both IMAGE and PDF placeholders. (#2672) (Vamshi Adimalla)

Improvement

• Add a GitHub Actions workflow to auto-generate release notes from the latest tag or version bump and draft a GitHub Release. This includes extracting the tag-specific notes into a release notes file and falling back to a basic draft if none are found. (#2671) (Vamshi Adimalla)

❤️ Contributors

A huge thank you to our contributors for this release: @penguine-ip, @A-Vamshi

🔥 DeepEval 4.0: Eval Harness for Coding Agents, 1-line integrations, TUI for trace inspection!

DeepEval 4.0 introduces an agent-native evaluation workflow designed for coding agents, rapid debugging, and production AI systems.

If you're vibe coding agents, on something like claude code, this release is for you.

Eval Harness for Coding Agents

Coding agents can now run eval-driven iterations directly in context.

• Agents see metric failures, scores, and reasoning inline
• Supports iterative patch → eval → retry workflows
• Built for Cursor, Claude Code, Codex, and agentic development loops
• Enables autonomous regression fixing with evaluation feedback

Quickstart: https://deepeval.com/docs/vibe-coder-quickstart

---

Terminal Trace Inspection (TUI)

Inspect traces locally without leaving the terminal.

• Navigate spans, inputs, outputs, and metric failures
• Debug agent execution step-by-step
• Understand why an eval failed, not just that it failed
• Lightweight local-first developer workflow

The idea is to develop rapidly by staying local on your machine. No more delegating to external UIs, unless collaboration is required.

---

10+ Native Integrations

One-line integrations for modern AI frameworks and providers.

Supports:

• LangChain
• OpenAI
• Anthropic
• LiteLLM
• PydanticAI
• CrewAI
• LlamaIndex
• DSPy
• Google ADK
• Custom agents

Here's a langchain example:

import pytest
from langchain.agents import create_agent
from deepeval import assert_test
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric

def multiply(a: int, b: int) -> int:
    return a * b

agent = create_agent(model="openai:gpt-4o-mini", tools=[multiply], system_prompt="Be concise.")

dataset = EvaluationDataset(goldens=[
    Golden(input="What is 8 multiplied by 6?"),
    Golden(input="What is 7 multiplied by 9?"),
])

@pytest.mark.parametrize("golden", dataset.goldens)
def test_langchain_agent(golden: Golden):
    agent.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler()]},
    )
    assert_test(golden=golden, metrics=[TaskCompletionMetric()])

That's it! Native CI/CD integration for LangChain agents, via Pytest, enabled by DeepEval.

Quickstart for integrations: https://deepeval.com/integrations

🎉 Metrics for AI agents, multi-turn synthetic data generation, and more!

Full support for agentic evals :)

If you're building agents, DeepEval can now analyze and give you metric scores based on the trace of your LLM app.

🎯 1. Task Completion

Evaluate whether an agent actually completes the intended task, not just whether its final output “looks correct.”

Captures:

• Goal completion
• Intermediate step correctness
• Error recovery
• Procedural accuracy

Docs: https://deepeval.com/docs/metrics-task-completion

---

🔧 2. Tool Correctness

Evaluates whether tools were invoked correctly, meaningfully, and in the right order.

Captures:

• Correct tool usage
• Correct argument formatting
• Avoiding hallucinated tools
• Using tools only when needed

Docs: https://deepeval.com/docs/metrics-tool-correctness

---

🧩 3. Argument Correctness

Evaluates whether the agent’s arguments to tools are valid, structured, and aligned to the task.

Captures:

• Correct parameter selection
• Type/format adherence
• Logical argument formation
• Avoiding semantically incorrect inputs

Docs: https://deepeval.com/docs/metrics-argument-correctness

---

⚡ 4. Step Efficiency

Measures how efficiently an agent completes a task — rewarding fewer unnecessary steps and penalizing detours.

Captures:

• Optimality of step count
• Redundant tool calls
• Unnecessary loops
• Waffling behavior

Docs: https://deepeval.com/docs/metrics-step-efficiency

---

🪜 5. Plan Adherence

Evaluates how well the agent follows a predefined or self-generated plan.

Captures:

• Alignment to planned steps
• Deviations and detours
• Fidelity to strategy
• Execution according to intent

Docs: https://deepeval.com/docs/metrics-plan-adherence

---

🧭 6. Plan Quality

Evaluates the quality of the plan itself when the agent generates one.

Captures:

• Clarity
• Completeness
• Achievability
• Logical ordering of steps

Docs: https://deepeval.com/docs/metrics-plan-quality

---

🧪 New: Multi-Turn Synthetic Goldens Generation

Synthetic data generation now supports multi-turn goldens instead of just single-turn.

You can now generate:

• 🎭 Multi-turn conversational scenarios
• 📝 Scenario + Expected Outcome pairs
• 🔁 Turn-by-turn dialogue structure
• 💬 Goldens instantly compatible with the Conversation Simulator
• 🚀 Direct pipeline: Generate → Simulate → Evaluate

Perfect for building large-scale synthetic datasets for support agents, sales agents, research assistants, workflow agents, and any multi-step conversational system.

from deepeval.synthesizer import Synthesizer

synthesizer = Synthesizer()
conversational_goldens = synthesizer.generate_conversational_goldens_from_docs(
    document_paths=['example.txt', 'example.docx', 'example.pdf'],
)

Docs here (click on the "multi-turn" tab): https://deepeval.com/docs/synthesizer-generate-from-docs

🎉 New Interfaces, Reduce ETL Code < 50%!

Less Code to Load Data In and Out of DeepEval's Ecosystem :)

If you're using any of the features below, you'll likely see a 50% reduction in code required, especially around ETL for formatting things in and out of DeepEval's ecosystem. This includes:

🆚 Arena-GEval

The first LLM-arena-as-a-Judge metric, now runs a blinded experiment and swaps positions randomly for a fair verdict on which LLM output is better.

Docs: https://deepeval.com/docs/metrics-arena-g-eval

⚛️ You can now run component-level evals by simply running a for loop against your dataset of goldens.

Simply run your loop -> call your agent X number of times -> get your evaluation results. No more trying to fit non-test-case-friendly formats. Instead DeepEval will find your LLM traces automatically to run evals.

from somewhere import your_async_llm_app # Replace with your async LLM app
from deepeval.dataset import EvaluationDataset, Golden

dataset = EvaluationDataset(goldens=[Golden(input="...")])

for golden in dataset.evals_iterator():
    # Create task to invoke your async LLM app
    task = asyncio.create_task(your_async_llm_app(golden.input))
    dataset.evaluate(task)

Docs: https://deepeval.com/docs/evaluation-component-level-llm-evals

💬 Conversation simulator is now based on goldens.

Previously you have to define a list of user intentions, profile items, with a ton of more configs to juggle between. Now you can define a list of goldens with a standardized benchmark of scenarios to have turns generated for.

from deepeval.test_case import Turn
from deepeval.simulator import ConversationSimulator

# Create ConversationalGolden
conversation_golden = ConversationalGolden(
    scenario="Andy Byron wants to purchase a VIP ticket to a cold play concert.",
    expected_outcome="Successful purchase of a ticket.",
    user_description="Andy Byron is the CEO of Astronomer.",
)

# Define chatbot callback
async def chatbot_callback(input):
    return Turn(role="assistant", content=f"Chatbot response to: {input}")

# Run Simulation
simulator = ConversationSimulator(model_callback=chatbot_callback)
conversational_test_cases = simulator.simulate(goldens=[conversation_golden])
print(conversational_test_cases)

Docs: https://deepeval.com/docs/conversation-simulator

We also updated our docs with more improvements to come 👀

🎉 Renewed datasets, single vs multi-turn

⚙️ New Features

DeepEval's 3.2.6 release focuses on single-vs multi-turn use cases in datasets!

🧩 Support for Single-Turn and Multi-Turn Datasets

• Single-turn datasets: Simple input → output pairs for one-off prompt testing.
• Multi-turn datasets: Full conversation flows with alternating user/assistant turns. Perfect for simulating real chat interactions.

DeepEval now automatically detects whether a dataset is single-turn or multi-turn based on structure and routes to the appropriate evaluation logic.

🧪 Conversational Goldens

Introduced a new concept: conversational goldens, which contains scenario, (and optionally expected_outcome) but not things like input and expected output as with single-turn use cases..

✅ Improvements

• Smarter dataset evaluation routing: Whether single-turn or multi-turn, DeepEval figures it out and builds test cases accordingly.
• Improved multi-turn context preservation: Each conversational turn is maintained during evaluation, giving more accurate multi-turn metrics.

This release is setting the stage for future multi-turn use cases.

Docs here: https://deepeval.com/docs/evaluation-datasets

🎉 New Arena GEval Metric, for Pairwise Comparisons

Metric that is alike LLM Arena is Here

In DeepEval's latest release, we are introducing ArenaGEval, the first ever metric to compare test cases to choose the best performing one based on your custom criteria.

It looks something like this:

from deepeval import evaluate
from deepeval.test_case import ArenaTestCase, LLMTestCaseParams
from deepeval.metrics import ArenaGEval

a_test_case = ArenaTestCase(
    contestants={
        "GPT-4": LLMTestCase(
            input="What is the capital of France?",
            actual_output="Paris",
        ),
        "Claude-4": LLMTestCase(
            input="What is the capital of France?",
            actual_output="Paris is the capital of France.",
        ),
    },
)
arena_geval = ArenaGEval(
    name="Friendly",
    criteria="Choose the winter of the more friendly contestant based on the input and actual output",
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
)


metric.measure(a_test_case)
print(metric.winner, metric.reason)

Docs here: https://deepeval.com/docs/metrics-arena-g-eval

🎉 New Multimodal Metrics, with Platform Support

In DeepEval's latest release, we are introducing multimodal G-Eval, plus 7+ multimodal metrics!

Previously we had great support for single-turn, text evaluation in the form of LLMTestCases, but now we're adding MLLMTestCase, which accepts images:

from deepeval.metrics import MultimodalGEval
from deepeval.test_case import MLLMTestCaseParams, MLLMTestCase, MLLMImage
from deepeval import evaluate

m_test_case = MLLMTestCase(
    input=["Show me how to fold an airplane"],
    actual_output=[
        "1. Take the sheet of paper and fold it lengthwise",
        MLLMImage(url="./paper_plane_1", local=True),
        "2. Unfold the paper. Fold the top left and right corners towards the center.",
        MLLMImage(url="./paper_plane_2", local=True)
    ]
)
text_image_coherence = MultimodalGEval(
    name="Text-Image Coherence",
    criteria="Determine whether the images and text is coherence in the actual output.",
    evaluation_params=[MLLMTestCaseParams.ACTUAL_OUTPUT],
)

evaluate(test_cases=[m_test_case], metrics=[text_image_coherence])

Docs here: https://deepeval.com/docs/multimodal-metrics-g-eval

PS. This also includes platform support

🎉 New Conversational Evaluation, LiteLLM Integration

In DeepEval's latest release, we are introducing a slight change in how a conversation is evaluated.

Previously we assumed a conversation as as a list of LLMTestCases, which might necessarily be the case. Now a conversational test case is made up of a list of Turns instead, which follows OpenAI's standard messages format:

from deepeval.test_case import Turn

turns = [Turn(role="user", content="...")]

Docs here: https://deepeval.com/docs/evaluation-test-cases#conversational-test-case