TruLens 2.11.0

Two features land in 2.11.0 that push the boundary of how you can observe your agents.

We added a direct path for Anthropic models in the new anthropic providers package. Anthropic becomes a first-class provider — Claude models are now usable as judges through a native trulens-providers-anthropic package with real cost tracking, rather than via a LiteLLM hop.

And online evaluation gets sampling controls, so you can turn on automatic post-ingest evals on a high-traffic app without evaluating (and paying for) every single record.

Alongside those, MCP tool calls get an end-to-end cookbook showing how to trace and evaluate them.

Three new contributors shipped every feature in this release.

New Features

Anthropic provider for Claude models

trulens-providers-anthropic (#2532 — @Oxygen56)

A new installable provider package for native Anthropic/Claude support, following the same pattern as the OpenAI and Google providers. Defaults to claude-sonnet-4-6, reads ANTHROPIC_API_KEY from the environment, and tracks cost per-model across the Opus 4, Sonnet 4, and Haiku 4.5 families.

Notable implementation details: OpenAI-style message lists are translated to Anthropic's format (system messages hoisted to the top-level system param, tool messages converted to tool_result blocks, consecutive same-role messages merged to satisfy Anthropic's alternating requirement). Structured output uses Anthropic's native tool_use with the Pydantic JSON Schema passed as a tool definition, falling back to text extraction on parse failure.

Closes #2404.

pip install trulens-providers-anthropic

from trulens.providers.anthropic import Anthropic

provider = Anthropic()  # or Anthropic(model_engine="claude-opus-4-1")

score, reasons = provider.relevance_with_cot_reasons(
    "What is the capital of France?",
    "Paris is the capital of France.",
)

Sampling for online evaluation

TruSession.configure_online_eval() (#2649 — @Payal2000)

Controls how much of your ingested traffic gets automatically evaluated. Three independent levers: sample_rate (global, or a per-app dict), throttle (max evals per minute), and cost_budget (daily USD cap).

Sampling is deterministic — a SHA-256 hash of record_id salted with app_name — so decisions are idempotent across retries and replays. Only the ingest path is gated: explicit compute_metrics() / compute_now() still evaluate everything, and batch backfills don't charge against the daily cost budget.

For records in scope, an EVAL_DECISION span is emitted whether or not the record was evaluated, so get_records_and_feedback() surfaces sampled, sample_rate, and eval_decision_reason columns and you can measure your actual eval coverage. Apps not covered by a config evaluate normally with no span overhead.

Also adds a reports_costs property to the Provider base (default False, overridden to True on OpenAI, LiteLLM, Google, and Cortex) so the cost budget knows which providers it can actually account for.

Refs #2630.

from trulens.core import TruSession

session = TruSession()

session.configure_online_eval(
    sample_rate={"prod-rag": 0.05, "staging-rag": 1.0},
    throttle=60,           # max evals/min
    cost_budget=25.00,     # daily USD cap
    feedbacks=[f_groundedness, f_context_relevance],
)

Dashboard coverage visibility and cost-warning display are follow-up work, as is alerting integration (#2619).

Examples

MCP tool instrumentation cookbook

MCP tracing and evaluation walkthrough (#2638 — @bhargavikalicheti)

A self-contained cookbook for tracing and evaluating Model Context Protocol tool calls. It starts a deterministic local MCP server over stdio using the official MCP Python SDK, discovers its tools through ClientSession, and invokes them from a LangGraph agent — capturing MCP span attributes (server name, tool name, arguments, output, error status, execution time) along the way, then running Tool Selection and Tool Calling evaluations on the result.

The bundled weather/temperature-conversion server needs no network access or external credentials; OpenAI is used only by the agent and the LLM-based metrics. Ships with integration test coverage.

Closes #2415.

Docs

• Rebuilt trulens.org home page (#2647 — @joshreini1) — closes #2413.
• Feedback providers page, plus a custom base_url guide (#2602 — @seven7763) — documents pointing the OpenAI provider at any OpenAI-compatible Chat Completions endpoint (vLLM, Ollama shims, OpenRouter, Together, Fireworks, DaoXE, reverse proxies) via base_url, with the equivalent LiteLLM route. Also surfaces the previously-unlinked Feedback Providers page in the docs nav.
• Stop baking stat values into the home page markup (#2659 — @joshreini1) — homepage numbers no longer go stale silently.
• Strict docs build is now warning-clean (#2646 — @joshreini1) — Docs CI can pass without grep-filtering expected warnings out of the log.

Bug Fixes

• Repair Bedrock Nova tests, eval_gate formatting, and a dangling docs link (#2654 — @joshreini1)

New Contributors (3)

@seven7763, @bhargavikalicheti, @Oxygen56 — every feature in this release came from a first-time contributor.

One-line takeaway: 2.11.0 makes Claude a native judge (trulens-providers-anthropic) and makes always-on evaluation affordable (configure_online_eval() with sampling, throttling, and a daily cost budget).

Full Changelog: https://github.com/truera/trulens/compare/trulens-2.10.0...trulens-2.11.0