TruLens 2.9.0

The TruLens community tent got a lot bigger in this release, and we all get to benefit. Fourteen new contributors and many other repeat contributors came together over the last couple months to give TruLens new capabilities for maturing your evaluation system, more robust and comprehensive tracing, and lots of new documentation and examples for CI/CD, agentic evals, and MCP instrumentation. Most of the new eval tooling even came from first-time contributors.

New Features

Ensemble & comparative evaluation

Jury — ensemble LLM judges into one feedback callable (#2517) Wraps N providers, calls the same named method on each in parallel, aggregates scores. Because __call__ mirrors the underlying method's signature, it drops straight into Metric(implementation=jury) with no pipeline changes. Per-juror breakdowns flow into meta["reason"] → OTEL spans + dashboard.

from trulens.core import Metric
from trulens.feedback.jury import Jury
from trulens.providers.openai import OpenAI
from trulens.providers.litellm import LiteLLM

jury = Jury(
    jurors=[
        OpenAI(model_engine="gpt-4o-mini"),
        OpenAI(model_engine="gpt-4.1-mini"),
        LiteLLM(model_engine="anthropic/claude-3-haiku-20240307"),
    ],
    method="relevance",
    aggregation="median",  # mean | median | trimmed_mean | majority_vote | weighted_mean
)
m = Metric(implementation=jury, name="Jury Relevance").on_input().on_output()

(Follow-up #2518 fixed positional-argument forwarding through the jurors.)

CriteriaABTest — compare two judge configurations (#2564) Scores a golden set with two feedback-function variants and reports which is closer to expected scores (MAE + significance). Good for "does this prompt/model produce better judges?"

from trulens.benchmark.criteria_ab_test import CriteriaABTest

ab = CriteriaABTest(
    golden_set=[
        {"query": "who invented the lightbulb?", "expected_response": "Thomas Edison", "expected_score": 1.0},
    ],
    variant_a={"fn": provider.relevance, "name": "gpt-4o"},
    variant_b={"fn": provider.relevance, "name": "gpt-4o-mini", "kwargs": {"model_engine": "gpt-4o-mini"}},
)
report = ab.run()
report.summary()  # prints winner + MAE delta at alpha

ScoreDistributionAnalyzer — feedback-function diagnostics (#2562) Runs a feedback function over a golden set and reports score histograms + calibration (predicted vs. expected). Catches degenerate judges that always score ~0.95 or are miscalibrated.

Benchmarking & curation

GoldenSetGenerator — curate golden sets from production records (#2598) Samples records out of a TruSession for annotation. Three strategies: random, stratified (low/mid/high score buckets), and uncertainty (scores near a decision boundary). Returns a GoldenSetSample with expected_score left blank.

from trulens.benchmark.golden_set_generator import GoldenSetGenerator

gen = GoldenSetGenerator(session, seed=42)
sample = gen.sample(
    n=50,
    app_name="my_rag",
    strategy="uncertainty",        # surface the ambiguous cases
    feedback_name="Answer Relevance",
    decision_boundary=0.5,
    max_records=10_000,
)
sample.to_json("golden_50.json")   # annotate expected_score, feed into CriteriaABTest

Leaderboard output format options (#2535) — configurable output formats for benchmark leaderboards.

Agent & memory evaluation

MemoryRecallMetric / GroundTruthAgreement.memory_recall (#2526) Evaluates whether an agent's memory store surfaces the right stored memories — distinct from RAG recall_at_k. Ground-truth entries carry expected_memories and optionally conversation_id to scope lookups; similarity_threshold < 1.0 enables fuzzy matching.

from trulens.core import Metric, Selector
from trulens.feedback import GroundTruthAgreement
from trulens.providers.openai import OpenAI

golden_set = [
    {
        "query": "What are the user's preferences?",
        "expected_memories": ["User prefers dark mode", "User likes Python"],
        "conversation_id": "conv_123",
    }
]
gta = GroundTruthAgreement(golden_set, provider=OpenAI(), conversation_id="conv_123")

feedback = Metric(
    implementation=gta.memory_recall,
    name="Memory Recall",
    selectors={
        "query": Selector.select_record_input(),
        "retrieved_memories": Selector.select_record_output(),
    },
)

OTEL & conversational tracing

Semconv aligned with OTEL GenAI spec — Phase 1 dual-emit (#2463) — TruLens span attributes now dual-emit alongside the standard gen_ai.* semconv so OTel-native tooling interoperates.

conversation_id for thread-based trace grouping (#2462) — settable on the app context manager; emitted as ai.observability.conversation_id on the record-root span.

with tru_app(conversation_id="conv-abc-123") as recording:
    result = chain.invoke("follow-up question")

Dashboard thread-based grouping (#2527) — records sharing a conversation_id render as a single thread; ungrouped records stay standalone.

Providers & runtime

• Google multimodal support (#2596)
• Python 3.13 support for the Snowflake connector + Cortex provider (#2607)
• Bedrock: Amazon Nova serialized with Messages API schema (#2566) — see Bug Fixes below since it was a serialization bug fix.

Dashboard

• Violin plot + threshold line on metric histograms (#2460)

Bug Fixes

• Jury positional-argument forwarding (#2518) — args now forwarded correctly to jurors.
• Bedrock: serialize Amazon Nova with the Messages API schema (#2566) — Nova requests were malformed under the old schema.
• instrument_method is now idempotent (#2603, closes #2477) — re-instrumenting a class no longer double-wraps / corrupts the method.
• Preserve metadata for cancelled async spans (#2610) — cancelled async spans were dropping their attributes.
• Nested record-root span linkage (#2583) — nested TruApp calls now link their record-root spans correctly instead of producing disconnected traces.

Refactors & chores

• Standardize Provider __init__.py exports (#2525)
• Fix ruff lint warnings: B026, B028, TRY004 (#2438)
• build(deps): bump pip group across 1 directory, 10 updates (#2613)

Docs & Examples

• Update README with OTEL tracing, agentic evals, providers, MCP (#2456)
• Agentic evaluations guide — all 7 evaluators (#2455)
• MCP instrumentation guide with working example (#2457)
• Custom non-LLM feedback functions cookbook (#2458)
• Feedback template reference page (#2587, closes #2466)
• LLM judge benchmarking cookbook (#2605)

CI/CD eval-gate examples (new this release)

• GitHub Actions eval gate (#2586, closes #2482)
• Azure Pipelines eval gate (#2589, closes #2480)
• pre-commit eval smoke test (#2594, closes #2483)

Tests

• Feedback template validation coverage (#2516)
• Full-pipeline RAG triad OTEL integration test (#2522)
• Score parsing through full pipeline (#2568, #2496)
• Method-to-template wiring verification (#2599)
• Feedback score parsing pipeline (#2554)

New Contributors (14)

@1Utkarsh1, @AditSKumar, @knight0940, @Animesh452, @maitraymukeshkumarmodi-aiml, @kimnamu, @desusaiteja, @aqilaziz, @Aftabbs, @thunderstornX, @RyanFlowerYes, @yubingz, @xuyuanfan, @YingzuoLiu — each shipped their first PR this release. Strong community signal: most of the new eval-tooling features (#2562, #2564, #2526) came from first-time contributors.

One-line takeaway: 2.9.0 turns "LLM-as-a-judge" into a real evaluation program — ensemble your judges (Jury), A/B them (CriteriaABTest), diagnose them (ScoreDistributionAnalyzer), source golden data from prod (GoldenSetGenerator), and evaluate agent memory (memory_recall) — while OTEL aligns with the GenAI spec and your traces become conversational via conversation_id.