TruLens 2.10.0

The judge-quality tooling that 2.9.0 introduced (Jury, CriteriaABTest, ScoreDistributionAnalyzer, GoldenSetGenerator) gets two major additions this release: AlignmentReport and CrossModelAlignment. Together they give you a full diagnostic loop — build a jury, A/B test prompt variants, check score distributions, and now formally measure how well any judge aligns with a ground-truth benchmark and where different models diverge from each other.

OTEL tracing takes another step toward standards compliance with correct SpanKind values per the OpenTelemetry GenAI spec and a new span_group() context manager for localizing metrics to specific pipeline segments. And a new citation_attribution feedback function extends RAG evaluation beyond groundedness to source attribution.

Eight new contributors shipped every feature in this release. The community is building the eval tooling.

New Features

Judge alignment diagnostics

AlignmentReport (#2577 — @furk4neg3)

Measures how well an LLM judge agrees with a ground-truth benchmark dataset. Reports per-label agreement metrics and surfaces systematic biases — e.g. a judge that consistently over-scores low-quality responses. Use it before promoting a judge to CI/CD.

from trulens.benchmark.alignment_report import AlignmentReport

report = AlignmentReport(
    golden_set=golden_data,
    feedback_fn=provider.relevance,
)
report.run().summary()

CrossModelAlignment (#2563 — @thunderstornX)

Runs the same inputs through multiple judge models and reports where they diverge. Practical use: pick the cheapest model whose scores stay within acceptable delta of your reference judge. Complements CriteriaABTest (which tests prompt/config variants against a golden set) — CrossModelAlignment tests model identity.

from trulens.benchmark.cross_model_alignment import CrossModelAlignment

alignment = CrossModelAlignment(
    golden_set=golden_data,
    models=[
        OpenAI(model_engine="gpt-4o"),
        OpenAI(model_engine="gpt-4o-mini"),
        LiteLLM(model_engine="anthropic/claude-3-haiku-20240307"),
    ],
    method="relevance",
)
alignment.run().summary()

New feedback function

citation_attribution (#2576 — @rsrijith)

Evaluates whether an LLM response correctly attributes claims to the provided source material. Distinct from groundedness (is the answer factually supported?) — citation attribution asks whether sources are explicitly and accurately credited. Rounds out the RAG triad with a fourth attribution dimension.

from trulens.core import Metric
from trulens.feedback.templates.rag import CitationAttribution

f_citation = Metric(
    implementation=provider.citation_attribution,
    name="Citation Attribution",
).on_context().on_output()

OTEL tracing improvements

SpanKind per OpenTelemetry GenAI conventions + resource attributes (#2633 — @Payal2000)

Instrumented spans now emit correct SpanKind values (e.g. CLIENT for LLM calls) and include resource attributes aligned with the OTel GenAI semantic specification. Improves out-of-the-box compatibility with OTEL-native observability backends (Jaeger, Honeycomb, Grafana Tempo, etc.).

span_group() context manager for per-segment metric localization (#2637 — @Payal2000)

Groups spans within a logical segment so feedback functions can target a specific step rather than the full trace. Useful for multi-turn pipelines where you want turn-level quality scores, or multi-stage pipelines where you want stage-level evals.

from trulens.core.otel.instrument import span_group

with span_group("retrieval_step"):
    contexts = retriever.retrieve(query)

with span_group("generation_step"):
    answer = llm.generate(query, contexts)

Bug Fixes

• Fix OSS log ingestion client metrics (#2634 — @joshreini1) — LOG_INGESTION metrics were silently missing when using the local Run API client.
• Fix dashboard tag filter splitting app tag strings into characters (#2617 — @feiiiiii5) — multi-character tag names now filter correctly in the app selector.
• Fix OpenAI judge score extraction from custom_tool_call output items (#2636 — @Anikhet) — score parser now handles the output_items structure returned by the OpenAI responses API. Closes #2631.

Tests

Four new test suites substantially expand provider and pipeline coverage:

• Bedrock provider capability tests (#2536 — @OrionArchitekton)
• HuggingFace provider capability tests (#2530 — @Animesh452)
• Cortex provider capability tests (#2537 — @OrionArchitekton)
• Feedback pipeline wiring integration coverage (#2604 — @aqilaziz). Closes #2494.
• Prompt-template rendering integration tests (#2549 — @wgu9). Closes #2493.

Docs

• Fix broken links in 2.6 release blog post (#2641 — @devdattatalele)

Dependencies

• npm/yarn bumps across 3 directories (#2611 — @dependabot)

New Contributors (8)

@furk4neg3, @Payal2000, @devdattatalele, @feiiiiii5, @rsrijith, @Anikhet, @OrionArchitekton, @wgu9 — every feature in this release came from a first-time contributor.

One-line takeaway: 2.10.0 closes the judge quality loop — align judges to benchmarks (AlignmentReport), compare them across models (CrossModelAlignment), attribute sources in responses (citation_attribution), and localize OTEL metrics to pipeline segments (span_group()).

Full Changelog: trulens-2.9.0...trulens-2.10.0

Full Changelog: https://github.com/truera/trulens/compare/trulens-2.9.0...trulens-2.10.0