DeepEval metric checklist
One row for each DeepEval metric, classifier and helper, with how trialkit compares. Snapshot: 2026-11-12.
65 metrics with sources. trialkit is ahead on 2 and at parity on 40. It is behind on 18, each with a plan. By group: Judge types and custom metrics (3); RAG and grounding (1); Agents (1); Safety and text quality (5); MCP (3); Images and voice (1); RAGAS adapters and legacy helpers (1); Classifiers (3). It leaves 5 out of scope.
The Tool comparison has the versions, the notes, the other DeepEval rows and the method. The data is in comparison.json.
DeepEval metric rows compare the property that a metric evaluates, not its number. Most numeric DeepEval metrics score from 0 to 1 and pass at a threshold that defaults to 0.5. Exceptions in this matrix: Exact Match, Pattern Match and Tool Permission default to a threshold of 1, the RAGAS adapters default to 0.3 in the source, Arena G-Eval returns a winner and no score, and classifiers return a label. A trialkit judge returns a pass or fail verdict or a label from a versioned template, and a scorer returns its own value, so scores from the two tools are not comparable. A pending recipe that is not merged counts as a gap.
Status shows how trialkit compares with DeepEval:
- ahead: trialkit does it in its open-source packages. The cited DeepEval sources show nothing equal, or show it only with an account.
- parity: both do it at a similar level.
- gap: DeepEval does it and trialkit does not yet. The status links the plan, or says planned.
- out of scope: DeepEval does it. trialkit leaves it to other tools by design (non-goals).
(unverified) marks a claim that the cited source did not establish, such as a feature that was not found. It does not prove that the feature is absent. It also marks a trialkit claim about work that is not merged: only the pull request states it, so the claim stays unverified until the pull request merges.
Judge types and custom metrics
Section titled “Judge types and custom metrics”| Feature | Status | DeepEval | trialkit |
|---|---|---|---|
| Custom criteria judge (G-Eval criteria) | parity | Python and TypeScript: GEval and ConversationalGEval judge any criteria that you write in plain words. You choose the fields that the judge sees, such as the input, output and expected output; single-turn G-Eval can instead judge the whole recorded trace, and the conversational form sees the turns. The result is a score from 0 to 1 with a reason source 2 |
rubricJudge passes or fails an output against a rubric that you write. classifierJudge picks one of your labels, and each label has a score from 0 to 1. Pickers choose the input, output, expected value and context; for a conversation, put the serialized turns in the output. Each verdict has a reason from the model reply. The verdict is binary or a label score, not a 1 to 10 grade source 2 |
| Judge configuration | parity | A G-Eval metric takes a model (an OpenAI model or any custom model class), criteria in plain words, the fields that it judges, and an evaluation template that replaces the default prompts source | rubricJudge and classifierJudge take a model client, a rubric (a classifier also takes labeled choices with scores), optional pickers for the input, output, context and expected value, and a template that you can replace. Templates are versioned, and a score records the template version and a hash of its system prompt. A temperature is sent only when you set it source |
| Judge calibration against human labels | ahead | No library-level calibration report found. The FAQ lists these remedies for noisy judge scores: verbose mode, strict mode, DAG metrics, a custom evaluation template and a stronger judge model. The hosted Confident AI platform has an alignment page (see the hosted annotation row) (unverified) source | calibrate(judge, labeled) runs a judge over human-labeled outputs and reports agreement with its 95% Wilson interval, Cohen’s kappa, a confusion matrix, each disagreement and each error, and the judge identity (template, model, prompt hash, settings hash) that eval scores can be checked against. It refuses a calibration whose cases were scored by different judges. The labels are your own data, and recorded judge replies work in CI source 2 |
| G-Eval workflow (evaluation steps and score controls) | gap (planned) | Python and TypeScript: G-Eval takes criteria or evaluation steps, not both. It turns criteria into steps, or follows the steps that you supply, and grades the output from 1 to 10 against them. An optional rubric of score bands confines the score range. The pass threshold defaults to 0.5, and strict mode makes the score binary and sets the threshold to 1. G-Eval weights the grade by the model’s token probabilities when the model returns them, and uses the raw integer otherwise. ConversationalGEval does the same over a whole conversation source 2 |
No steps workflow. A judge sends the rubric or labels that you write, with no generated or frozen steps, and returns a verdict or a label, not a 1 to 10 grade. It has no score bands and no strict mode: a classifier passes a label when its score is at least 0.5, or when your pass function says so. The model port returns text and usage, with no token probabilities. Probability weighting needs a model-port contract first source 2 |
| Pairwise judging (Arena G-Eval) | gap (planned) | Python and TypeScript: ArenaGEval compares contestants, which are test cases from competing versions of your app, with a blinded, randomized-position, pairwise judge. It returns a winner and a reason, with no score and no threshold source 2 |
No pairwise judge. classifierJudge can pick A, B or tie from two texts that you put in the output, but nothing blinds the contestants or balances their order. trialkit compare compares the results of two separate runs; it is not a pairwise judgment source 2 |
| DAG execution (decision-graph judge) | gap (planned) | Python and TypeScript: DAGMetric and ConversationalDAGMetric run a graph of task, judgment and verdict nodes. Each node asks the model one narrow question, the answers choose a branch, and a verdict node gives the score or hands the case to another metric. The conversational form selects turn windows source 2 3 |
No graph judge. A judge asks the model once and returns one verdict or label, so a branching decision procedure is your own scorer code source 2 |
| Custom metric class (BaseMetric) | parity | Python and TypeScript: you subclass BaseMetric (or BaseConversationalMetric) and write any algorithm. The docs show a ROUGE metric and a composite metric source 2 |
A Scorer has a name, a version and a score function that returns a value from 0 to 1, a pass flag and an optional reason. It can run any algorithm, also an async one, in TypeScript and Python. Change version when the logic changes source 2 |
| Jev metrics and modes (JevEval, hybrid, system_one) | out of scope | Python and TypeScript: JevEval and ConversationalJevEval score bounded questions with Jev, a System One model that returns probabilities, not an LLM. Many built-in metrics also have hybrid and system_one modes that replace the decisions of the LLM with Jev’s probabilities source 2 3 |
No Jev integration. The model port returns text and no probabilities, and no package bundles or depends on a scoring model. A classifier label score is a value that you set, not a probability. You can run an external scorer inside a core Scorer source 2 |
RAG and grounding
Section titled “RAG and grounding”| Feature | Status | DeepEval | trialkit |
|---|---|---|---|
| Answer relevancy | parity | Python and TypeScript: extracts the statements of the output and scores the share that are relevant to the input. Needs the input and the output source 2 | answerRelevanceV2 (answer_relevance_v2 in Python) is a metric for metricJudge. It takes the request and the answer, with no sources. The answer must address the requested topic and every material subquestion, and an appropriate clarification can pass. It does not assess truth, so a relevant but false answer can pass. The verdict is pass or fail, not a share of relevant statements. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Faithfulness | parity | Python and TypeScript: extracts the truths of the retrieved context and the claims of the output, and scores the share of claims that the truths do not contradict. Needs the input, the output and the retrieved context source 2 | faithfulnessV2 (faithfulness_v2) is a metric for metricJudge. It takes the request, the answer and the sources, which the judge numbers. Every factual claim must follow from the sources: a paraphrase can pass, and an unsupported addition, contradiction, changed number or negation fails. A nonempty explicit abstention can pass. This is stricter than the non-contradiction test of DeepEval, and the verdict is pass or fail. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Context sufficiency (answerability) | ahead | No metric found that judges, without a reference answer, whether the retrieved context is enough to answer the input. Contextual Recall comes closest: it extracts the statements of an expected output and scores the share that the retrieved context supports, so it needs an expected output. Contextual Relevancy scores relevance, not sufficiency (unverified) source 2 3 4 | contextSufficiencyV2 (context_sufficiency_v2) is a label metric for metricJudge. It takes the request and the sources, which the judge numbers, never the answer, and needs no reference answer. Its template fixes the ordered labels and scores { sufficient: 1, insufficient: 0, conflicting: 0 }. insufficient includes empty, irrelevant or partial sources. conflicting means an unresolved material contradiction, which takes precedence over missing evidence, and an explicit superseding policy resolves an older conflict. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Contextual relevancy | parity | Python and TypeScript: extracts the statements of the retrieved context and scores the share that are relevant to the input. The score is relevant statements over all statements. The page lists the input, the output and the retrieved context as required arguments, but the calculation reads only the input and the retrieved context. The default threshold is 0.5. Strict mode needs a score of 1, so every statement that the LLM extracts must be relevant. The page says that it extracts all statements, but the prompt in the Python source asks for the high-level information of each context, so a detail that it does not extract is not judged source 2 3 | contextRelevanceV2 (context_relevance_v2) is a metric for metricJudge. It takes the request and the retrieved passages as sources. It does not take the answer. Every substantive statement in the passages must help answer the question or give necessary background. Unrelated material fails. One relevant but incomplete passage can pass, because the check is for noise, not for enough evidence. Empty or all-blank passages fail. One judge call returns a pass or fail verdict with a reason. There is no statement extraction and no share. DeepEval passes at a share of 0.5 by default. Its strict mode needs a score of 1, but only over the statements that its LLM extracts, so irrelevant material that it does not extract can pass. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Contextual precision (ranking) | parity | Python and TypeScript: judges each retrieved node against the input and the expected output. It scores weighted cumulative precision, the mean of the precision at the rank of each relevant node, so relevant nodes that rank early score higher. Nodes with the same source count as one node when RetrievedContextData is used. By the equation, the score is 1 only when no irrelevant node comes before a relevant one, and the source scores 0 for an empty list or when no node is relevant. Needs the input, the output, the expected output and the retrieved context. The default threshold is 0.5. Strict mode needs a score of 1 source 2 3 |
contextRankingV2 (context_ranking_v2) is a metric for metricJudge. It takes the request, the factual reference answer as reference (never the human label) and the retrieved passages in their original order as sources. A passage is relevant when it supports a question-relevant fact of the reference answer. The verdict passes only when at least one passage is relevant and every relevant passage comes before every irrelevant one. One inversion fails, at any position. An empty list or a list with no relevant passage fails. A relevant duplicate below an irrelevant passage is an inversion. This matches the DeepEval score of 1, which strict mode needs, and nothing between 0 and 1: there is no rank weighting, no partial credit and no top-k option. Each passage keeps its own slot, where DeepEval can group nodes that share a source. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Contextual recall (reference coverage) | parity | Python and TypeScript: extracts the statements of the expected output and scores the share that can be attributed to the retrieved context. The score is attributable statements over all statements. It uses the expected output, not the actual output, because it measures the retriever. Needs the input, the output, the expected output and the retrieved context. The default threshold is 0.5. Strict mode needs a score of 1 source 2 | referenceCoverageV2 (reference_coverage_v2) is a metric for metricJudge. It takes the request, the factual reference answer as reference (never the human label) and the retrieved passages as sources. It does not take the answer. The passages together must support every factual assertion of the reference answer, without an unresolved contradiction about that assertion. Partial coverage fails. One passage passes only if it supports every assertion. Empty or all-blank passages fail, and blank or duplicate entries add no support. The verdict is pass or fail. This matches the DeepEval score of 1, which strict mode needs, not the share of attributable statements. The rule about an unresolved contradiction is an addition: the DeepEval page asks only whether each statement can be attributed. The check covers a reference answer. contextSufficiencyV2 is the separate check that needs no reference. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Hallucination (ideal-context alignment) | parity | Python and TypeScript: judges, for each ideal context, whether the output contradicts it. It scores the share of contexts that the output stays consistent with, so higher is better and the threshold (default 0.5) is a minimum. Strict mode needs a score of 1. The ideal contexts are curated ground truth that you supply, not retrieved text, and for a live RAG system the page sends you to Faithfulness. Needs the input, the output and the ideal context source 2 | contextAlignmentV2 (context_alignment_v2) is a metric for metricJudge. It takes the request, the answer and the curated ideal contexts as sources, not retrieved passages. It needs no reference. The answer must not contradict any source: a changed fact, number or negation fails. Unsupported but noncontradicting additions can pass, and so can a nonempty nonfactual abstention. The check is for contradiction, not entailment. An empty answer fails, and so do empty or all-blank contexts. One verdict covers all contexts. This matches the DeepEval score of 1, which strict mode needs, not the share of aligned contexts. faithfulnessV2 is stricter: it fails an unsupported claim, and it reads retrieved sources, not ideal contexts. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Citation faithfulness | parity | Python, as a community metric that you import from deepeval.metrics.community: a binary judge. The verdict is faithful (score 1) only when every factual claim is supported and every [N] marker points to a passage that supports its attached claim. DeepEval numbers the retrieved passages before the judge sees them. Needs the input, the output and the retrieved context. The default threshold is 1. The page’s code example is Python, and the published TypeScript 0.9.25 exports no such metric. The page does not say how it treats a marker such as [1,2] or a marker with no passage source 2 |
citationGroundingV2 (citation_grounding_v2) is a metric for metricJudge. It takes the request, the answer with its citation markers and the retrieved passages as sources. It needs no reference. Every factual claim must follow from the passages, and each citation must point to a passage that supports its attached claim. The citation dialect is explicit. Standalone [1] and [2] markers name 1-based passage slots, and adjacent [1][2] are two citations. An unknown or out-of-range target fails, and so does other syntax such as [1,2]. Repeated correct markers pass, and a supported claim can have no citation. An empty answer fails. With empty or all-blank passages, only an uncited nonfactual abstention passes. You normalize other citation formats to this dialect before you build the evidence, because the metric has no citation parser; the judge numbers the passages as canonical JSON. Like DeepEval, it returns a pass or fail verdict. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Reference entity coverage | gap (planned) | Python: the RAGAS adapter RAGASContextualEntitiesRecall in the DeepEval source delegates to the installed RAGAS version to score how many entities of the expected output the retrieved context covers. The docs page lists four RAGAS components and omits it source 2 |
No ready-made recipe. setMatch scores entity coverage only when you already extracted the entities of both sides as sets source 2 |
Agents
Section titled “Agents”| Feature | Status | DeepEval | trialkit |
|---|---|---|---|
| Task completion | parity | Python and TypeScript: extracts the task, or takes one that you give, and the outcome from a recorded trace or, in an end-to-end run, from the test case, and scores how well they align source 2 | taskCompletionV2 (task_completion_v2) is a metric for metricJudge. It takes the request and the successCriteria, which the judge numbers, and it grades the recorded spans. Every criterion needs observed evidence in the recorded final message or tool results. A claim of success is not proof, and partial completion or absent confirmation fails. A recovered tool error can pass when later evidence confirms every criterion. You supply the task and the criteria; DeepEval infers the task from the trace when you give none. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Step efficiency | parity | Python and TypeScript: a trace-only metric. It extracts the task from the trace, then scores how well the execution steps align with that task. It penalizes any action that was not strictly required, and its FAQ lists duplicates, retries and detours. You cannot run it on a standalone test case. The default threshold is 0.5. Strict mode needs a score of 1 source 2 | stepEfficiencyV2 (step_efficiency_v2) is a metric for metricJudge. It takes the request and the tools. The judge reads the recorded spans in record order, not the task output. Record order shows the order of recording, not elapsed time or causality. Every recorded substantive step must be justified by the task and by the evidence available when it was taken. Repeating a successful lookup, including a safety check, without a new need fails. A retry that the record justifies, such as a retry after a failed call, can pass, where the DeepEval FAQ lists retries among the penalized actions. A necessary verification or safety check can pass too, and so can a model-only run. Copied message history and a call that a model message only describes are not extra executed actions. Opaque evidence that does not show whether a step was justified fails. A wrong final answer can still pass, because completion is a separate check. A run with no recorded spans is an error. The verdict is pass or fail, not a score. You supply the task; DeepEval extracts it from the trace. toolPolicy stays the deterministic check for a call cap or an identical repeat. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 |
| Plan adherence | parity | Python and TypeScript: a trace-only metric. It extracts the task and the plan from the trace, taking the plan from the thinking or reasoning of the agent, and scores how well the execution steps follow the plan. When it finds no plan, it passes with a score of 1. The default threshold is 0.5. Strict mode needs a score of 1 source 2 3 | planAdherenceV2 (plan_adherence_v2) is a metric for metricJudge. It takes the request, the tools and the planSpanId. You choose planSpanId before the run and record the actual public planner response in that span’s gen_ai.output.messages, so the plan is one fixed text. The judge reads the recorded spans, never hidden reasoning. Every applicable commitment and dependency of the plan needs recorded proof, and a material unplanned action fails. An equivalent implementation can pass, meaning a different recorded call that performs the same planned operation on the same inputs. An ordering that the plan does not constrain can pass: plan numbering alone does not order steps that do not depend on each other. A conditional branch that the plan allows and the record supports can pass. Replanning has no automatic exemption. A bad plan that the run follows faithfully can pass, and a tool failure alone does not necessarily break adherence. A missing designated span, a blank plan and a plan that comes after the actions fail. DeepEval passes with a score of 1 when it finds no plan. A run with no recorded spans is an error. Record order shows the order of recording, not elapsed time or causality. The verdict is pass or fail, not a score. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Plan quality | parity | Python and TypeScript: a trace-only metric. It extracts the task and the plan from the trace, taking the plan from the thinking or reasoning of the agent, and scores how well the plan fits the task. The catalog describes the property as a plan that is logical, complete and efficient. When it finds no plan, it passes with a score of 1. The default threshold is 0.5. Strict mode needs a score of 1 source 2 3 4 | planQualityV2 (plan_quality_v2) is a metric for metricJudge. It takes the request, the tools and the public plan. The judge reads no execution results and no hidden reasoning, and it does not infer a plan from later actions. The plan must be actionable and possible with the listed tools, cover the task and its prerequisites, be in a logical order and have no unnecessary step. A concise one-step plan can pass. A blank or vague plan, a missing prerequisite, an impossible capability and a dependency inversion fail. The judge assesses the plan as written, without hindsight from the outcome. A blank plan fails, and a missing plan is a caller error. A missing plan does not pass by default. DeepEval scores 1 when it finds no plan. The verdict is pass or fail, not a score. You supply the public plan, for example the output.plan of your task; DeepEval extracts a plan from thinking or reasoning. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Tool matching: names, arguments and order | parity | Python and TypeScript: compares the tools that the agent called with the expected tools. It always matches names. Options also match input parameters and output, check order, or require an exact match. The score is correct tools over tools called source 2 | toolCalls (tool_calls in Python) compares { name, args } lists with order exact, inOrder or any, and with arguments equal, subset or ignore. Non-empty recorded spans take precedence over the output. The score is matched calls over the longer list (exact) or over the expected calls, and DeepEval divides by the tools called, so the scores differ source 2 3 |
| Tool matching: tool results | parity | Python and TypeScript: the OUTPUT evaluation parameter of ToolCorrectnessMetric also requires the output of each called tool to match the output of the expected tool. It is off by default. The Python source compares the two outputs with plain equality in every mode (exact, partial and ordered), so it has no subset mode source 2 3 |
toolCalls({ results }) (tool_calls(results=...) in Python) compares the JSON result of each call. 'equal' needs the same value, and 'subset' allows extra object fields at every depth, with arrays of the same length and order. With either, every expected call needs a result (null is valid), and a recorded call with no result never matches, nor does a call whose span status is error. It works with every call order and independently of args. Strings compare as recorded: trialkit does not parse a JSON string, and whitespace counts. The default 'ignore' skips results. A mismatch fails the scorer, and the value is the fraction of matched calls source 2 3 4 5 |
| Semantic tool use (selection and arguments) | parity | Python and TypeScript: ArgumentCorrectnessMetric judges whether the arguments of each tool call are correct for the input. ToolCorrectnessMetric can also judge, against a list of available tools, whether the chosen tools were the most appropriate, and takes the lower of its two scores source 2 3 |
toolAppropriatenessV2 (tool_appropriateness_v2) is a metric for metricJudge. It takes the request and the tools, a list of { name, description, inputSchema }. The judge checks the ordered recorded decisions, calls, arguments and results against the request and those tools. A schema-valid argument with the wrong meaning fails, an unnecessary action or an omitted necessary tool fails, and justified execution without tools can pass. toolCalls still checks exact or subset arguments against a reference. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Agent loop detection | gap (planned) | Python, as a community metric: a deterministic score from a recorded trace. It weights repeated identical tool calls (0.40), stagnant consecutive LLM outputs (0.35) and recursive span cycles (0.25), and rebalances the weights of the checks that you enable source 2 | toolPolicy with allowRepeated: false rejects repeated identical tool calls, which is one of the three checks. Nothing detects stagnant outputs or recursive span cycles source 2 3 |
| Tool permission | parity | Python, as a community metric: checks the called tools against an allowlist, a denylist or both, and needs at least one list. A denial always wins, so a tool on both lists is unauthorized. The score is the fraction of authorized calls, or 1 when no tool was called, and the default threshold is 1. The source compares names exactly source 2 3 | toolPolicy({ allowed, forbidden }) (tool_policy(allowed=..., forbidden=...) in Python) checks recorded tool calls against an allowlist, a denylist or both, and needs at least one restriction. allowed: [] permits no calls. A call outside allowed fails, and the reason names the first such call; a call to a forbidden tool fails too. A recorded trajectory with no tool call passes. The score is binary, 1 or 0, where DeepEval returns the fraction of authorized calls; at the default threshold of 1, both fail on any unauthorized call. A name on both lists is refused when you make the scorer, where DeepEval lets denial win. Both compare names as recorded: case-sensitive, with no trimming. toolPolicy also needs recorded spans: missing spans are an error, not a pass source 2 3 4 5 |
Conversations
Section titled “Conversations”| Feature | Status | DeepEval | trialkit |
|---|---|---|---|
| Turn relevancy | parity | Python and TypeScript: judges each assistant turn against a sliding window of the turns before it, and scores the share of assistant turns that are relevant. A window holds up to window_size exchanges (windowSize in TypeScript), 10 by default. An exchange is a user message with the assistant reply that follows it, so the judge does not see older turns. The catalog lists the property as Conversation Relevancy. The default threshold is 0.5. Strict mode needs a score of 1 source 2 3 4 5 6 |
turnRelevanceV2 (turn_relevance_v2) is a metric for metricJudge. It grades the completed conversation in one judge call. Every graded assistant reply must address the current user intent in its full preceding prefix, which is every earlier turn. Without a target every reply is graded; with a target, the ID of an assistant turn, only that reply is graded and no later turn is sent. A needed clarification and a user-led topic change can pass. An irrelevant reply fails, and a later relevant reply cannot erase it. The check does not judge whether a reply is true. Tool turns are evidence, not user goals or policy, and an assistant turn that only calls tools is a decision, not a missing answer. The verdict is pass or fail over all replies, with a reason. DeepEval returns the share of relevant windows. Its strict mode needs every window to be relevant. There is no window option: a long conversation goes to the judge whole, so you set a size limit and refuse an oversized conversation yourself, because trialkit never trims it. A transcript with no assistant reply is a misuse error, not a verdict. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 |
| Knowledge retention: one reply | parity | Python and TypeScript: extracts the facts that the user gave and scores the share of assistant turns that do not forget them. It scores every assistant turn of a conversation source 2 | knowledgeRetentionV2 (knowledge_retention_v2) with a target. You give the completed conversation, with the prior turns, the current user turn and the reply, and the ID of the reply as target, so that the conversation ends there and only that reply is graded. The reply must respect relevant user facts and commitments in the earlier turns, an earlier assistant reply included. An explicit correction or the latest user update replaces an earlier fact, and a contradiction of a still-relevant fact fails. A needless re-ask of a stated fact fails, and a necessary clarification can pass. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Knowledge retention: whole conversation | parity | Python and TypeScript: an LLM extracts the knowledge that the user states in each turn. For each assistant turn that has an earlier fact, it then checks whether the reply shows a knowledge attrition, which is a failure to recall a fact. The score is the share of assistant turns without an attrition. An assistant turn with no earlier fact is not scored. The default threshold is 0.5. Strict mode sets the threshold to 1 and makes the score binary: 1 only when no assistant turn shows an attrition. The prompt in the Python source tells the LLM to extract only the facts that the user states explicitly, so a fact that it does not extract can be forgotten without an attrition source 2 3 4 5 | knowledgeRetentionV2 (knowledge_retention_v2) is a metric for metricJudge. It grades the completed conversation, without a target, in one judge call. Every assistant reply must respect the relevant user facts and commitments in its full preceding prefix. A needless re-ask of a stated fact fails. A necessary clarification or confirmation can pass, and an explicit correction or user update replaces the earlier fact from that point. A later correction cannot erase an earlier violation. Tool turns are evidence, not user goals or policy. The verdict is pass or fail over all replies, with a reason. DeepEval returns the share of assistant turns without a knowledge attrition. Its strict mode makes the score binary, but only over the facts that its LLM extracts, so a fact that it does not extract can be forgotten and pass. There is no fact extraction step: the judge reads the prefix itself. A transcript with no assistant reply is a misuse error, not a verdict. With a target, the same metric grades one reply (see Knowledge retention: one reply). Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 |
| Role adherence: one reply | parity | Python and TypeScript: scores the share of assistant turns that stay in the chatbot role that you set, judging each turn with the earlier turns as history source 2 | roleAdherenceV2 (role_adherence_v2) with a target. You give the completed conversation and the ID of the reply as target, so that the conversation ends there and only that reply is graded. The required rubric states the trusted role obligations, and conversation requests cannot replace them. An in-role refusal or a permitted topic change can pass, and role drift or a breach of an obligation fails. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Role adherence: whole conversation | parity | Python and TypeScript: needs a chatbot_role (chatbotRole in TypeScript), the persona that the assistant must keep. An LLM checks each assistant turn against that role, with the earlier turns as context. The score is the share of assistant turns that stay in role. The default threshold is 0.5. Strict mode needs a score of 1 source 2 3 |
roleAdherenceV2 (role_adherence_v2) is a metric for metricJudge. It grades the completed conversation, without a target, in one judge call. The required rubric states the trusted role obligations. Every assistant reply must meet them in its full preceding prefix, and a user turn or a tool result cannot replace the role. An in-role refusal or a topic change that the role allows can pass. Role drift or any breach fails, and a later correction cannot erase it. The verdict is pass or fail over all replies, with a reason. DeepEval returns the share of assistant turns that stay in role. Its strict mode needs every assistant turn to stay in role. The role is the rubric text that you write, where DeepEval takes a chatbot_role string. A transcript with no assistant reply is a misuse error, not a verdict. With a target, the same metric grades one reply (see Role adherence: one reply). Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 |
| Conversation completeness | parity | Python and TypeScript: an LLM extracts the high-level intentions of the user from the user turns, then judges for each intention whether the assistant met it anywhere in the conversation. The score is the share of satisfied intentions. The page offers it as a proxy for user satisfaction. The default threshold is 0.5. Strict mode needs a score of 1, so every extracted intention must be met. The prompt in the Python source tells the LLM to consider the overall intention and not to dwell on the specifics, so a specific request that it does not extract is not checked source 2 3 4 | completenessV2 (completeness_v2) is a metric for metricJudge. It grades the completed conversation in one judge call. By the end, every uncancelled user request must have a substantive conversational resolution. This is closure of the dialogue, not proof that the goal was achieved: a justified refusal can resolve a request, and a promise to answer or act later is not delivery. A request that awaits a clarification stays incomplete, and a later reply can resolve an earlier request. A topic change does not cancel a request. Only an explicit cancellation does, and executing the cancelled action afterward fails. An acknowledged stop request with observed cessation can be resolved, and an explicit correction replaces the affected request. Tool turns are evidence, not user goals or policy. The verdict is pass or fail, with a reason. DeepEval extracts intentions and returns the share that the assistant satisfied. Its strict mode needs a score of 1, but only over the intentions that its LLM extracts, so a request that it does not extract can go unmet and pass. A transcript with no assistant reply is a misuse error, not a verdict. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 |
| Goal accuracy | parity | Python and TypeScript: finds the goal and the steps of each user-and-assistant exchange, taking the goal from the user messages and the steps from the assistant replies and their tools_called. It scores the goal of each exchange and the plan that the assistant followed, which covers plan quality and plan adherence. The final score is the mean of the average goal score and the average plan score. Needs only the turns. The default threshold is 0.5. Strict mode needs a score of 1 source 2 3 4 |
goalAccuracyV2 (goal_accuracy_v2) is a metric for metricJudge. It grades the completed conversation in one judge call. By the end, every active user goal must be achieved according to the visible replies and the recorded calls and results. A success claim or a promise alone does not prove an external action, and missing proof fails. A later successful result can recover an earlier tool error. A topic change does not cancel a goal. An explicit correction replaces the affected goal. An explicit cancellation removes the obligation, but executing the cancelled action afterward fails. An acknowledged stop request with observed cessation can succeed. A justified refusal does not achieve the goal, and conversational completeness is a separate check. Tool turns are evidence, not user goals or policy. The verdict is pass or fail, with a reason. DeepEval averages a goal score with a plan score. The plan has no part in this verdict: planQualityV2 and planAdherenceV2 stay separate checks of one trial, and trialkit does not average them in. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 |
| Tool use over a conversation | parity | Python and TypeScript: takes one available_tools list (availableTools in TypeScript) for the whole conversation. For each user-and-assistant exchange, it scores tool selection, and for each exchange with a tool call it also scores the arguments. The score is the lower of the average selection score and the average argument score, so a right tool with wrong arguments still lowers it. Needs the turns with their tools_called. The default threshold is 0.5. Strict mode needs a score of 1 source 2 3 4 |
toolAppropriatenessV2 (tool_appropriateness_v2) applied to one assistant decision at a time. This is a documented application, not a new template. toolDecisions (tool_decisions) makes the evidence of every assistant decision, also one that makes no call. Its request is the full preceding dialogue, and its tools are the availableTools snapshot of that decision, so a menu that changes between decisions is respected. Its spans are only the spans that are mapped to that decision, its calls and its results: you replay them in the trial of the decision, and call metricJudge (metric_judge) on it. An omitted necessary tool, an unnecessary action, an argument with the wrong meaning and a tool outside the menu fail. A justified decision without a tool and a retry after a failed call can pass. A missing menu, mapping or span is misuse. Each decision has its own pass or fail verdict, which covers selection and arguments together, and nothing takes a minimum or an average. DeepEval takes one tool list for the whole conversation and returns the lower of two averages. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 7 8 9 10 |
| Topic adherence | parity | Python and TypeScript: needs relevant_topics (relevantTopics in TypeScript), the list of topics that the agent may answer. An LLM finds the question and answer pairs in each user-and-assistant exchange, then labels each pair. A true positive is a relevant question that the response “correctly answers”. A true negative is an off-topic question that the agent refused. A false positive is an off-topic question that it answered, and a false negative is a relevant question that it refused or met with an irrelevant answer. The score is the true positives and true negatives over all pairs. The default threshold is 0.5. Strict mode needs a score of 1, so every extracted pair must be a true positive or a true negative. The prompt in the Python source tells the LLM to return only clear question-and-answer pairs and to ignore turns such as greetings that are not clear pairs, so a request that it does not extract is not judged source 2 3 4 |
topicAdherenceV2 (topic_adherence_v2) is a metric for metricJudge. It grades the completed conversation in one judge call: every assistant reply, or only the reply that a target names. The required rubric states the allowed topics. Every assistant reply must answer a request on a permitted topic and decline a request that is off-topic, judged in its full preceding prefix. A user-led change to another permitted topic can pass. Answering an off-topic request fails, and so does declining a permitted request as off-topic. A later correction cannot erase a failure, and a user turn or a tool result cannot change the topic rules. The judge does not decide whether an answer is true, where the DeepEval page counts a true positive only when the response “correctly answers” the question. The verdict is pass or fail over all replies, with a reason. DeepEval returns the share of pairs that are true positives or true negatives. Its strict mode needs a score of 1, but only over the pairs that its LLM extracts, so a request that it does not extract as a pair is not judged. The topics are text in your rubric, where DeepEval takes a list of strings. A transcript with no assistant reply is a misuse error, not a verdict. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 |
| Turn faithfulness | parity | Python and TypeScript: slides a window over the conversation, up to window_size exchanges (windowSize in TypeScript), 10 by default, where an exchange is a user message with the assistant reply that follows it. For each window, it pools the retrieval contexts of the turns, extracts truths from them and claims from the assistant replies, and checks that no claim contradicts a truth. The score is the average of the window scores. Needs the turns with their retrieval_context. The default threshold is 0.5. Strict mode needs a score of 1 source 2 3 4 5 |
faithfulnessV2 (faithfulness_v2) applied to one turn at a time. This is a documented application, not a new template. retrievalTurns (retrieval_turns) makes the evidence of each assistant turn that has retrievedContexts, and you call metricJudge (metric_judge) on it. The request is the full preceding dialogue, the answer is the reply of that turn, and the sources are the snapshot of that turn. Every claim must follow from that snapshot. An earlier snapshot adds no support, so a retained source has to be in the snapshot of the turn that uses it. A nonempty explicit abstention can pass, and it is the only reply that can pass with an empty snapshot. Each turn gets its own pass or fail verdict, with no average and no window. DeepEval pools the retrieval contexts of a window, checks that no claim contradicts them and averages the window scores. This check is stricter: it needs support, not only the absence of a contradiction. Zero retrieval turns is misuse. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 7 8 9 10 |
| Turn contextual precision | parity | Python and TypeScript: slides a window over the conversation, up to window_size exchanges (windowSize in TypeScript), 10 by default, where an exchange is a user message with the assistant reply that follows it. For each window, it pools the retrieval contexts of the turns, judges which nodes help to reach the expected_outcome (expectedOutcome in TypeScript) of the conversation, and scores rank-weighted precision, so relevant nodes that rank early score higher. The score is the average of the window scores. Needs the turns with their retrieval_context and one expected_outcome. The default threshold is 0.5. Strict mode needs a score of 1 source 2 3 4 5 |
contextRankingV2 (context_ranking_v2) applied to one turn at a time. This is a documented application, not a new template. retrievalTurns (retrieval_turns) makes the evidence of each assistant turn that has retrievedContexts, and you call metricJudge (metric_judge) on it. The request is the full preceding dialogue, the sources are the snapshot of that turn in their original order, and the reference is the nonblank factual reference of that turn, from the references that you give retrievalTurns. At least one passage must support the reference, and every relevant passage must come before every irrelevant one. One inversion fails, an empty snapshot fails, and a source retained from an earlier turn is noise when it does not help the current reference. As in the single-turn row (see Contextual precision), this matches only a precision of 1, with no rank weighting. DeepEval pools the nodes of a window, uses one expected_outcome for the whole conversation and averages the window scores. Here each turn has its own reference and its own pass or fail verdict, with no pooling and no average. Zero retrieval turns or a missing reference is misuse. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 7 8 9 10 |
| Turn contextual recall | parity | Python and TypeScript: slides a window over the conversation, up to window_size exchanges (windowSize in TypeScript), 10 by default, where an exchange is a user message with the assistant reply that follows it. For each window, it splits the expected_outcome (expectedOutcome in TypeScript) of the conversation into statements and scores the share that the pooled retrieval contexts of the window support. The score is the average of the window scores. Needs the turns with their retrieval_context and one expected_outcome. The default threshold is 0.5. Strict mode needs a score of 1 source 2 3 4 5 |
referenceCoverageV2 (reference_coverage_v2) applied to one turn at a time. This is a documented application, not a new template. retrievalTurns (retrieval_turns) makes the evidence of each assistant turn that has retrievedContexts, and you call metricJudge (metric_judge) on it. The request is the full preceding dialogue, the sources are the snapshot of that turn, and the reference is the nonblank factual reference of that turn, from the references that you give retrievalTurns. The passages of that turn together must support every factual assertion of the reference, without an unresolved contradiction. Partial coverage fails, an empty or all-blank snapshot fails, and an earlier snapshot adds no support. As in the single-turn row (see Contextual recall), this matches only a recall of 1, not the share of attributable statements. DeepEval pools the retrieval contexts of a window, uses one expected_outcome for the whole conversation and averages the window scores. Here each turn has its own reference and its own pass or fail verdict, with no pooling and no average. Zero retrieval turns or a missing reference is misuse. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 7 8 9 10 |
| Turn contextual relevancy | parity | Python and TypeScript: slides a window over the conversation, up to window_size exchanges (windowSize in TypeScript), 10 by default, where an exchange is a user message with the assistant reply that follows it. For each window, it extracts statements from the pooled retrieval contexts and scores the share that is relevant to the user input. The score is the average of the window scores. Needs the turns with their retrieval_context. The default threshold is 0.5. Strict mode needs a score of 1 for each window, so every statement that the LLM extracts must be relevant. As in the single-turn metric, the prompt in the Python source asks for the high-level information of each context, so a detail that it does not extract is not judged source 2 3 4 5 |
contextRelevanceV2 (context_relevance_v2) applied to one turn at a time. This is a documented application, not a new template. retrievalTurns (retrieval_turns) makes the evidence of each assistant turn that has retrievedContexts, and you call metricJudge (metric_judge) on it. The request is the full preceding dialogue, so the judge reads the current request in context, and the sources are the snapshot of that turn. Every substantive statement of the passages must help answer the current request or give necessary background. One relevant but incomplete passage can pass, because this checks noise, not sufficiency. A source retained from an earlier turn fails when it does not help the current request, and an empty snapshot fails. As in the single-turn row (see Contextual relevancy), DeepEval’s strict mode needs a relevancy of 1 only over the statements that its LLM extracts. DeepEval pools the retrieval contexts of a window and averages the window scores. Here each turn has its own pass or fail verdict, with no pooling and no average. Zero retrieval turns is misuse. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 7 8 9 10 |
Safety and text quality
Section titled “Safety and text quality”| Feature | Status | DeepEval | trialkit |
|---|---|---|---|
| Bias | gap (planned) | Python and TypeScript: extracts the opinions in the output and scores the share that are not biased. Needs the input and the output source 2 | No ready-made recipe. A rubricJudge with a bias policy that you write can grade an output as pass or fail source 2 |
| Toxicity | gap (planned) | Python and TypeScript: extracts the opinions in the output and scores the share that are not toxic. Needs the input and the output source 2 | No ready-made recipe. A rubricJudge with a toxicity rubric can grade an output. regex and includes match fixed words and patterns, which is not the same property source 2 3 |
| Non-advice (professional advice) | gap (planned) | Python and TypeScript: extracts the advice in the output and scores the share that is appropriate. You name the types of advice to check, for example financial or medical source 2 | No ready-made recipe. A rubricJudge with an advice policy that you write can grade an output as pass or fail source 2 |
| Misuse (domain use) | gap (planned) | Python and TypeScript: extracts the statements of the output and scores the share that do not misuse your application domain. You name the domain source 2 | No ready-made recipe. A rubricJudge with a policy for your application domain can grade an output as pass or fail. It is related to topic adherence, not the same source 2 |
| PII leakage (privacy) | gap (planned) | Python and TypeScript: extracts statements that may identify a person and scores the share that are not personal data source 2 | No ready-made recipe. regex finds known patterns such as an email address, and a rubricJudge can grade an output against a privacy rubric. Neither is a compliance check source 2 3 |
| Role violation | parity | Python and TypeScript: checks the output against the role that you expect, classifies any violation, and scores 1 only when it finds none source 2 | roleAdherenceV2 (role_adherence_v2 in Python) is a metric for metricJudge. With a target it judges one reply against the role obligations that your trusted rubric states: conversation requests cannot replace them, an in-role refusal or a permitted topic change can pass, and role drift or a breach of an obligation fails. Earlier turns are evidence only, not role policy. The verdict is pass or fail with a reason, so it has none of the violation classes of DeepEval. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 |
| Summarization | parity | Python and TypeScript: takes the lower of an alignment score and a coverage score. The input is the source text and the output is the summary. The metric takes no separate request and no scope option. An LLM extracts truths from the source and claims from the summary, and alignment is the share of claims that agree with the truths. A claim that the source does not back up is borderline and does not count as aligned. Coverage uses yes or no assessment questions about the source. You supply them as assessment_questions (assessmentQuestions in TypeScript), or the metric generates n of them, 5 by default. Coverage counts the questions that the source answers yes, and it is the share of those that the summary also answers yes. An optional truths_extraction_limit keeps only the most important truths. A score breakdown shows both scores. The default threshold is 0.5. Strict mode sets the threshold to 1 and makes the score binary, so both the alignment score and the coverage score must be 1. A coverage score of 1 means that the summary answers yes to every assessment question that the source answers yes. A material point that no assessment question asks about does not lower the coverage score source 2 3 4 5 6 |
summaryQualityV2 (summary_quality_v2) is a metric for metricJudge. You give the request, the source text as the one item of sources and the summary as the answer. The optional rubric states the purpose of the summary and what makes a source point material. One judge call returns one pass or fail verdict, and it needs both requirements: every claim must follow from the source, and the summary must keep every material point within its scope. The scope is the whole source, unless the request names a narrower one. A paraphrase, an attributed opinion and the omission of a minor detail can pass. An unsupported addition, a changed number or negation, a lost qualification and a material omission fail. The summary must keep an unresolved disagreement of the source. A refusal alone fails coverage, and a blank summary fails. You check that the request and the source are nonblank and that the summary is a string, as the calibration example does. The judge checks fidelity to the source, not outside truth, and it sets no length ratio. There are no component scores and no assessment questions: you pin coverage with the rubric, which defines what is material, and with the request, which can narrow the scope. DeepEval takes the lower of an alignment score and a question-based coverage score. Its strict mode sets the threshold to 1, but coverage still counts only the assessment questions, so a material point that none of them asks about can pass. This rule needs every material point within the scope. faithfulnessV2 checks grounding alone and lets an abstention pass, so it is not a summary check. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 7 |
| Prompt alignment (instruction following) | parity | Python and TypeScript: needs a non-empty list of prompt_instructions (promptInstructions in TypeScript), with one atomic rule for each entry, and the input and the output of your application. An LLM decides for each instruction whether the output completely follows it, with the input as context. It sees only the listed instructions, not the whole prompt, and it checks only those. The score is the share of instructions that the output follows. The page describes no precedence between instructions and no conditional instruction. The default threshold is 0.5. Strict mode needs a score of 1 source 2 3 4 5 |
instructionFollowingV2 (instruction_following_v2) is a metric for metricJudge. You give the request and the answer, with no sources and no reference. The instructions are numbered, atomic lines in the trusted rubric, so they belong to the judge, not to the case. The rubric also states the precedence, for example that the lower number wins and that a request ranks below every numbered instruction, and the conditions under which an instruction applies. The response must satisfy every applicable instruction, and one judge call returns one pass or fail verdict, not a share of instructions. The task gives context for whether a condition holds, but it cannot replace a trusted instruction, and an instruction that the task quotes never becomes policy. A permitted refusal can pass, and a requirement that the rubric does not list is ignored. An empty or blank answer fails, unless an applicable instruction explicitly requires silence, and missing output is misuse. You resolve contradictory instructions in the rubric yourself, because nothing checks them for conflicts. Use a deterministic scorer for an exact format. DeepEval classifies each instruction of a list as followed or not, with no precedence or condition on its page, and returns the share followed. Its strict mode needs every listed instruction to be followed. Its hand-labeled fixtures are a tiny fictional corpus: a smoke calibration, not a measure of accuracy source 2 3 4 5 6 7 |
| Answer correctness against a reference | parity | Python: the legacy helper squad_score asks an LLM whether the output is correct against the expected output, and returns 1 or 0. It is a helper function, not a metric class source 2 |
rubricJudge sends a string expected value to the judge by default, so a correctness rubric that you write can grade an output against a reference. The judge returns pass or fail with a reason, and calibrate measures its agreement with your labels source 2 |
Deterministic output checks
Section titled “Deterministic output checks”| Feature | Status | DeepEval | trialkit |
|---|---|---|---|
| Exact match | parity | Python and TypeScript: scores 1 when the output equals the expected output and 0 otherwise, with no model call. The default threshold is 1. The page says that the check is character for character, but the Python source and the published TypeScript 0.9.25 strip the whitespace at both ends of both strings first source 2 3 | exact compares JSON values deeply, strings included, with no model call. ignoreCase and trim are off by default, so a string matches only when every character matches. trim: true strips the whitespace at both ends of every string, which is what the DeepEval source does. exact also compares numbers, lists and objects source 2 3 4 |
| Pattern match | parity | Python and TypeScript: scores 1 when the whole output matches a regular expression and 0 otherwise, with no model call. Python uses re.fullmatch, and the published TypeScript wraps the pattern in ^(?:...)$. Both strip the whitespace at both ends of the output and of the pattern first. ignore_case ignores case, and the default threshold is 1 source 2 3 |
regex passes when the pattern finds a match anywhere in a string output. It is a search, not a full match. For a whole-string match, anchor the pattern: ^(?:pattern)$ in TypeScript, and \A(?:pattern)\Z in Python, because Python’s $ also matches before a trailing newline. In TypeScript, keep the multiline flag off, because with it a match on one line of several passes. The group keeps alternatives inside the anchors. regex does not trim: pick the trimmed text with output source 2 3 4 |
| JSON correctness | parity | Python and TypeScript: scores 1 when the output parses as JSON and fits the expected schema (a Pydantic model in Python, a Zod schema in TypeScript) and 0 otherwise. The model call only writes the reason for a failure source 2 | schema passes when a value satisfies any Standard Schema v1 validator, such as Zod, Valibot or ArkType; your app brings the library. To check JSON text, give output a function that parses it. A parse error is a failed score, not an error, and JSON that breaks the schema is a failed score too. In Python, pass an object with a ~standard validate function, which wraps your own validator, for example a Pydantic model source 2 3 4 5 6 |
| Feature | Status | DeepEval | trialkit |
|---|---|---|---|
| MCP use | gap (planned) | Python and TypeScript: judges whether the MCP tools, resources and prompts that the app called, and their arguments, fit the task and the primitives that the servers offer. It can also judge whether a skipped call was needed. Needs the input, the output, the server capabilities and the calls source 2 | No ready-made recipe. The judges keep only the supported GenAI span attributes, so raw MCP spans do not carry the evidence on their own. toolAppropriatenessV2 takes the tools that you list, not MCP resources or prompts. The packages take no MCP SDK dependency source 2 3 4 5 |
| Multi-turn MCP use | gap (planned) | Python and TypeScript: scores how well the primitives called in each MCP interaction of a conversation fit the task, and averages the scores. Needs the turns, the servers and the primitive calls of each turn source 2 | No ready-made recipe. No check reads MCP primitive calls from the turns of a conversation, and the packages take no MCP SDK dependency source 2 3 |
| MCP task completion | gap (planned) | Python and TypeScript: scores the share of user tasks that the interactions of a conversation satisfied. Needs the turns, the servers, and the primitive calls and results of each turn source 2 | No ready-made recipe. taskCompletionV2 checks success criteria in the spans of one trial, not MCP results over a conversation source 2 3 4 5 |
Images and voice
Section titled “Images and voice”| Feature | Status | DeepEval | trialkit |
|---|---|---|---|
| Image metrics | out of scope | Python and TypeScript: five multimodal metrics judge images with a multimodal model: Image Coherence, Image Helpfulness, Image Reference, Text-to-Image and Image Editing. The test case must hold images source 2 3 4 5 6 7 | No image input. The model port carries text only (the content of a message is text), so a judge cannot inspect pixels. Media support would need a media-aware port first source 2 |
| Acoustic voice metrics | out of scope | Python: eight metrics score assistant audio and turn timing with signal rules, not an LLM (Audio Glitch uses an LLM only to write its reason): Voice Naturalness, Speech Intelligibility, Voice Consistency, Turn-Taking Naturalness, Agent Responsiveness, Audio Integrity, Audio Glitch and Voice Reliability. The turns must carry audio or audio timing. The published TypeScript 0.9.25 exports none of them source 2 3 4 5 6 7 8 9 10 | No audio input or waveform decoding. The model port carries text only, so these checks fall outside the packages. Transcription accuracy on text that you supply is a separate row source 2 |
| Transcription accuracy (supplied text) | gap (planned) | Python: scores the share of exchanges that the speech-to-text of the agent heard faithfully. It ignores harmless variation and penalizes changed names, numbers and negation. Audio mode re-transcribes the clips. The page shows TypeScript code, but the published TypeScript 0.9.25 exports no such metric source 2 | No ready-made recipe. A rubricJudge can compare two texts that you supply. Getting audio and running speech-to-text stay outside the packages source 2 |
RAGAS adapters and legacy helpers
Section titled “RAGAS adapters and legacy helpers”| Feature | Status | DeepEval | trialkit |
|---|---|---|---|
| RAGAS adapters | out of scope | Python: wrappers around the optional RAGAS package for answer relevancy, faithfulness, contextual precision, contextual recall, contextual entities recall and a composite RagasMetric. The installed RAGAS version computes the scores. The docs page describes four components with a default threshold of 0.5; the source adds entities recall, averages five and defaults to 0.3 source 2 |
No RAGAS adapter. The packages take no dependency on RAGAS, and trialkit matches the evaluated property, not the numbers. The rows above cover the same properties: answer relevancy, faithfulness, contextual precision and recall, and reference entity coverage source 2 |
| Legacy match helpers | parity | Python: three helper functions of deepeval.scorer, with no threshold or reason: exact_match_score (equality after stripping whitespace, 0 for an empty prediction), quasi_exact_match_score (equality after normalizing case, punctuation, articles and whitespace) and truth_identification_score (matched predictions over expected items, times 100, on integer-list strings, counting a repeated item each time) source 2 |
exact with trim: true covers exact_match_score, but two empty strings are equal in exact, and DeepEval fails an empty prediction. For quasi_exact_match_score, give output and expected a function that normalizes case, punctuation, articles and whitespace: ignoreCase and trim do not remove punctuation or articles. setMatch with metric: 'recall' on parsed lists covers truth_identification_score; it counts duplicates once and scores from 0 to 1, not from 0 to 100 source 2 |
| Legacy lexical helpers (ROUGE, BLEU, membership, pass@k) | gap (planned) | Python: four helper functions of deepeval.scorer: rouge_score (stemmed ROUGE-1, ROUGE-2 and ROUGE-L F-measure), sentence_bleu_score (BLEU with chosen n-gram weights and a brevity penalty), quasi_contains_score (the normalized output equals one of the candidate references, not a substring) and pass_at_k (1 minus C(n-c, k) over C(n, k), from counts) source 2 |
No ROUGE, BLEU, normalized-membership or pass@k scorer. includes is a substring check, not normalized membership. The atLeast and majority verdicts over trials are not the pass@k estimator source 2 |
| Legacy learned-model helpers | out of scope | Python: helper functions of deepeval.scorer that run local models: bert_score (BERTScore), faithfulness_score (SummaC), hallucination_score (a Vectara model), neural_toxic_score (Detoxify), answer_relevancy_score (an encoder) and neural_bias_score. PII_score raises NotImplementedError. They are helpers, not metric classes source 2 |
No bundled model weights or local-model runtime: the packages declare no model library, so they carry no PyTorch and download no model. The recipe rows cover the same properties: faithfulness, toxicity, bias and answer relevancy. An external model can run inside a core Scorer source 2 |
Classifiers
Section titled “Classifiers”| Feature | Status | DeepEval | trialkit |
|---|---|---|---|
| Custom classifier | parity | Python: Classifier with Label names and descriptions that you write. The model picks exactly one label and gives a reason. A case can name the label that it expects; without one, the classifier only reports the label source 2 3 |
classifierJudge takes your choices (each label has a score from 0 to 1), picks exactly one label and returns the reason of the model. A pass function decides which labels pass. A reply with a label that is not in choices ends the trial in error, so add an explicit other label if you need one. The pass rule applies to the whole judge, not to one case source 2 3 |
| Behavioral classifiers | gap (planned) | Python: five built-in classifiers, each with a closed label set and a reason: Refusal (complied, refused, partial_refusal), Escalation (escalated, offered, not_escalated), Scope Adherence (in_scope, deflected, out_of_scope_answered), Clarification (asked_clarification, answered_directly, guessed) and Resolution (resolved, unresolved, handed_over) source 2 3 4 5 6 7 |
No built-in template for these labels. classifierJudge with choices and a rubric that you write does the same job, with one pass rule for every case source 2 |
| Grounding and security classifiers | gap (planned) | Python: three built-in classifiers, each with a closed label set and a reason: Abstention (abstained, answered, fabricated), Prompt Injection (resisted, partially_followed, followed_injection) and Data Leakage (no_leak, system_prompt_leak, pii_or_secret_leak) source 2 3 4 5 |
No built-in template for these labels. classifierJudge with choices and a rubric that you write does the same job. The rubric goes in the trusted system prompt, and the case text stays untrusted data source 2 |
| Policy and format classifiers | gap (planned) | Python: five built-in classifiers, each with a closed label set and a reason: Forbidden Commitments (clean, unauthorized_commitment, competitor_mention, disparagement), Tone Adherence (on_tone, off_tone), Required Disclosure (present, missing, partial), Response Language (matches_user, mismatch) and Instruction Completeness (complete, partial, ignored) source 2 3 4 5 6 7 |
No built-in template for these labels. classifierJudge with choices and a rubric that you write does the same job. For wording that the policy fixes, includes is the deterministic check source 2 3 |
Evidence for the DeepEval metric checklist
Section titled “Evidence for the DeepEval metric checklist”| Feature | Side | Source | Retrieved | Measurement |
|---|---|---|---|---|
| Custom criteria judge (G-Eval criteria) | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Custom criteria judge (G-Eval criteria) | trialkit | source 2 | 2026-11-09 | Read the cited sections of packages/judge/README.md and python/judge/README.md; documented behavior, not a live run |
| Judge configuration | DeepEval | source | 2026-11-09 | Read the cited documentation; feature description, not a live integration test |
| Judge configuration | trialkit | source | 2026-11-09 | Read the Judges and Templates sections of packages/judge/README.md; documented behavior, not a live run |
| Judge calibration against human labels | DeepEval | source | 2026-11-09 | Read the cited FAQ entry and searched deepeval.com/docs for calibration, agreement and human labels; found no library report. Absence is not proof |
| Judge calibration against human labels | trialkit | source 2 | 2026-11-14 | Read the Calibrate section of packages/judge/README.md, the calibration section of docs/guide.md and spec/fixtures/calibration-report.json; documented behavior, not a live run |
| G-Eval workflow (evaluation steps and score controls) | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| G-Eval workflow (evaluation steps and score controls) | trialkit | source 2 | 2026-11-09 | Read the cited sections of packages/judge/README.md and packages/core/README.md; documented behavior, not a live run |
| Pairwise judging (Arena G-Eval) | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Pairwise judging (Arena G-Eval) | trialkit | source 2 | 2026-11-09 | Read the cited sections of packages/judge/README.md and packages/cli/README.md; documented behavior, not a live run |
| DAG execution (decision-graph judge) | DeepEval | source 2 3 | 2026-11-09 | Read each cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| DAG execution (decision-graph judge) | trialkit | source 2 | 2026-11-09 | Read the cited sections of packages/judge/README.md and docs/design.md; documented behavior, not a live run |
| Custom metric class (BaseMetric) | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Custom metric class (BaseMetric) | trialkit | source 2 | 2026-11-09 | Read the cited sections of packages/core/README.md and docs/guide.md; documented behavior, not a live run |
| Jev metrics and modes (JevEval, hybrid, system_one) | DeepEval | source 2 3 | 2026-11-09 | Read each cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Jev metrics and modes (JevEval, hybrid, system_one) | trialkit | source 2 | 2026-11-09 | Read the cited sections of packages/core/README.md and docs/design.md; documented design, not a live run |
| Answer relevancy | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Answer relevancy | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Faithfulness | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Faithfulness | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Context sufficiency (answerability) | DeepEval | source 2 3 4 | 2026-11-09 | Read the cited pages, the metrics catalog and the published TypeScript 0.9.25 metric declarations; found no referenceless sufficiency metric. Absence is not proof |
| Context sufficiency (answerability) | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Contextual relevancy | DeepEval | source 2 3 | 2026-11-12 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Contextual relevancy | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Contextual precision (ranking) | DeepEval | source 2 3 | 2026-11-11 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test. The statement about a score of 1 follows from the equation and the source code; it was not run |
| Contextual precision (ranking) | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Contextual recall (reference coverage) | DeepEval | source 2 | 2026-11-11 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Contextual recall (reference coverage) | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Hallucination (ideal-context alignment) | DeepEval | source 2 | 2026-11-11 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Hallucination (ideal-context alignment) | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Citation faithfulness | DeepEval | source 2 | 2026-11-11 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations, which export no such metric; feature description, not a live integration test |
| Citation faithfulness | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Reference entity coverage | DeepEval | source 2 | 2026-11-09 | Read the cited source file and documentation page; feature description, not a live integration test |
| Reference entity coverage | trialkit | source 2 | 2026-11-09 | Read the cited sections of packages/scorers/README.md and python/scorers/README.md; documented behavior, not a live run |
| Task completion | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Task completion | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Step efficiency | DeepEval | source 2 | 2026-11-11 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Step efficiency | trialkit | source 2 3 4 5 6 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Plan adherence | DeepEval | source 2 3 | 2026-11-11 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Plan adherence | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Plan quality | DeepEval | source 2 3 4 | 2026-11-11 | Read the cited documentation page, the metrics catalog, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Plan quality | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Tool matching: names, arguments and order | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Tool matching: names, arguments and order | trialkit | source 2 3 | 2026-11-09 | Read the cited sections of packages/scorers/README.md, python/scorers/README.md and docs/guide.md; documented behavior, not a live run |
| Tool matching: tool results | DeepEval | source 2 3 | 2026-11-09 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Tool matching: tool results | trialkit | source 2 3 4 5 | 2026-11-14 | Read the cited sections of packages/scorers/README.md, python/scorers/README.md and spec/scorers.md, and the cited fixture and code files; documented behavior, not a live run |
| Semantic tool use (selection and arguments) | DeepEval | source 2 3 | 2026-11-09 | Read each cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Semantic tool use (selection and arguments) | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md and python/judge/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Agent loop detection | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations, which export no such metric; feature description, not a live integration test |
| Agent loop detection | trialkit | source 2 3 | 2026-11-09 | Read the cited sections of packages/scorers/README.md, python/scorers/README.md and docs/guide.md; documented behavior, not a live run |
| Tool permission | DeepEval | source 2 3 | 2026-11-09 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 metric declarations, which export no such metric; feature description, not a live integration test |
| Tool permission | trialkit | source 2 3 4 5 | 2026-11-14 | Read the cited sections of packages/scorers/README.md, python/scorers/README.md and spec/scorers.md, and the cited fixture and code files; documented behavior, not a live run |
| Turn relevancy | DeepEval | source 2 3 4 5 6 | 2026-11-12 | Read the cited documentation page, the metrics catalog, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Turn relevancy | trialkit | source 2 3 4 5 6 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/conversation.md and spec/judge-templates/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Knowledge retention: one reply | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Knowledge retention: one reply | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md and spec/judge-templates/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Knowledge retention: whole conversation | DeepEval | source 2 3 4 5 | 2026-11-12 | Read the cited documentation page, the Python source and prompt template, and the published TypeScript 0.9.25 build and declarations of the metric; feature description, not a live integration test. The page says that strict mode sets the threshold to 0, but both sources set it to 1, so the cell follows the sources |
| Knowledge retention: whole conversation | trialkit | source 2 3 4 5 6 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/conversation.md and spec/judge-templates/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Role adherence: one reply | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Role adherence: one reply | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md and spec/judge-templates/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Role adherence: whole conversation | DeepEval | source 2 3 | 2026-11-12 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Role adherence: whole conversation | trialkit | source 2 3 4 5 6 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/conversation.md and spec/judge-templates/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Conversation completeness | DeepEval | source 2 3 4 | 2026-11-12 | Read the cited documentation page, the Python source and prompt template, and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Conversation completeness | trialkit | source 2 3 4 5 6 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/conversation.md and spec/judge-templates/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Goal accuracy | DeepEval | source 2 3 4 | 2026-11-12 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Goal accuracy | trialkit | source 2 3 4 5 6 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/conversation.md and spec/judge-templates/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Tool use over a conversation | DeepEval | source 2 3 4 | 2026-11-12 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Tool use over a conversation | trialkit | source 2 3 4 5 6 7 8 9 10 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/calibration.md and spec/judge-templates/README.md, the README and the projection code of the calibration example (recipes.ts and recipes.py), and the cited template and fixture files; documented behavior, not a live run |
| Topic adherence | DeepEval | source 2 3 4 | 2026-11-12 | Read the cited documentation page, the Python source and prompt template, and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Topic adherence | trialkit | source 2 3 4 5 6 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/conversation.md and spec/judge-templates/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Turn faithfulness | DeepEval | source 2 3 4 5 | 2026-11-12 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Turn faithfulness | trialkit | source 2 3 4 5 6 7 8 9 10 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/calibration.md and spec/judge-templates/README.md, the README and the projection code of the calibration example (recipes.ts and recipes.py), and the cited template and fixture files; documented behavior, not a live run |
| Turn contextual precision | DeepEval | source 2 3 4 5 | 2026-11-12 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Turn contextual precision | trialkit | source 2 3 4 5 6 7 8 9 10 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/calibration.md and spec/judge-templates/README.md, the README and the projection code of the calibration example (recipes.ts and recipes.py), and the cited template and fixture files; documented behavior, not a live run |
| Turn contextual recall | DeepEval | source 2 3 4 5 | 2026-11-12 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Turn contextual recall | trialkit | source 2 3 4 5 6 7 8 9 10 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/calibration.md and spec/judge-templates/README.md, the README and the projection code of the calibration example (recipes.ts and recipes.py), and the cited template and fixture files; documented behavior, not a live run |
| Turn contextual relevancy | DeepEval | source 2 3 4 5 | 2026-11-12 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Turn contextual relevancy | trialkit | source 2 3 4 5 6 7 8 9 10 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/calibration.md and spec/judge-templates/README.md, the README and the projection code of the calibration example (recipes.ts and recipes.py), and the cited template and fixture files; documented behavior, not a live run |
| Bias | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Bias | trialkit | source 2 | 2026-11-14 | Read the cited sections of packages/judge/README.md and spec/judge-templates/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2); documented behavior, not a live run |
| Toxicity | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Toxicity | trialkit | source 2 3 | 2026-11-14 | Read the cited sections of packages/judge/README.md, spec/judge-templates/README.md and packages/scorers/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2); documented behavior, not a live run |
| Non-advice (professional advice) | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Non-advice (professional advice) | trialkit | source 2 | 2026-11-14 | Read the cited sections of packages/judge/README.md and spec/judge-templates/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2); documented behavior, not a live run |
| Misuse (domain use) | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Misuse (domain use) | trialkit | source 2 | 2026-11-14 | Read the cited sections of packages/judge/README.md and spec/judge-templates/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2); documented behavior, not a live run |
| PII leakage (privacy) | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| PII leakage (privacy) | trialkit | source 2 3 | 2026-11-14 | Read the cited sections of packages/judge/README.md, spec/judge-templates/README.md and packages/scorers/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2); documented behavior, not a live run |
| Role violation | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Role violation | trialkit | source 2 3 4 5 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md and spec/judge-templates/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Summarization | DeepEval | source 2 3 4 5 6 | 2026-11-12 | Read the cited documentation page, the Python source and prompt templates, and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Summarization | trialkit | source 2 3 4 5 6 7 | 2026-11-14 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/calibration.md, spec/judge-templates/README.md and examples/metric-recipes/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Prompt alignment (instruction following) | DeepEval | source 2 3 4 5 | 2026-11-12 | Read the cited documentation page, the Python source and prompt templates, and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Prompt alignment (instruction following) | trialkit | source 2 3 4 5 6 7 | 2026-11-13 | Read the cited sections of packages/judge/README.md, python/judge/README.md, spec/calibration.md, spec/judge-templates/README.md and examples/metric-recipes/README.md, and the cited template and fixture files; documented behavior, not a live run |
| Answer correctness against a reference | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the DeepEval source of the helpers; feature description, not a live integration test |
| Answer correctness against a reference | trialkit | source 2 | 2026-11-09 | Read the cited sections of packages/judge/README.md; documented behavior, not a live run |
| Exact match | DeepEval | source 2 3 | 2026-11-09 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 build of the metric; feature description, not a live integration test |
| Exact match | trialkit | source 2 3 4 | 2026-11-09 | Read the cited sections of packages/scorers/README.md and python/scorers/README.md; documented behavior, not a live run |
| Pattern match | DeepEval | source 2 3 | 2026-11-09 | Read the cited documentation page, the Python source and the published TypeScript 0.9.25 build of the metric; feature description, not a live integration test |
| Pattern match | trialkit | source 2 3 4 | 2026-11-09 | Read the cited sections of packages/scorers/README.md and python/scorers/README.md; documented behavior, not a live run |
| JSON correctness | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| JSON correctness | trialkit | source 2 3 4 5 6 | 2026-11-09 | Read the cited sections of packages/scorers/README.md, python/scorers/README.md and docs/guide.md; documented behavior, not a live run |
| MCP use | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| MCP use | trialkit | source 2 3 4 5 | 2026-11-14 | Read the cited sections of packages/judge/README.md, docs/design.md and spec/judge-templates/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2), and the cited template files; documented behavior, not a live run |
| Multi-turn MCP use | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Multi-turn MCP use | trialkit | source 2 3 | 2026-11-14 | Read the cited sections of packages/judge/README.md, docs/design.md and spec/judge-templates/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2); documented behavior, not a live run |
| MCP task completion | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| MCP task completion | trialkit | source 2 3 4 5 | 2026-11-14 | Read the cited sections of packages/judge/README.md, docs/design.md and spec/judge-templates/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2), and the cited template files; documented behavior, not a live run |
| Image metrics | DeepEval | source 2 3 4 5 6 7 | 2026-11-09 | Read each cited documentation page and the published TypeScript 0.9.25 metric declarations; feature description, not a live integration test |
| Image metrics | trialkit | source 2 | 2026-11-09 | Read the cited sections of packages/core/README.md and docs/design.md; documented design, not a live run |
| Acoustic voice metrics | DeepEval | source 2 3 4 5 6 7 8 9 10 | 2026-11-09 | Read each cited documentation page and the published TypeScript 0.9.25 metric declarations, which export none of them; feature description, not a live integration test |
| Acoustic voice metrics | trialkit | source 2 | 2026-11-09 | Read the cited sections of packages/core/README.md and docs/design.md; documented design, not a live run |
| Transcription accuracy (supplied text) | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the published TypeScript 0.9.25 metric declarations, which export no such metric; feature description, not a live integration test |
| Transcription accuracy (supplied text) | trialkit | source 2 | 2026-11-14 | Read the cited sections of packages/judge/README.md and spec/judge-templates/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2); documented behavior, not a live run |
| RAGAS adapters | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the adapter source; feature description, not a live integration test |
| RAGAS adapters | trialkit | source 2 | 2026-11-09 | Read the cited sections of docs/design.md and packages/core/README.md; documented design, not a live run |
| Legacy match helpers | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the DeepEval source of the helpers; feature description, not a live integration test |
| Legacy match helpers | trialkit | source 2 | 2026-11-09 | Read the cited sections of packages/scorers/README.md and python/scorers/README.md; documented behavior, not a live run |
| Legacy lexical helpers (ROUGE, BLEU, membership, pass@k) | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the DeepEval source of the helpers; feature description, not a live integration test |
| Legacy lexical helpers (ROUGE, BLEU, membership, pass@k) | trialkit | source 2 | 2026-11-09 | Read the cited sections of packages/scorers/README.md and python/scorers/README.md; documented behavior, not a live run |
| Legacy learned-model helpers | DeepEval | source 2 | 2026-11-09 | Read the cited documentation page and the DeepEval source of the helpers; feature description, not a live integration test |
| Legacy learned-model helpers | trialkit | source 2 | 2026-11-09 | Read the cited sections of docs/design.md and packages/core/README.md; documented design, not a live run |
| Custom classifier | DeepEval | source 2 3 | 2026-11-09 | Read each cited documentation page and the published TypeScript 0.9.25 metric declarations, which export none of them; feature description, not a live integration test |
| Custom classifier | trialkit | source 2 3 | 2026-11-14 | Read the cited sections of packages/judge/README.md, spec/judge-templates/README.md and python/judge/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2); documented behavior, not a live run |
| Behavioral classifiers | DeepEval | source 2 3 4 5 6 7 | 2026-11-09 | Read each cited documentation page and the published TypeScript 0.9.25 metric declarations, which export none of them; feature description, not a live integration test |
| Behavioral classifiers | trialkit | source 2 | 2026-11-14 | Read the cited sections of packages/judge/README.md and spec/judge-templates/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2); documented behavior, not a live run |
| Grounding and security classifiers | DeepEval | source 2 3 4 5 | 2026-11-09 | Read each cited documentation page and the published TypeScript 0.9.25 metric declarations, which export none of them; feature description, not a live integration test |
| Grounding and security classifiers | trialkit | source 2 | 2026-11-14 | Read the cited sections of packages/judge/README.md and spec/judge-templates/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2); documented behavior, not a live run |
| Policy and format classifiers | DeepEval | source 2 3 4 5 6 7 | 2026-11-09 | Read each cited documentation page and the published TypeScript 0.9.25 metric declarations, which export none of them; feature description, not a live integration test |
| Policy and format classifiers | trialkit | source 2 3 | 2026-11-14 | Read the cited sections of packages/judge/README.md, spec/judge-templates/README.md and packages/scorers/README.md (the Files section lists the template files: the generic rubric and classifier templates, the trajectory template and the 21 metric templates of version 2); documented behavior, not a live run |