Skip to content

Tool comparison

Versions, dependency counts and features of trialkit and other eval tools, each with its source, retrieval date and measurement. Snapshot: 2026-11-12.

The data is in comparison.json, and the site renders it when it builds. The DeepEval metric rows are on the DeepEval metric checklist.

A documentation snapshot, not a head-to-head benchmark. Unverified means the cited source did not establish the claim or it was not rechecked; it does not mean the feature is absent.

npm counts describe only the named JavaScript package, not Python distributions or all optional integrations. Install counts include the target package, default optional packages and automatically resolved non-optional peers; they vary with platform, registry state and npm version.

Evalite features and install counts use the 1.0 beta, while npm latest remains 0.19.0. Viteval advertises an rc on its site while npm latest remains 0.5.9; do not treat the website badge as registry availability.

DeepEval ships a Python package (PyPI 4.3.3) and a TypeScript package (npm 0.9.25) whose documented features differ, so a fact or feature row names the package it describes. Confident AI is DeepEval’s optional hosted platform, and DeepTeam is a separate red-teaming package.

promptfoo is part of OpenAI since March 2026 and stays MIT-licensed (https://www.promptfoo.dev/blog/promptfoo-joining-openai). OpenAI shuts down its hosted Evals platform and points to promptfoo (https://developers.openai.com/api/docs/deprecations). The shutdown date is in the migration notes.

20 features with sources. trialkit is ahead on 8 and at parity on 6. It leaves 6 out of scope.

Status shows how trialkit compares with DeepEval:

  • ahead: trialkit does it in its open-source packages. The cited DeepEval sources show nothing equal, or show it only with an account.
  • parity: both do it at a similar level.
  • gap: DeepEval does it and trialkit does not yet. The status links the plan, or says planned.
  • out of scope: DeepEval does it. trialkit leaves it to other tools by design (non-goals).

(unverified) marks a claim that the cited source did not establish, such as a feature that was not found. It does not prove that the feature is absent. It also marks a trialkit claim about work that is not merged: only the pull request states it, so the claim stays unverified until the pull request merges.

Feature Status DeepEval trialkit
Repeated trials of a case parity Python: deepeval test run –repeat n (-r) reruns each test case n times; the TypeScript option table lists no –repeat. Confident AI (hosted) can call your endpoint several times per golden (multi-generation) source 2 trials sets how many counted trials each case runs (default 1), and errorRetries adds trials after infrastructure errors only. Trials of one case run in order; cases can run concurrently source 2
Verdict rule, intervals and early stop ahead No per-case verdict rule (all, majority, at least n), interval or early stop found. A test case marked flaky prints a warning instead of failing the test, and a metric marked flaky never fails its test case. Hosted multi-generation runs show per-case pass consistency, mean scores and their spread (unverified) source 2 verdict is all, majority or { atLeast: n } over up to trials counted trials, and earlyStop (default on) ends a case when the remaining trials cannot change the verdict. Core computes 95% Wilson intervals for pass rates and Newcombe intervals for rate changes; trialkit report and trialkit compare show them source 2 3 4 5
Errors apart from failures ahead ignore_errors (default false) ignores exceptions raised while metrics run, and skip_on_missing_params (default false) skips test cases that lack metric inputs. No rule that tells infrastructure errors from wrong answers was found (unverified) source A trial ends in error for infrastructure (SDK HTTP error, timeout, network failure, unreadable judge reply); any other throw is a fail with its message. error stays out of pass rates, errorRetries can retry it, and trialkit compare treats a verdict that turns into error as a regression source
Result caching parity use_cache (default false) reuses cached results that are keyed by test case content and metric configuration. write_cache (default true) writes results to disk source No task or judge response cache. @trialkit/replay records and replays HTTP replies only when you give its wrapper to the client’s fetch source 2
Test runners ahead pytest through deepeval test run and assert_test(); Vitest through npx deepeval test run, which adds the toPass() matcher. The page advises against the plain pytest and vitest commands source Adapters for Vitest, Jest, node:test and pytest run under each runner’s own command. Core also runs without a runner source 2
CI gate ahead deepeval test run fails the build when a metric is below its threshold. deepeval gate exits non-zero when the project fails its Confident AI governance policy source 2 Set checks fail a run on a rate threshold. trialkit compare --fail-on-regression gates a head run against a base run: flips of tagged cases, set-check flips, a verdict that turns into error, a fall of the trial pass rate whose 95% interval is wholly below zero, an incomplete head run, and an optional cost limit. Other flips are listed, not gated source
Feature Status DeepEval trialkit
Datasets in files parity Goldens and test cases load from and save to JSON, JSONL and CSV files (Python also loads Hugging Face datasets). Datasets can also sync with Confident AI source readCases reads JSON Lines and CSV files, @trialkit/cases reads YAML and TOML, and casesFromRows checks rows from a database or an API. A case has id, input, expected, tags, metadata, skip and xfail. trialkit stores no datasets source 2
Case generation parity Python: Synthesizer and deepeval generate create single-turn and multi-turn goldens from documents, contexts, scratch or existing goldens, and write JSON, CSV or JSONL files. The datasets page marks the TypeScript Synthesizer as not yet implemented source 2 generateCases (generate_cases in Python) makes one model call that drafts new cases from seed cases, documents or a description. It asks for 1 to 100 candidates, drops repeats, tags each row generated with provenance, and returns the rows for a person to review. writeCases (write_cases) saves them as a new YAML or JSON Lines draft and never overwrites a file. A reference that a model writes is not ground truth. There is no CLI command: an app script passes its own model client source 2 3 4 5
Case generation: evolutions, document chunking and multi-turn goldens out of scope Python: beyond drafting goldens, the Synthesizer evolves inputs across complexity levels (EvolutionConfig), splits documents into token chunks and groups them into contexts with an embedding model, filters goldens with a critic model (FiltrationConfig), and generates multi-turn (conversational) goldens as well as single-turn goldens. The datasets page marks the TypeScript Synthesizer as not yet implemented source 2 3 None of these. The generator sends the sources whole in one prompt, and a prompt over 64 KiB is refused. It has no multi-turn goldens. The design lists as not built: evolutions, critic loops, embeddings or chunking, PDF parsing and conversation simulation source 2
Conversation simulation out of scope Python: ConversationSimulator plays a fake user against your chatbot callback until max_user_simulations or a stopping controller ends the conversation, and returns conversational test cases. Confident AI adds hosted chat simulations from its Starter plan source 2 No user simulator. The task under test is a plain async function that gets the input of a case source
Red teaming out of scope A separate framework, DeepTeam (deepteam), runs 40+ vulnerabilities and adversarial attacks such as prompt injection and jailbreaking against your app and scores the replies with red-team metrics. It is powered by DeepEval source No red-team generator. Red-team input generators are a non-goal for core, and a plugin is possible later source
Feature Status DeepEval trialkit
Local results parity Results print in the terminal with metric scores and reasons. A run can save as a JSON file in a results folder you set, or in a local SQLite file. Python exports an HTML or Markdown report and TypeScript exports Markdown or MDX. deepeval view opens the run in Confident AI, not in a local page source 2 3 Each run is a JSON Lines file. trialkit report writes HTML (default), Markdown or JSON: pass-rate trends with 95% intervals, case consistency, score histograms, flips, usage and cost, and each run’s cases and trials with scores, errors, task output and tool calls. Nothing is uploaded source
Local trace and log view parity deepeval inspect opens a saved run in a terminal UI: a span tree with inputs and outputs, tool calls, retriever context, metric scores and reasons. It needs a run that captured a trace from an instrumented app; Python needs the inspect extra source ctx.log(message, data?) (Python: ctx.log(message, *, data=None)) records a log entry with a sequence number and a trial-relative offset, and ctx.span records spans. trialkit report shows each trial’s span tree and logs next to its scores, judge metadata and errors, in one self-contained HTML file with static disclosure controls or in Markdown. Model spans show their messages, and tool spans show their arguments and results. Capture is opt-in, and a collapsed value is not redacted. You open the report in a browser or read it as text; it is not a terminal UI source 2 3 4 5 6
Run comparison ahead Confident AI’s Regressions view compares two test runs side by side, matches cases by input or name, and lists regressions and improvements. The CLI page lists no local compare command source 2 trialkit compare base head pairs the latest completed runs by eval and labels, compares the cases that both runs have, lists flips, and shows 95% Newcombe intervals for rate changes. It reads local JSON Lines files source 2
Production tracing out of scope Confident AI traces instrumented apps (a span decorator, framework integrations or OpenTelemetry), runs online evals on live traffic, and adds drift monitors and alerts. The free plan includes 1 GB-month of spans and drops the rest source 2 Core records finished spans for each trial and does not join production tracing. @trialkit/report-otel sends eval spans through your tracer provider source 2
Feature Status DeepEval trialkit
Hosted dashboards out of scope Confident AI shows score distributions, pass/fail results, an AI-generated summary and a drill-down to single test cases for each test run. The free plan allows 5 test runs per week. The Starter plan ($200 a month) adds no-code evaluation workflows, online evals and alerts source 2 No hosted service or account: a non-goal. Reports are local files. The MLflow and OpenTelemetry reporters send results to a backend that you choose source
Human annotation and judge alignment (hosted) out of scope Confident AI collects human annotations and validations of metric results. Its Eval Alignment page shows agreement for each metric, a confusion matrix (too strict, too lenient) and the rank correlation of scores with human ratings. Annotation queues are on the Starter plan source 2 No annotation tool: a non-goal. Calibration reads labels from your own data source
Feature Status DeepEval trialkit
npm footprint ahead deepeval 0.9.25 lists 20 dependencies, 2 optional dependencies (ink, react) and 11 optional peers (including vitest). A dry-run install adds 212 packages source @trialkit/core has no dependencies. An adapter peers on core and your runner, so 2 packages install (0.1.0, unpublished) source 2
PyPI footprint ahead deepeval 4.3.3 lists 31 requirement entries: 27 unconditional (including pytest and four pytest plugins), 2 posthog entries that exclude each other by Python version, and 2 in the optional inspect extra. An entry is a requirement line, not an installed package source trialkit-core has no dependencies. pytest-trialkit requires trialkit-core and pytest (0.1.0, unpublished) source 2
Telemetry ahead Telemetry through PostHog is on by default: event names, metric names, an anonymous id and the public IP address. DEEPEVAL_TELEMETRY_OPT_OUT=1 turns it off. A CONFIDENT_API_KEY also uploads test results and traces to Confident AI source 2 No telemetry, account or hosted service source
Feature Side Source Retrieved Measurement
Repeated trials of a case DeepEval source 2 2026-11-09 Read the Python and TypeScript option tables of the cited documentation and the multi-generation page; feature description, not a live integration test
Repeated trials of a case trialkit source 2 2026-11-09 Read the Trial policies section of docs/guide.md and the run options of packages/core/README.md; documented behavior, not a live run
Verdict rule, intervals and early stop DeepEval source 2 2026-11-09 Read the cited section, the CLI and configuration pages and the multi-generation page, and searched deepeval.com/docs for verdict, majority and interval; found none. Absence is not proof
Verdict rule, intervals and early stop trialkit source 2 3 4 5 2026-11-14 Read the Trial policies and History and reports sections of docs/guide.md, the Statistics section of docs/design.md and packages/core/README.md, and the cited rendering code; documented behavior, not a live run
Errors apart from failures DeepEval source 2026-11-09 Read the cited section, the FAQ and the local-storage page; found no rule that tells infrastructure errors from wrong answers. Absence is not proof
Errors apart from failures trialkit source 2026-11-09 Read the Errors and failures section of docs/guide.md; documented behavior, not a live run
Result caching DeepEval source 2026-11-09 Read the cited documentation; feature description, not a live integration test
Result caching trialkit source 2 2026-11-09 Read the judges and replay sections of docs/guide.md; documented behavior, not a live run
Test runners DeepEval source 2026-11-09 Read the cited documentation; feature description, not a live integration test
Test runners trialkit source 2 2026-11-09 Read docs/guide.md and README.md; documented behavior, not a live run
CI gate DeepEval source 2 2026-11-09 Read the cited documentation; feature description, not a live integration test
CI gate trialkit source 2026-11-09 Read the regression rule in packages/cli/README.md; documented behavior, not a live run
Datasets in files DeepEval source 2026-11-09 Read the cited documentation; feature description, not a live integration test
Datasets in files trialkit source 2 2026-11-09 Read the case readers in packages/core/README.md and packages/cases/README.md; documented behavior, not a live run
Case generation DeepEval source 2 2026-11-09 Read the cited documentation and the generate section of the CLI page; feature description, not a live integration test
Case generation trialkit source 2 3 4 5 2026-11-09 Read the cited sections of packages/judge/README.md, python/judge/README.md, packages/cases/README.md and docs/design.md, and the cited template file; documented behavior, not a live run
Case generation: evolutions, document chunking and multi-turn goldens DeepEval source 2 3 2026-11-09 Read the cited documentation pages; feature description, not a live integration test
Case generation: evolutions, document chunking and multi-turn goldens trialkit source 2 2026-11-09 Read the cited sections of docs/design.md and packages/judge/README.md; documented design, not a live run
Conversation simulation DeepEval source 2 2026-11-09 Read the cited documentation and the pricing page; the simulator page shows Python code only; feature description, not a live integration test
Conversation simulation trialkit source 2026-11-09 Read the goals and non-goals of docs/design.md and searched the repository documentation for simulation; found none
Red teaming DeepEval source 2026-11-09 Read the cited documentation; feature description, not a live integration test
Red teaming trialkit source 2026-11-09 Read the non-goals of docs/design.md
Local results DeepEval source 2 3 2026-11-09 Read the cited documentation; feature description, not a live integration test
Local results trialkit source 2026-11-09 Read the Report section of packages/cli/README.md and the report code in packages/cli/src; documented behavior, not a live run
Local trace and log view DeepEval source 2026-11-09 Read the cited documentation; feature description, not a live integration test
Local trace and log view trialkit source 2 3 4 5 6 2026-11-09 Read the cited sections of packages/cli/README.md, docs/guide.md, packages/core/README.md and spec/trial-logs.md, and the cited code files; documented behavior, not a live run
Run comparison DeepEval source 2 2026-11-09 Read the cited documentation and the command list of the CLI page; feature description, not a live integration test
Run comparison trialkit source 2 2026-11-09 Read the Compare section of packages/cli/README.md and the Comparing runs section of docs/design.md; documented behavior, not a live run
Production tracing DeepEval source 2 2026-11-09 Read the cited documentation and the pricing page; feature description, not a live integration test
Production tracing trialkit source 2 2026-11-14 Read the Spans and trajectories section of docs/design.md and packages/report-otel/README.md; documented design, not a live run
Hosted dashboards DeepEval source 2 2026-11-09 Read the cited documentation and the pricing page; prices and limits can change; feature description, not a live integration test
Hosted dashboards trialkit source 2026-11-09 Read the goals and non-goals of docs/design.md; documented design, not a live run
Human annotation and judge alignment (hosted) DeepEval source 2 2026-11-09 Read the cited documentation and the pricing page; feature description, not a live integration test
Human annotation and judge alignment (hosted) trialkit source 2026-11-09 Read the goals and non-goals of docs/design.md; documented design, not a live run
npm footprint DeepEval source 2026-11-09 Read dependencies, optionalDependencies, peerDependencies and peerDependenciesMeta of the version; install count with npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> deepeval@0.9.25 (Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile)
npm footprint trialkit source 2 2026-11-09 Counted from the manifests, not a registry dry run (unpublished): core and each adapter declare no dependencies; the adapter peers on @trialkit/core and its runner
PyPI footprint DeepEval source 2026-11-09 Count info.requires_dist in the version’s PyPI JSON, split by environment marker and extra; no install measured
PyPI footprint trialkit source 2 2026-11-09 Read dependencies in the local pyproject.toml files; no registry release
Telemetry DeepEval source 2 2026-11-09 Read the cited documentation; feature description, not a live integration test
Telemetry trialkit source 2026-11-09 Read the Why trialkit section of README.md; documented design, not an audit of network calls
Fact Value Source Retrieved Measurement
version 0.125.1 source 2026-11-03 version from npm latest, then pinned to this release URL
vitestPeer Not declared source 2026-11-03 Read peerDependencies.vitest only; absence does not mean runner compatibility
dependencies 76 source 2026-11-03 Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers
installPackages 613 source 2026-11-03 npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> promptfoo@0.125.1; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile
runner Own CLI/config runner source 2026-11-03 Read the cited documentation; feature description, not a live integration test
repeat –repeat source 2026-11-03 Read the cited documentation; feature description, not a live integration test
outcome ERROR filtering and –retry-errors, separate from assertion failures source 2026-11-03 Read the cited documentation; feature description, not a live integration test
history Local results, viewer and exports source 2026-11-03 Read the cited documentation; feature description, not a live integration test
hosted No; cloud sharing optional source 2026-11-03 Read the cited documentation; feature description, not a live integration test
languages JavaScript/TypeScript and YAML source 2026-11-03 Read the cited documentation; feature description, not a live integration test
Fact Value Source Retrieved Measurement
version 1.0.0-beta.16 source 2026-11-03 version of the reviewed 1.0 beta; npm latest is recorded separately
vitestPeer Not declared source 2026-11-03 Read peerDependencies.vitest only; absence does not mean runner compatibility
dependencies 14 source 2026-11-03 Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers
installPackages 129 source 2026-11-03 npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> evalite@1.0.0-beta.16; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile
runner Own CLI/UI, built on Vitest source 2026-11-03 Read the cited documentation; feature description, not a live integration test
repeat trialCount (1.0 beta) source 2026-11-03 Read the cited documentation; feature description, not a live integration test
outcome How a throwing task is recorded is not stated (unverified) source 2026-11-07 Read the cited documentation; feature description, not a live integration test
history SQLite in node_modules/.evalite by default source 2026-11-07 Read the cited documentation; feature description, not a live integration test
hosted No; a local dev server source 2026-11-07 Read the cited documentation; feature description, not a live integration test
languages TypeScript source 2026-11-03 Read the cited documentation; feature description, not a live integration test
latestVersion 0.19.0 source 2026-11-03 Read npm latest separately from the 1.0 beta used for the feature and install measurements
cache cache defaults true (AI SDK model outputs) source 2026-11-03 Read the cited documentation; feature description, not a live integration test
Fact Value Source Retrieved Measurement
version 0.17.0 source 2026-11-03 version from npm latest, then pinned to this release URL
vitestPeer >=4 <5 source 2026-11-03 Read peerDependencies.vitest only; absence does not mean runner compatibility
dependencies 2 source 2026-11-03 Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers
installPackages 48 source 2026-11-03 npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> vitest-evals@0.17.0; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile
runner Vitest extension with harness/report CLI source 2026-11-03 Read the cited documentation; feature description, not a live integration test
repeat No repeat option documented; the harness runs once per eval test (unverified) source 2026-11-07 Read the cited documentation; feature description, not a live integration test
outcome Errors that the harness throws are kept as errors, with partial run data source 2026-11-07 Read the cited documentation; feature description, not a live integration test
history Local JSON reports and viewer source 2026-11-03 Read the cited documentation; feature description, not a live integration test
hosted No; local execution source 2026-11-03 Read the cited documentation; feature description, not a live integration test
languages TypeScript source 2026-11-03 Read the cited documentation; feature description, not a live integration test
Fact Value Source Retrieved Measurement
version 0.5.9 source 2026-11-03 version from npm latest, then pinned to this release URL
vitestPeer Not declared source 2026-11-03 Read peerDependencies.vitest only; absence does not mean runner compatibility
dependencies 5 source 2026-11-03 Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers
installPackages 451 source 2026-11-03 npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> viteval@0.5.9; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile
runner Own viteval CLI, powered by Vitest source 2026-11-03 Read the cited documentation; feature description, not a live integration test
repeat retries: more attempts after a failed case source 2026-11-07 Read the cited documentation; feature description, not a live integration test
outcome Task or scorer exceptions and timeouts fail the case source 2026-11-07 Read the cited documentation; feature description, not a live integration test
history Local datasets/results and dashboard source 2026-11-03 Read the cited documentation; feature description, not a live integration test
hosted No; hosted account not required source 2026-11-03 Read the cited documentation; feature description, not a live integration test
languages TypeScript source 2026-11-03 Read the cited documentation; feature description, not a live integration test
advertisedRelease 1.0.0-rc.6 source 2026-11-03 Read the website badge; this is not npm latest; the release-specific npm endpoint returned HTTP 404
vitestDependency ^4.1.2 source 2026-11-03 Read dependencies.vitest; not a peer range
Fact Value Source Retrieved Measurement
version 3.42.0 source 2026-11-03 version from npm latest, then pinned to this release URL
vitestPeer Not declared source 2026-11-03 Read peerDependencies.vitest only; absence does not mean runner compatibility
dependencies 28 source 2026-11-03 Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers
installPackages 116 source 2026-11-03 npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> braintrust@3.42.0; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile
runner Eval() SDK and bt eval CLI source 2026-11-03 Read the cited documentation; feature description, not a live integration test
repeat trialCount (TS); trial_count (Python) source 2026-11-03 Read the cited documentation; feature description, not a live integration test
outcome Task and scorer errors are recorded as errors; the default handler logs score 0 for scorers that did not run source 2026-11-07 Read the cited documentation; feature description, not a live integration test
history Console locally; experiment history on platform source 2026-11-03 Read the cited documentation; feature description, not a live integration test
hosted No for no-upload runs; platform for experiment history source 2026-11-03 Read the cited documentation; feature description, not a live integration test
languages TypeScript, Python, Go, Ruby, Java and C# source 2026-11-03 Read the cited documentation; feature description, not a live integration test
Fact Value Source Retrieved Measurement
version 0.11.8 source 2026-11-03 version from npm latest, then pinned to this release URL
vitestPeer Not declared source 2026-11-03 Read peerDependencies.vitest only; absence does not mean runner compatibility
dependencies 1 source 2026-11-03 Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers
installPackages 5 source 2026-11-03 npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> langsmith@0.11.8; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile
runner Vitest/Jest and pytest integrations; evaluate() SDK source 2026-11-03 Read the cited documentation; feature description, not a live integration test
repeat No repeat option in the test integrations; the docs show ls.test.each and pytest parametrize (unverified) source 2026-11-07 Read the cited documentation; feature description, not a live integration test
outcome Assertions logged as pass/fail feedback; portable error policy not established (unverified) source 2026-11-03 Read the cited documentation; feature description, not a live integration test
history Runner output locally; tracked datasets/experiments on platform source 2026-11-03 Read the cited documentation; feature description, not a live integration test
hosted No for untracked tests; platform for experiment history source 2026-11-03 Read the cited documentation; feature description, not a live integration test
languages JavaScript/TypeScript and Python source 2026-11-03 Read the cited documentation; feature description, not a live integration test
Fact Value Source Retrieved Measurement
version 7.18.2 source 2026-11-07 version from npm latest, then pinned to this release URL
vitestPeer >=1 (optional) source 2026-11-07 Read peerDependencies.vitest and peerDependenciesMeta; absence does not mean runner compatibility
dependencies 7 source 2026-11-07 Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers
installPackages 57 source 2026-11-07 npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> @arizeai/phoenix-client@7.18.2; added count; Windows x64, Node 22.23.4, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile
runner pytest plugin; Vitest and Jest integrations (TypeScript beta) source 2026-11-07 Read the cited documentation; feature description, not a live integration test
repeat repetitions per test or suite; PHOENIX_TEST_REPETITIONS source 2026-11-07 Read the cited documentation; feature description, not a live integration test
outcome Failed assertions and acceptance criteria fail the test; upload problems only warn; a separate task-error outcome is not stated (unverified) source 2026-11-07 Read the cited documentation; feature description, not a live integration test
history Each suite run becomes a Phoenix experiment source 2026-11-07 Read the cited documentation; feature description, not a live integration test
hosted Phoenix endpoint for tracked runs; PHOENIX_TEST_TRACKING=false runs untracked source 2026-11-07 Read the cited documentation; feature description, not a live integration test
languages Python and TypeScript source 2026-11-07 Read the cited documentation; feature description, not a live integration test
Fact Value Source Retrieved Measurement
version 0.9.25 source 2026-11-09 version from npm latest, then pinned to this release URL
vitestPeer >=1.0.0 (optional) source 2026-11-09 Read peerDependencies.vitest and peerDependenciesMeta; absence does not mean runner compatibility
dependencies 20 source 2026-11-09 Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers
installPackages 212 source 2026-11-09 npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> deepeval@0.9.25; added count; Windows x64, Node 24.22.1, npm 11.19.1 (npm 10.9.9 on Node 22.23.4 also added 212); default optional and peer resolution; no existing manifest or lockfile
runner pytest/Vitest; deepeval test run advised source 2026-11-03 Read the cited documentation; feature description, not a live integration test
repeat Python: deepeval test run –repeat (-r); the TypeScript option table lists no –repeat source 2026-11-09 Read the Python and TypeScript option tables of the cited documentation; feature description, not a live integration test
outcome A metric fails below its threshold; evaluation errors are separate (–ignore-errors) source 2026-11-07 Read the cited documentation; feature description, not a live integration test
history JSON files under .deepeval/ by default, or SQLite source 2026-11-07 Read the cited documentation; feature description, not a live integration test
hosted No; Confident AI upload optional source 2026-11-03 Read the cited documentation; feature description, not a live integration test
languages TypeScript and Python source 2026-11-03 Read the cited documentation; feature description, not a live integration test
pypiVersion 4.3.3 source 2026-11-09 version from PyPI latest (info.version of https://pypi.org/pypi/deepeval/json), then pinned to this release URL
pypiRequirements 31 requirement entries: 27 unconditional, 2 posthog entries that exclude each other by Python version, and 2 in the optional inspect extra source 2026-11-09 Count info.requires_dist in the version’s PyPI JSON, split by environment marker and extra; entries are requirement lines, not installed packages; no install measured
Fact Value Source Retrieved Measurement
version 2.6.0 (unverified) source 2026-11-02 2026-11-02 market review snapshot; PyPI version not rechecked
vitestPeer Not applicable (Python distribution) source 2026-11-03 npm-only measurement; not a Python dependency or install-tree count
dependencies Not applicable (Python distribution) source 2026-11-03 npm-only measurement; not a Python dependency or install-tree count
installPackages Not applicable (Python distribution) source 2026-11-03 npm-only measurement; not a Python dependency or install-tree count
runner Dataset.evaluate() library/CLI source 2026-11-03 Read the cited documentation; feature description, not a live integration test
repeat repeat source 2026-11-03 Read the cited documentation; feature description, not a live integration test
outcome Report failures/exceptions apart from aggregated scores source 2026-11-03 Read the cited documentation; feature description, not a live integration test
history Local report objects; persistence not rechecked (unverified) source 2026-11-03 Read the cited documentation; feature description, not a live integration test
hosted No; Logfire optional (unverified) source 2026-11-03 Read the cited documentation; feature description, not a live integration test
languages Python source 2026-11-03 Read the cited documentation; feature description, not a live integration test
Fact Value Source Retrieved Measurement
version 0.3.287 (unverified) source 2026-11-02 2026-11-02 market review snapshot; PyPI version not rechecked
vitestPeer Not applicable (Python distribution) source 2026-11-03 npm-only measurement; not a Python dependency or install-tree count
dependencies Not applicable (Python distribution) source 2026-11-03 npm-only measurement; not a Python dependency or install-tree count
installPackages Not applicable (Python distribution) source 2026-11-03 npm-only measurement; not a Python dependency or install-tree count
runner Own inspect eval runner and Inspect View source 2026-11-03 Read the cited documentation; feature description, not a live integration test
repeat epochs source 2026-11-03 Read the cited documentation; feature description, not a live integration test
outcome Sample errors; fail_on_error, retry_on_error and score_on_error source 2026-11-03 Read the cited documentation; feature description, not a live integration test
history Local eval/JSON logs and Inspect View source 2026-11-03 Read the cited documentation; feature description, not a live integration test
hosted No; local logs by default source 2026-11-03 Read the cited documentation; feature description, not a live integration test
languages Python source 2026-11-03 Read the cited documentation; feature description, not a live integration test
Fact Value Source Retrieved Measurement
version 1.10.11 source 2026-11-03 version from npm latest, then pinned to this release URL
vitestPeer >=3.0.0 <6.0.0 (optional) source 2026-11-03 Read peerDependencies.vitest only; absence does not mean runner compatibility
dependencies 4 source 2026-11-03 Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers
installPackages 153 source 2026-11-03 npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> @mastra/evals@1.10.11; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile
runner Mastra framework scorers and Vitest helpers source 2026-11-03 Read the cited documentation; feature description, not a live integration test
repeat No repeat option stated for runEvals (unverified) source 2026-11-07 Read the cited documentation; feature description, not a live integration test
outcome Execution errors count apart from low scores (failedCount) source 2026-11-07 Read the cited documentation; feature description, not a live integration test
history Scores in configured database; Studio source 2026-11-03 Read the cited documentation; feature description, not a live integration test
hosted No; configured storage can be local source 2026-11-03 Read the cited documentation; feature description, not a live integration test
languages TypeScript source 2026-11-03 Read the cited documentation; feature description, not a live integration test
Fact Value Source Retrieved Measurement
version 0.1.0 (unpublished) source 2026-11-03 Read local packages/core/package.json and README pre-release notice
vitestPeer ^4.1.12 || ^5.0.3 source 2026-11-07 Read peerDependencies.vitest in local Vitest adapter
dependencies 0 source 2026-11-03 Count dependencies keys in local core manifest; no registry release
installPackages 2 (@trialkit/core and one runner adapter; the runner is a peer you already have) source 2026-11-07 Counted from the manifests, not a registry dry run (unpublished): core and each adapter declare no dependencies; the adapter peers on @trialkit/core and its runner
runner Vitest, Jest, node:test and pytest adapters; standalone core source 2026-11-03 Read local README and docs/guide.md; contract described by shared spec fixtures
repeat trials + verdict; early stop source 2026-11-03 Read local README and docs/guide.md; contract described by shared spec fixtures
outcome Typed error apart from fail through trials, cases and sets source 2026-11-03 Read local README and docs/guide.md; contract described by shared spec fixtures
history Portable TypeScript/Python JSONL; static HTML reports source 2026-11-03 Read local README and docs/guide.md; contract described by shared spec fixtures
hosted No; optional reporters/uploaders source 2026-11-03 Read local README and docs/guide.md; contract described by shared spec fixtures
languages TypeScript and Python source 2026-11-03 Read local README and docs/guide.md; contract described by shared spec fixtures

The generator and pnpm check:docs use local JSON only. They do not refresh facts or make network requests. A person must retrieve the package metadata and feature documentation, record each source URL, retrieval date and measurement method, and set verified to false for anything not established. Then run pnpm docs:comparison.

An optional short label keeps a README cell concise; value keeps the full claim and its evidence here. The README shows ? for an unverified fact and puts trialkit first in bold.

For npm metadata, read https://registry.npmjs.org/<package>/latest (encode a scoped slash as %2f), then record the version-specific URL. Keep prerelease and website release claims separate from latest. Count only dependencies keys and read the exact peerDependencies.vitest field; a Vitest dependency is not a peer.

For install counts, create a new directory under the OS temporary directory and run npm install --dry-run --ignore-scripts --json --prefix <temp-dir> <package>@<version>. npm prints one add line for each package before the JSON object; read added from the object. Record the added count, npm/Node versions, platform and flags in the fact. Do not run the measurement in this repository. Remove the temporary directory afterwards. This is a snapshot, not a lockfile or a claim about the smallest possible install.

Unverified values remain visible. A missing repeat/error fact is not proof that a tool lacks it. Hosted history and local execution are separate facts. These are documentation checks, not live model or service tests.

Each section of features compares trialkit with one tool from tools and is a matrix on this page. Each section of checklists has rows of the same shape and is the metric checklist page of that tool; checklistNotes holds the notes of that page. A row has a feature label, a tool fact, a trialkit fact and a status: ahead, parity, gap or out of scope. Both facts have the fields of any fact, and an optional also list adds source URLs for the same claim. Link a trialkit fact to the file of this repository that states it; its source is always such a file. For work that is not merged, also link its pull request in also, as extra evidence only, and set verified to false until it merges. Set verified to false for a feature that you did not find. A gap also needs a plan: the word planned, or an https link to its issue or design section. An optional group names the sub-heading above the row. Keep the rows of one group next to each other, because each change of group starts a new table. When every gap has a group, a checklist page lists the gaps by group, not by name. To compare another tool, add a section with its name.