Versions, dependency counts and features of trialkit and other eval tools, each with its source, retrieval date and measurement. Snapshot: 2026-11-12.
The data is in comparison.json, and the site renders it when it builds. The DeepEval metric rows are on the DeepEval metric checklist.
A documentation snapshot, not a head-to-head benchmark. Unverified means the cited source did not establish the claim or it was not rechecked; it does not mean the feature is absent.
npm counts describe only the named JavaScript package, not Python distributions or all optional integrations. Install counts include the target package, default optional packages and automatically resolved non-optional peers; they vary with platform, registry state and npm version.
Evalite features and install counts use the 1.0 beta, while npm latest remains 0.19.0. Viteval advertises an rc on its site while npm latest remains 0.5.9; do not treat the website badge as registry availability.
DeepEval ships a Python package (PyPI 4.3.3) and a TypeScript package (npm 0.9.25) whose documented features differ, so a fact or feature row names the package it describes. Confident AI is DeepEval’s optional hosted platform, and DeepTeam is a separate red-teaming package.
promptfoo is part of OpenAI since March 2026 and stays MIT-licensed (https://www.promptfoo.dev/blog/promptfoo-joining-openai). OpenAI shuts down its hosted Evals platform and points to promptfoo (https://developers.openai.com/api/docs/deprecations). The shutdown date is in the migration notes.
20 features with sources. trialkit is ahead on 8 and at parity on 6. It leaves 6 out of scope.
Status shows how trialkit compares with DeepEval:
- ahead: trialkit does it in its open-source packages. The cited DeepEval sources show nothing equal, or show it only with an account.
- parity: both do it at a similar level.
- gap: DeepEval does it and trialkit does not yet. The status links the plan, or says planned.
- out of scope: DeepEval does it. trialkit leaves it to other tools by design (non-goals).
(unverified) marks a claim that the cited source did not establish, such as a feature that was not found. It does not prove that the feature is absent. It also marks a trialkit claim about work that is not merged: only the pull request states it, so the claim stays unverified until the pull request merges.
| Feature |
Status |
DeepEval |
trialkit |
| Repeated trials of a case |
parity |
Python: deepeval test run –repeat n (-r) reruns each test case n times; the TypeScript option table lists no –repeat. Confident AI (hosted) can call your endpoint several times per golden (multi-generation) source 2 |
trials sets how many counted trials each case runs (default 1), and errorRetries adds trials after infrastructure errors only. Trials of one case run in order; cases can run concurrently source 2 |
| Verdict rule, intervals and early stop |
ahead |
No per-case verdict rule (all, majority, at least n), interval or early stop found. A test case marked flaky prints a warning instead of failing the test, and a metric marked flaky never fails its test case. Hosted multi-generation runs show per-case pass consistency, mean scores and their spread (unverified) source 2 |
verdict is all, majority or { atLeast: n } over up to trials counted trials, and earlyStop (default on) ends a case when the remaining trials cannot change the verdict. Core computes 95% Wilson intervals for pass rates and Newcombe intervals for rate changes; trialkit report and trialkit compare show them source 2 3 4 5 |
| Errors apart from failures |
ahead |
ignore_errors (default false) ignores exceptions raised while metrics run, and skip_on_missing_params (default false) skips test cases that lack metric inputs. No rule that tells infrastructure errors from wrong answers was found (unverified) source |
A trial ends in error for infrastructure (SDK HTTP error, timeout, network failure, unreadable judge reply); any other throw is a fail with its message. error stays out of pass rates, errorRetries can retry it, and trialkit compare treats a verdict that turns into error as a regression source |
| Result caching |
parity |
use_cache (default false) reuses cached results that are keyed by test case content and metric configuration. write_cache (default true) writes results to disk source |
No task or judge response cache. @trialkit/replay records and replays HTTP replies only when you give its wrapper to the client’s fetch source 2 |
| Test runners |
ahead |
pytest through deepeval test run and assert_test(); Vitest through npx deepeval test run, which adds the toPass() matcher. The page advises against the plain pytest and vitest commands source |
Adapters for Vitest, Jest, node:test and pytest run under each runner’s own command. Core also runs without a runner source 2 |
| CI gate |
ahead |
deepeval test run fails the build when a metric is below its threshold. deepeval gate exits non-zero when the project fails its Confident AI governance policy source 2 |
Set checks fail a run on a rate threshold. trialkit compare --fail-on-regression gates a head run against a base run: flips of tagged cases, set-check flips, a verdict that turns into error, a fall of the trial pass rate whose 95% interval is wholly below zero, an incomplete head run, and an optional cost limit. Other flips are listed, not gated source |
| Feature |
Status |
DeepEval |
trialkit |
| Datasets in files |
parity |
Goldens and test cases load from and save to JSON, JSONL and CSV files (Python also loads Hugging Face datasets). Datasets can also sync with Confident AI source |
readCases reads JSON Lines and CSV files, @trialkit/cases reads YAML and TOML, and casesFromRows checks rows from a database or an API. A case has id, input, expected, tags, metadata, skip and xfail. trialkit stores no datasets source 2 |
| Case generation |
parity |
Python: Synthesizer and deepeval generate create single-turn and multi-turn goldens from documents, contexts, scratch or existing goldens, and write JSON, CSV or JSONL files. The datasets page marks the TypeScript Synthesizer as not yet implemented source 2 |
generateCases (generate_cases in Python) makes one model call that drafts new cases from seed cases, documents or a description. It asks for 1 to 100 candidates, drops repeats, tags each row generated with provenance, and returns the rows for a person to review. writeCases (write_cases) saves them as a new YAML or JSON Lines draft and never overwrites a file. A reference that a model writes is not ground truth. There is no CLI command: an app script passes its own model client source 2 3 4 5 |
| Case generation: evolutions, document chunking and multi-turn goldens |
out of scope |
Python: beyond drafting goldens, the Synthesizer evolves inputs across complexity levels (EvolutionConfig), splits documents into token chunks and groups them into contexts with an embedding model, filters goldens with a critic model (FiltrationConfig), and generates multi-turn (conversational) goldens as well as single-turn goldens. The datasets page marks the TypeScript Synthesizer as not yet implemented source 2 3 |
None of these. The generator sends the sources whole in one prompt, and a prompt over 64 KiB is refused. It has no multi-turn goldens. The design lists as not built: evolutions, critic loops, embeddings or chunking, PDF parsing and conversation simulation source 2 |
| Conversation simulation |
out of scope |
Python: ConversationSimulator plays a fake user against your chatbot callback until max_user_simulations or a stopping controller ends the conversation, and returns conversational test cases. Confident AI adds hosted chat simulations from its Starter plan source 2 |
No user simulator. The task under test is a plain async function that gets the input of a case source |
| Red teaming |
out of scope |
A separate framework, DeepTeam (deepteam), runs 40+ vulnerabilities and adversarial attacks such as prompt injection and jailbreaking against your app and scores the replies with red-team metrics. It is powered by DeepEval source |
No red-team generator. Red-team input generators are a non-goal for core, and a plugin is possible later source |
| Feature |
Status |
DeepEval |
trialkit |
| Local results |
parity |
Results print in the terminal with metric scores and reasons. A run can save as a JSON file in a results folder you set, or in a local SQLite file. Python exports an HTML or Markdown report and TypeScript exports Markdown or MDX. deepeval view opens the run in Confident AI, not in a local page source 2 3 |
Each run is a JSON Lines file. trialkit report writes HTML (default), Markdown or JSON: pass-rate trends with 95% intervals, case consistency, score histograms, flips, usage and cost, and each run’s cases and trials with scores, errors, task output and tool calls. Nothing is uploaded source |
| Local trace and log view |
parity |
deepeval inspect opens a saved run in a terminal UI: a span tree with inputs and outputs, tool calls, retriever context, metric scores and reasons. It needs a run that captured a trace from an instrumented app; Python needs the inspect extra source |
ctx.log(message, data?) (Python: ctx.log(message, *, data=None)) records a log entry with a sequence number and a trial-relative offset, and ctx.span records spans. trialkit report shows each trial’s span tree and logs next to its scores, judge metadata and errors, in one self-contained HTML file with static disclosure controls or in Markdown. Model spans show their messages, and tool spans show their arguments and results. Capture is opt-in, and a collapsed value is not redacted. You open the report in a browser or read it as text; it is not a terminal UI source 2 3 4 5 6 |
| Run comparison |
ahead |
Confident AI’s Regressions view compares two test runs side by side, matches cases by input or name, and lists regressions and improvements. The CLI page lists no local compare command source 2 |
trialkit compare base head pairs the latest completed runs by eval and labels, compares the cases that both runs have, lists flips, and shows 95% Newcombe intervals for rate changes. It reads local JSON Lines files source 2 |
| Production tracing |
out of scope |
Confident AI traces instrumented apps (a span decorator, framework integrations or OpenTelemetry), runs online evals on live traffic, and adds drift monitors and alerts. The free plan includes 1 GB-month of spans and drops the rest source 2 |
Core records finished spans for each trial and does not join production tracing. @trialkit/report-otel sends eval spans through your tracer provider source 2 |
| Feature |
Status |
DeepEval |
trialkit |
| Hosted dashboards |
out of scope |
Confident AI shows score distributions, pass/fail results, an AI-generated summary and a drill-down to single test cases for each test run. The free plan allows 5 test runs per week. The Starter plan ($200 a month) adds no-code evaluation workflows, online evals and alerts source 2 |
No hosted service or account: a non-goal. Reports are local files. The MLflow and OpenTelemetry reporters send results to a backend that you choose source |
| Human annotation and judge alignment (hosted) |
out of scope |
Confident AI collects human annotations and validations of metric results. Its Eval Alignment page shows agreement for each metric, a confusion matrix (too strict, too lenient) and the rank correlation of scores with human ratings. Annotation queues are on the Starter plan source 2 |
No annotation tool: a non-goal. Calibration reads labels from your own data source |
| Feature |
Status |
DeepEval |
trialkit |
| npm footprint |
ahead |
deepeval 0.9.25 lists 20 dependencies, 2 optional dependencies (ink, react) and 11 optional peers (including vitest). A dry-run install adds 212 packages source |
@trialkit/core has no dependencies. An adapter peers on core and your runner, so 2 packages install (0.1.0, unpublished) source 2 |
| PyPI footprint |
ahead |
deepeval 4.3.3 lists 31 requirement entries: 27 unconditional (including pytest and four pytest plugins), 2 posthog entries that exclude each other by Python version, and 2 in the optional inspect extra. An entry is a requirement line, not an installed package source |
trialkit-core has no dependencies. pytest-trialkit requires trialkit-core and pytest (0.1.0, unpublished) source 2 |
| Telemetry |
ahead |
Telemetry through PostHog is on by default: event names, metric names, an anonymous id and the public IP address. DEEPEVAL_TELEMETRY_OPT_OUT=1 turns it off. A CONFIDENT_API_KEY also uploads test results and traces to Confident AI source 2 |
No telemetry, account or hosted service source |
| Feature |
Side |
Source |
Retrieved |
Measurement |
| Repeated trials of a case |
DeepEval |
source 2 |
2026-11-09 |
Read the Python and TypeScript option tables of the cited documentation and the multi-generation page; feature description, not a live integration test |
| Repeated trials of a case |
trialkit |
source 2 |
2026-11-09 |
Read the Trial policies section of docs/guide.md and the run options of packages/core/README.md; documented behavior, not a live run |
| Verdict rule, intervals and early stop |
DeepEval |
source 2 |
2026-11-09 |
Read the cited section, the CLI and configuration pages and the multi-generation page, and searched deepeval.com/docs for verdict, majority and interval; found none. Absence is not proof |
| Verdict rule, intervals and early stop |
trialkit |
source 2 3 4 5 |
2026-11-14 |
Read the Trial policies and History and reports sections of docs/guide.md, the Statistics section of docs/design.md and packages/core/README.md, and the cited rendering code; documented behavior, not a live run |
| Errors apart from failures |
DeepEval |
source |
2026-11-09 |
Read the cited section, the FAQ and the local-storage page; found no rule that tells infrastructure errors from wrong answers. Absence is not proof |
| Errors apart from failures |
trialkit |
source |
2026-11-09 |
Read the Errors and failures section of docs/guide.md; documented behavior, not a live run |
| Result caching |
DeepEval |
source |
2026-11-09 |
Read the cited documentation; feature description, not a live integration test |
| Result caching |
trialkit |
source 2 |
2026-11-09 |
Read the judges and replay sections of docs/guide.md; documented behavior, not a live run |
| Test runners |
DeepEval |
source |
2026-11-09 |
Read the cited documentation; feature description, not a live integration test |
| Test runners |
trialkit |
source 2 |
2026-11-09 |
Read docs/guide.md and README.md; documented behavior, not a live run |
| CI gate |
DeepEval |
source 2 |
2026-11-09 |
Read the cited documentation; feature description, not a live integration test |
| CI gate |
trialkit |
source |
2026-11-09 |
Read the regression rule in packages/cli/README.md; documented behavior, not a live run |
| Datasets in files |
DeepEval |
source |
2026-11-09 |
Read the cited documentation; feature description, not a live integration test |
| Datasets in files |
trialkit |
source 2 |
2026-11-09 |
Read the case readers in packages/core/README.md and packages/cases/README.md; documented behavior, not a live run |
| Case generation |
DeepEval |
source 2 |
2026-11-09 |
Read the cited documentation and the generate section of the CLI page; feature description, not a live integration test |
| Case generation |
trialkit |
source 2 3 4 5 |
2026-11-09 |
Read the cited sections of packages/judge/README.md, python/judge/README.md, packages/cases/README.md and docs/design.md, and the cited template file; documented behavior, not a live run |
| Case generation: evolutions, document chunking and multi-turn goldens |
DeepEval |
source 2 3 |
2026-11-09 |
Read the cited documentation pages; feature description, not a live integration test |
| Case generation: evolutions, document chunking and multi-turn goldens |
trialkit |
source 2 |
2026-11-09 |
Read the cited sections of docs/design.md and packages/judge/README.md; documented design, not a live run |
| Conversation simulation |
DeepEval |
source 2 |
2026-11-09 |
Read the cited documentation and the pricing page; the simulator page shows Python code only; feature description, not a live integration test |
| Conversation simulation |
trialkit |
source |
2026-11-09 |
Read the goals and non-goals of docs/design.md and searched the repository documentation for simulation; found none |
| Red teaming |
DeepEval |
source |
2026-11-09 |
Read the cited documentation; feature description, not a live integration test |
| Red teaming |
trialkit |
source |
2026-11-09 |
Read the non-goals of docs/design.md |
| Local results |
DeepEval |
source 2 3 |
2026-11-09 |
Read the cited documentation; feature description, not a live integration test |
| Local results |
trialkit |
source |
2026-11-09 |
Read the Report section of packages/cli/README.md and the report code in packages/cli/src; documented behavior, not a live run |
| Local trace and log view |
DeepEval |
source |
2026-11-09 |
Read the cited documentation; feature description, not a live integration test |
| Local trace and log view |
trialkit |
source 2 3 4 5 6 |
2026-11-09 |
Read the cited sections of packages/cli/README.md, docs/guide.md, packages/core/README.md and spec/trial-logs.md, and the cited code files; documented behavior, not a live run |
| Run comparison |
DeepEval |
source 2 |
2026-11-09 |
Read the cited documentation and the command list of the CLI page; feature description, not a live integration test |
| Run comparison |
trialkit |
source 2 |
2026-11-09 |
Read the Compare section of packages/cli/README.md and the Comparing runs section of docs/design.md; documented behavior, not a live run |
| Production tracing |
DeepEval |
source 2 |
2026-11-09 |
Read the cited documentation and the pricing page; feature description, not a live integration test |
| Production tracing |
trialkit |
source 2 |
2026-11-14 |
Read the Spans and trajectories section of docs/design.md and packages/report-otel/README.md; documented design, not a live run |
| Hosted dashboards |
DeepEval |
source 2 |
2026-11-09 |
Read the cited documentation and the pricing page; prices and limits can change; feature description, not a live integration test |
| Hosted dashboards |
trialkit |
source |
2026-11-09 |
Read the goals and non-goals of docs/design.md; documented design, not a live run |
| Human annotation and judge alignment (hosted) |
DeepEval |
source 2 |
2026-11-09 |
Read the cited documentation and the pricing page; feature description, not a live integration test |
| Human annotation and judge alignment (hosted) |
trialkit |
source |
2026-11-09 |
Read the goals and non-goals of docs/design.md; documented design, not a live run |
| npm footprint |
DeepEval |
source |
2026-11-09 |
Read dependencies, optionalDependencies, peerDependencies and peerDependenciesMeta of the version; install count with npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> deepeval@0.9.25 (Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile) |
| npm footprint |
trialkit |
source 2 |
2026-11-09 |
Counted from the manifests, not a registry dry run (unpublished): core and each adapter declare no dependencies; the adapter peers on @trialkit/core and its runner |
| PyPI footprint |
DeepEval |
source |
2026-11-09 |
Count info.requires_dist in the version’s PyPI JSON, split by environment marker and extra; no install measured |
| PyPI footprint |
trialkit |
source 2 |
2026-11-09 |
Read dependencies in the local pyproject.toml files; no registry release |
| Telemetry |
DeepEval |
source 2 |
2026-11-09 |
Read the cited documentation; feature description, not a live integration test |
| Telemetry |
trialkit |
source |
2026-11-09 |
Read the Why trialkit section of README.md; documented design, not an audit of network calls |
| Fact |
Value |
Source |
Retrieved |
Measurement |
| version |
0.125.1 |
source |
2026-11-03 |
version from npm latest, then pinned to this release URL |
| vitestPeer |
Not declared |
source |
2026-11-03 |
Read peerDependencies.vitest only; absence does not mean runner compatibility |
| dependencies |
76 |
source |
2026-11-03 |
Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers |
| installPackages |
613 |
source |
2026-11-03 |
npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> promptfoo@0.125.1; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile |
| runner |
Own CLI/config runner |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| repeat |
–repeat |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| outcome |
ERROR filtering and –retry-errors, separate from assertion failures |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| history |
Local results, viewer and exports |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| hosted |
No; cloud sharing optional |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| languages |
JavaScript/TypeScript and YAML |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| Fact |
Value |
Source |
Retrieved |
Measurement |
| version |
1.0.0-beta.16 |
source |
2026-11-03 |
version of the reviewed 1.0 beta; npm latest is recorded separately |
| vitestPeer |
Not declared |
source |
2026-11-03 |
Read peerDependencies.vitest only; absence does not mean runner compatibility |
| dependencies |
14 |
source |
2026-11-03 |
Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers |
| installPackages |
129 |
source |
2026-11-03 |
npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> evalite@1.0.0-beta.16; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile |
| runner |
Own CLI/UI, built on Vitest |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| repeat |
trialCount (1.0 beta) |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| outcome |
How a throwing task is recorded is not stated (unverified) |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| history |
SQLite in node_modules/.evalite by default |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| hosted |
No; a local dev server |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| languages |
TypeScript |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| latestVersion |
0.19.0 |
source |
2026-11-03 |
Read npm latest separately from the 1.0 beta used for the feature and install measurements |
| cache |
cache defaults true (AI SDK model outputs) |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| Fact |
Value |
Source |
Retrieved |
Measurement |
| version |
0.17.0 |
source |
2026-11-03 |
version from npm latest, then pinned to this release URL |
| vitestPeer |
>=4 <5 |
source |
2026-11-03 |
Read peerDependencies.vitest only; absence does not mean runner compatibility |
| dependencies |
2 |
source |
2026-11-03 |
Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers |
| installPackages |
48 |
source |
2026-11-03 |
npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> vitest-evals@0.17.0; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile |
| runner |
Vitest extension with harness/report CLI |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| repeat |
No repeat option documented; the harness runs once per eval test (unverified) |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| outcome |
Errors that the harness throws are kept as errors, with partial run data |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| history |
Local JSON reports and viewer |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| hosted |
No; local execution |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| languages |
TypeScript |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| Fact |
Value |
Source |
Retrieved |
Measurement |
| version |
0.5.9 |
source |
2026-11-03 |
version from npm latest, then pinned to this release URL |
| vitestPeer |
Not declared |
source |
2026-11-03 |
Read peerDependencies.vitest only; absence does not mean runner compatibility |
| dependencies |
5 |
source |
2026-11-03 |
Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers |
| installPackages |
451 |
source |
2026-11-03 |
npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> viteval@0.5.9; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile |
| runner |
Own viteval CLI, powered by Vitest |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| repeat |
retries: more attempts after a failed case |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| outcome |
Task or scorer exceptions and timeouts fail the case |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| history |
Local datasets/results and dashboard |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| hosted |
No; hosted account not required |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| languages |
TypeScript |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| advertisedRelease |
1.0.0-rc.6 |
source |
2026-11-03 |
Read the website badge; this is not npm latest; the release-specific npm endpoint returned HTTP 404 |
| vitestDependency |
^4.1.2 |
source |
2026-11-03 |
Read dependencies.vitest; not a peer range |
| Fact |
Value |
Source |
Retrieved |
Measurement |
| version |
3.42.0 |
source |
2026-11-03 |
version from npm latest, then pinned to this release URL |
| vitestPeer |
Not declared |
source |
2026-11-03 |
Read peerDependencies.vitest only; absence does not mean runner compatibility |
| dependencies |
28 |
source |
2026-11-03 |
Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers |
| installPackages |
116 |
source |
2026-11-03 |
npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> braintrust@3.42.0; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile |
| runner |
Eval() SDK and bt eval CLI |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| repeat |
trialCount (TS); trial_count (Python) |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| outcome |
Task and scorer errors are recorded as errors; the default handler logs score 0 for scorers that did not run |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| history |
Console locally; experiment history on platform |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| hosted |
No for no-upload runs; platform for experiment history |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| languages |
TypeScript, Python, Go, Ruby, Java and C# |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| Fact |
Value |
Source |
Retrieved |
Measurement |
| version |
0.11.8 |
source |
2026-11-03 |
version from npm latest, then pinned to this release URL |
| vitestPeer |
Not declared |
source |
2026-11-03 |
Read peerDependencies.vitest only; absence does not mean runner compatibility |
| dependencies |
1 |
source |
2026-11-03 |
Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers |
| installPackages |
5 |
source |
2026-11-03 |
npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> langsmith@0.11.8; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile |
| runner |
Vitest/Jest and pytest integrations; evaluate() SDK |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| repeat |
No repeat option in the test integrations; the docs show ls.test.each and pytest parametrize (unverified) |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| outcome |
Assertions logged as pass/fail feedback; portable error policy not established (unverified) |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| history |
Runner output locally; tracked datasets/experiments on platform |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| hosted |
No for untracked tests; platform for experiment history |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| languages |
JavaScript/TypeScript and Python |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| Fact |
Value |
Source |
Retrieved |
Measurement |
| version |
7.18.2 |
source |
2026-11-07 |
version from npm latest, then pinned to this release URL |
| vitestPeer |
>=1 (optional) |
source |
2026-11-07 |
Read peerDependencies.vitest and peerDependenciesMeta; absence does not mean runner compatibility |
| dependencies |
7 |
source |
2026-11-07 |
Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers |
| installPackages |
57 |
source |
2026-11-07 |
npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> @arizeai/phoenix-client@7.18.2; added count; Windows x64, Node 22.23.4, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile |
| runner |
pytest plugin; Vitest and Jest integrations (TypeScript beta) |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| repeat |
repetitions per test or suite; PHOENIX_TEST_REPETITIONS |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| outcome |
Failed assertions and acceptance criteria fail the test; upload problems only warn; a separate task-error outcome is not stated (unverified) |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| history |
Each suite run becomes a Phoenix experiment |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| hosted |
Phoenix endpoint for tracked runs; PHOENIX_TEST_TRACKING=false runs untracked |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| languages |
Python and TypeScript |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| Fact |
Value |
Source |
Retrieved |
Measurement |
| version |
0.9.25 |
source |
2026-11-09 |
version from npm latest, then pinned to this release URL |
| vitestPeer |
>=1.0.0 (optional) |
source |
2026-11-09 |
Read peerDependencies.vitest and peerDependenciesMeta; absence does not mean runner compatibility |
| dependencies |
20 |
source |
2026-11-09 |
Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers |
| installPackages |
212 |
source |
2026-11-09 |
npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> deepeval@0.9.25; added count; Windows x64, Node 24.22.1, npm 11.19.1 (npm 10.9.9 on Node 22.23.4 also added 212); default optional and peer resolution; no existing manifest or lockfile |
| runner |
pytest/Vitest; deepeval test run advised |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| repeat |
Python: deepeval test run –repeat (-r); the TypeScript option table lists no –repeat |
source |
2026-11-09 |
Read the Python and TypeScript option tables of the cited documentation; feature description, not a live integration test |
| outcome |
A metric fails below its threshold; evaluation errors are separate (–ignore-errors) |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| history |
JSON files under .deepeval/ by default, or SQLite |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| hosted |
No; Confident AI upload optional |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| languages |
TypeScript and Python |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| pypiVersion |
4.3.3 |
source |
2026-11-09 |
version from PyPI latest (info.version of https://pypi.org/pypi/deepeval/json), then pinned to this release URL |
| pypiRequirements |
31 requirement entries: 27 unconditional, 2 posthog entries that exclude each other by Python version, and 2 in the optional inspect extra |
source |
2026-11-09 |
Count info.requires_dist in the version’s PyPI JSON, split by environment marker and extra; entries are requirement lines, not installed packages; no install measured |
| Fact |
Value |
Source |
Retrieved |
Measurement |
| version |
2.6.0 (unverified) |
source |
2026-11-02 |
2026-11-02 market review snapshot; PyPI version not rechecked |
| vitestPeer |
Not applicable (Python distribution) |
source |
2026-11-03 |
npm-only measurement; not a Python dependency or install-tree count |
| dependencies |
Not applicable (Python distribution) |
source |
2026-11-03 |
npm-only measurement; not a Python dependency or install-tree count |
| installPackages |
Not applicable (Python distribution) |
source |
2026-11-03 |
npm-only measurement; not a Python dependency or install-tree count |
| runner |
Dataset.evaluate() library/CLI |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| repeat |
repeat |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| outcome |
Report failures/exceptions apart from aggregated scores |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| history |
Local report objects; persistence not rechecked (unverified) |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| hosted |
No; Logfire optional (unverified) |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| languages |
Python |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| Fact |
Value |
Source |
Retrieved |
Measurement |
| version |
0.3.287 (unverified) |
source |
2026-11-02 |
2026-11-02 market review snapshot; PyPI version not rechecked |
| vitestPeer |
Not applicable (Python distribution) |
source |
2026-11-03 |
npm-only measurement; not a Python dependency or install-tree count |
| dependencies |
Not applicable (Python distribution) |
source |
2026-11-03 |
npm-only measurement; not a Python dependency or install-tree count |
| installPackages |
Not applicable (Python distribution) |
source |
2026-11-03 |
npm-only measurement; not a Python dependency or install-tree count |
| runner |
Own inspect eval runner and Inspect View |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| repeat |
epochs |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| outcome |
Sample errors; fail_on_error, retry_on_error and score_on_error |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| history |
Local eval/JSON logs and Inspect View |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| hosted |
No; local logs by default |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| languages |
Python |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| Fact |
Value |
Source |
Retrieved |
Measurement |
| version |
1.10.11 |
source |
2026-11-03 |
version from npm latest, then pinned to this release URL |
| vitestPeer |
>=3.0.0 <6.0.0 (optional) |
source |
2026-11-03 |
Read peerDependencies.vitest only; absence does not mean runner compatibility |
| dependencies |
4 |
source |
2026-11-03 |
Count Object.keys(dependencies ?? {}); excludes optionalDependencies and peers |
| installPackages |
153 |
source |
2026-11-03 |
npm install –dry-run –ignore-scripts –json –prefix <new OS temp dir> @mastra/evals@1.10.11; added count; Windows x64, Node 24.22.1, npm 11.19.1; default optional and peer resolution; no existing manifest or lockfile |
| runner |
Mastra framework scorers and Vitest helpers |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| repeat |
No repeat option stated for runEvals (unverified) |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| outcome |
Execution errors count apart from low scores (failedCount) |
source |
2026-11-07 |
Read the cited documentation; feature description, not a live integration test |
| history |
Scores in configured database; Studio |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| hosted |
No; configured storage can be local |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| languages |
TypeScript |
source |
2026-11-03 |
Read the cited documentation; feature description, not a live integration test |
| Fact |
Value |
Source |
Retrieved |
Measurement |
| version |
0.1.0 (unpublished) |
source |
2026-11-03 |
Read local packages/core/package.json and README pre-release notice |
| vitestPeer |
^4.1.12 || ^5.0.3 |
source |
2026-11-07 |
Read peerDependencies.vitest in local Vitest adapter |
| dependencies |
0 |
source |
2026-11-03 |
Count dependencies keys in local core manifest; no registry release |
| installPackages |
2 (@trialkit/core and one runner adapter; the runner is a peer you already have) |
source |
2026-11-07 |
Counted from the manifests, not a registry dry run (unpublished): core and each adapter declare no dependencies; the adapter peers on @trialkit/core and its runner |
| runner |
Vitest, Jest, node:test and pytest adapters; standalone core |
source |
2026-11-03 |
Read local README and docs/guide.md; contract described by shared spec fixtures |
| repeat |
trials + verdict; early stop |
source |
2026-11-03 |
Read local README and docs/guide.md; contract described by shared spec fixtures |
| outcome |
Typed error apart from fail through trials, cases and sets |
source |
2026-11-03 |
Read local README and docs/guide.md; contract described by shared spec fixtures |
| history |
Portable TypeScript/Python JSONL; static HTML reports |
source |
2026-11-03 |
Read local README and docs/guide.md; contract described by shared spec fixtures |
| hosted |
No; optional reporters/uploaders |
source |
2026-11-03 |
Read local README and docs/guide.md; contract described by shared spec fixtures |
| languages |
TypeScript and Python |
source |
2026-11-03 |
Read local README and docs/guide.md; contract described by shared spec fixtures |
The generator and pnpm check:docs use local JSON only. They do not refresh facts or make network requests. A person must retrieve the package metadata and feature documentation, record each source URL, retrieval date and measurement method, and set verified to false for anything not established. Then run pnpm docs:comparison.
An optional short label keeps a README cell concise; value keeps the full claim and its evidence here. The README shows ? for an unverified fact and puts trialkit first in bold.
For npm metadata, read https://registry.npmjs.org/<package>/latest (encode a scoped slash as %2f), then record the version-specific URL. Keep prerelease and website release claims separate from latest. Count only dependencies keys and read the exact peerDependencies.vitest field; a Vitest dependency is not a peer.
For install counts, create a new directory under the OS temporary directory and run npm install --dry-run --ignore-scripts --json --prefix <temp-dir> <package>@<version>. npm prints one add line for each package before the JSON object; read added from the object. Record the added count, npm/Node versions, platform and flags in the fact. Do not run the measurement in this repository. Remove the temporary directory afterwards. This is a snapshot, not a lockfile or a claim about the smallest possible install.
Unverified values remain visible. A missing repeat/error fact is not proof that a tool lacks it. Hosted history and local execution are separate facts. These are documentation checks, not live model or service tests.
Each section of features compares trialkit with one tool from tools and is a matrix on this page. Each section of checklists has rows of the same shape and is the metric checklist page of that tool; checklistNotes holds the notes of that page. A row has a feature label, a tool fact, a trialkit fact and a status: ahead, parity, gap or out of scope. Both facts have the fields of any fact, and an optional also list adds source URLs for the same claim. Link a trialkit fact to the file of this repository that states it; its source is always such a file. For work that is not merged, also link its pull request in also, as extra evidence only, and set verified to false until it merges. Set verified to false for a feature that you did not find. A gap also needs a plan: the word planned, or an https link to its issue or design section. An optional group names the sub-heading above the row. Keep the rows of one group next to each other, because each change of group starts a new table. When every gap has a group, a checklist page lists the gaps by group, not by name. To compare another tool, add a section with its name.