@trialkit/studio
Alpha, parked: it works and is tested, but its API and page may change, or the package may be removed. Do not build a workflow that depends on it yet.
A local page to change an eval’s config and cases, run a few trials, compare your changes with the saved config, and save a candidate. Use it when the config or the cases of an eval live outside the code and you want to try a change before you save it. Your app keeps the config and the cases; the studio stores nothing.
Install
Section titled “Install”npm install --save-dev @trialkit/studio @trialkit/core trialkitPeers: @trialkit/core and trialkit. No other dependencies. ESM, and a CommonJS build of @trialkit/studio/draft for test files; Node.js 22.23.4+. No other trialkit package imports the studio. The studio is a development tool; a test file can use studioDraft both inside and outside the studio.
Example
Section titled “Example”A studio file connects an eval that takes a config to functions that your app already has:
import { studio } from '@trialkit/studio';import { type Config, queryTickets, readConfig, writeCandidate, writeTickets } from '../src/db.ts';import { triageEval } from './triage.eval.ts';
export default studio({ eval: (config: Config) => triageEval(client, config), runOptions: (config) => ({ labels: { model: config.model } }), config: { load: () => readConfig('candidate'), save: (next, { loaded }) => writeCandidate(next, { expect: loaded }), }, cases: { load: async () => (await queryTickets()).rows, save: (next, { loaded }) => writeTickets(next, { expect: loaded }), },});Start the server with the file, then open the address that it prints:
npx trialkit-studio evals/triage.studio.ts# http://127.0.0.1:41873/?token=<64 hexadecimal characters>- A run makes real model calls. Before you start it, the page shows about how many calls it makes. Run and compare runs your changes and compares them with the saved config (Run and compare). Run without comparing runs only your changes.
- Save writes through your
savefunctions, after a completed run of your changes without an error case or trial. Save a candidate, not production: promotion stays withtrialkit comparein CI or in a promotion script. - The server listens on
127.0.0.1only. Usessh -Lto reach it on another machine.
A complete app is examples/config-from-db. The design says why it works this way.
Command
Section titled “Command”trialkit-studio <file> [--port <n>] starts the server, prints the address of the page, and runs until Ctrl+C.
| Argument | Default | Meaning |
|---|---|---|
<file> |
Required | The studio file: a module whose default export is the result of studio(). A relative path starts in the current folder. |
--port <n> |
A free port | An integer from 0 to 65535. 0 lets the system choose. |
--help, -h |
None | Prints the usage and exits with code 0. |
A bad argument exits with code 2. A server that cannot start exits with code 1.
The studio file
Section titled “The studio file”studio(definition) checks the definition when you create it and returns the same object. A missing field, a field that is not a function, or a field that it does not know throws a TypeError that names the field. The default export of the file must be this object: any other value fails the load with studio file must default-export a value returned by studio().
| Export | Contract |
|---|---|
studio(definition) |
Returns definition. The generic arguments follow from eval: the config type C is the type of its parameter, so config.load, config.save and runOptions must use the same type. |
StudioDefinition<C, I, O, E> |
The shape of definition: the four parts below. |
StudioRunOptions |
The run options that runOptions can return: labels, tokenBudget, maxErrors, runTimeoutMs and concurrency of core’s RunOptions. See Run options. |
StudioTestDefinition<C> |
The shape of a definition that has test in place of eval. See Test files. |
studioDraft(path) |
Exported from @trialkit/studio/draft. Reads the config, the cases and the options of a studio run in a test file. See Test files. |
Part of StudioDefinition |
Required | Rule |
|---|---|---|
eval(config) |
Yes, or test |
Builds the eval for one config: an EvalDefinition whose task reads the config from this argument. A run replaces dataset.cases with the cases that the page selects and trials with the trial count of the page. dataset.id and the rest of the trial policy stay as eval sets them. Core validates the result before any model call, so an empty selection, or a verdict that the trial count cannot reach, is refused. |
test |
Or eval |
Runs a describeEval test file in your own runner. See Test files. |
runOptions(config) |
No | Returns the StudioRunOptions of a run of this config. |
config.load() |
Yes | Returns the saved config, or a promise of it: a plain object with JSON values. |
config.save(next, { loaded }) |
Yes | Writes next, the config of a completed run. loaded is the value that config.load returned. Throw to refuse a stale write: when the store no longer holds loaded, nothing must be written. |
cases.load() |
No | Returns the rows of your case store, or a promise of them. The rows go through casesFromRows, with the rules of readCases: select the case fields, because a column that is not a case field is an error. Without cases, the page shows the dataset of eval and cannot edit it. |
cases.save(next, { loaded }) |
No | Writes next, the checked cases of the page (EvalCase<unknown, unknown>[]). loaded is the checked cases that cases.load returned. Without it, the cases are read-only. Throw to refuse a stale write. |
- The studio starts a child process for each action (load, run, save), and the child imports your file again. A code edit applies to the next action, and a crash or a hang in your code cannot stop the server. Top-level
awaitworks in the file. - The child writes its protocol on stdout, so
console.login your code goes to stderr. Loader output written before the redirect is retained with stderr unless it is a JSON object withtype: "timeout",type: "event"ortype: "result". When an action fails, the page shows the error, the exit code and the output. The protocol is language-neutral. - The config is shown on the page and recorded in each run. Do not put a secret in it.
eval(config)must check what it uses: a config field can name an endpoint or a tool.
Run options
Section titled “Run options”runOptions(config) returns options for one run. The studio adds its defaults, and the label studio: 'true' replaces any label of that name. As in core, a field set to null or undefined is not supplied.
| Field | Default | Meaning |
|---|---|---|
labels |
{ studio: 'true' } |
String run labels. Reports group runs by them (trialkit report eval-studio --by model). |
tokenBudget |
No budget | Positive limit for the input and output tokens that tasks report with ctx.addUsage. Running trials finish; later work does not start. The page says so when no trial reported usage. |
maxErrors |
10 |
Stop the run after this number of error trials. Core has no limit by default. |
runTimeoutMs |
600000 |
Limit for the whole run, in milliseconds. Core has no limit by default. |
concurrency |
1 |
The number of cases that run at the same time (core’s default). Trials of one case stay in sequence. |
Other options are not available: config comes from the page, signal is the Stop button, the trial count comes from the page, and trialTimeoutMs keeps core’s default of 120 seconds. The reporters write the run to eval-studio/ and send its events to the page.
A run that tokenBudget, maxErrors or runTimeoutMs stops ends as a partial run. It stays in eval-studio/, and the page names the limit that stopped it. A partial run is not a completed run: you cannot save it or compare it.
Test files
Section titled “Test files”Use test in place of eval to run an existing describeEval test file in your own runner. The runner config, setup files, hooks, fixtures and assertions apply.
import { studio } from '@trialkit/studio';import { queryTickets, readConfig, writeCandidate, writeTickets } from '../src/db.ts';
export default studio({ test: { runner: 'vitest', file: './triage.eval.test.ts', args: ['--project', 'evals'] }, config: { load: () => readConfig('candidate'), save: (next, { loaded }) => writeCandidate(next, { expect: loaded }), }, cases: { load: async () => (await queryTickets()).rows, save: (next, { loaded }) => writeTickets(next, { expect: loaded }), },});import { studioDraft } from '@trialkit/studio/draft';import { describeEval } from '@trialkit/vitest';import { expect } from 'vitest';import { type Config, readConfigSync, readTicketsSync } from '../src/db.ts';import { triage } from '../src/triage.ts';
const studioRun = studioDraft<Config, string, string>(process.env['TRIALKIT_STUDIO_DRAFT']); // undefined outside the studioconst config = studioRun?.config ?? readConfigSync('production');const cases = studioRun?.cases ?? readTicketsSync();describeEval( 'triage', { task: (input: string) => triage(client, config, input), trials: 3, ...studioRun?.options, config, }, (it) => { it.for(cases)('$id', async ({ input, expected }, { run }) => expect(await run(input)).toBe(expected), ); },);test.runnerisvitest,jestornode-test.test.fileis relative to the folder of the studio file. Fornode-test,test.argsare Node.js options before the file. For Vitest and Jest, they are runner options after the file; useNODE_OPTIONSfor Node.js flags.testandevalexclude each other, andrunOptionsmay not set a non-nullconcurrencywithtest.- The studio starts the runner in the folder where you started the studio, from your installed runner. The runner gets your environment and
TRIALKIT_STUDIO_DRAFT, the path of the file that holds the config, the cases and the options of the run. This is the one variable that a trialkit package sets, and only for the process that it starts. No package reads a variable; your test file does. studioDraft(path)reads that file synchronously, so Jest in CommonJS can call it where it declares the tests. It returnsundefinedforundefined, and it throws for a file that it cannot read. Otherwise it returns{ config, cases, options }.casesare rows ascasesFromRowsreturns them, orundefinedwhen the studio file has nocasesand the test file declares its own.optionsholds onlydescribeEvaloptions:trials,labels,maxErrors,runTimeoutMs,tokenBudget(when set),filter(when the page selected cases) andreporters(onejsonlFileat an absolute path). Spreadoptionsafter the defaults of the file. The labels replace the labels of the file, because the spread is shallow; they arestudio: 'true'and the labels ofrunOptions. The@trialkit/studio/draftentry has an ESM build and a CommonJS build.- The runner runs the whole file, and the studio selects cases by id, not by name: runners join test names in different ways. The other cases end in
skipwithout a call of the task, and a set check over them is partial. - The studio checks the recorded config, labels and trial count as the run starts, then checks each completed case’s
inputHashas it arrives. It stops the runner on a mismatch, including a selected case skipped without a reason matching one of the configured limits. A skip whose reason matches a limit remains provisional untilRunCompletedconfirms that the same limit stopped the run. It requires exactly one completed run; a selected case missing from the events is an error. A runner that exits with code 1 and recorded failures completed a run, and so does a runner that exits with code 1 aftertokenBudget,maxErrorsorrunTimeoutMsstopped the run: a selected case that did not start is thenskip, with noinputHash. A crash or no run is an error, and so is exit code 1 with no recorded failure and no such stop. The server’s deadline bounds the studio file’s import and setup, then lastsrunTimeoutMsplus 5 seconds from the limit announcement and is re-armed once at the firstRunStarted. The runner has its own cleanup deadline ofrunTimeoutMsplus one second, counted from that event, because core starts the limit there; before that event, the server’s deadline bounds the start of the runner. The event stream does not record the budget and error limit, so the studio cannot verify that the test file applied those options. Stop signals the runner, and the studio ends its process tree after the grace period. - Edited and new cases run only when the test is row-driven:
it.for(cases)that passes the input of the row torun. The studio cannot change a hard-codedit('case', …). - Without
cases, the case list is read-only and comes from the last run. Before the first run, the cases, the policy and the number of calls are unknown, and Run without comparing starts a first run of the whole file. Run and compare needs a selected case.
The page
Section titled “The page”One screen in two columns; below 900 px wide, in one. The server renders it as HTML, and htmx swaps in the parts that change.
| Area | Does |
|---|---|
| Action bar | Stays at the top of the window: the selected cases, the trial count, about how many calls a run makes, Run and compare, Run without comparing, Stop, and one status line for the result of every action. The Run buttons are off when no case is selected or the form is not valid. Before a test file has run once, the page has no cases yet: Run without comparing runs the whole file and lists them. |
| Your changes | Left column. The heading of the column, a status line, and Reset to saved. The status line says Same as saved, or how many config fields and how many cases changed from the saved ones. |
| Config | Left column. A form that follows the value types: a text field (a text area when the value has a line break), a number field or a checkbox. Arrays, objects and null use a JSON text area. A field that you changed shows changed and its saved value. |
| Cases | Left column. The trials for each case, and one row for each case: a checkbox, the id and a short input. A row that you edited shows edited, and a row that you added shows new. With cases.save, a row opens an editor for id, input, expected and tags (a string as text, any other value as JSON), and you can add and remove cases. |
| Trials | Right column. Each case with its trials: the outcome and the scores first, then the output and the trajectory in <details>, clipped. A label says when you changed the config or the cases after the run shown. A comparison shows the trials of the saved config and of your changes side by side, and marks the cases that flipped. |
| Compare | Right column, with the heading Your changes vs saved. It says which run of the saved config the page used, then No gate regression, or the reasons; the flips first, then the rates. See Run and compare. |
| Save | Right column. The changed lines of the config and of the cases, or No changes. Save needs a change and a completed run of your current changes without an error case or trial. |
- The estimate is cases × trials × (1 +
errorRetries) from the policy of the saved config. For a test file the policy comes from its last run. The page says About N calls when a run of the saved config can be reused, and About N calls (2N with the saved config) when Run and compare has to run it too. A config change can change the policy, and scorers and judges can make more calls. One run at a time. - Changes in the config and the cases show where you made them. The status line of Your changes counts the config fields and the cases that differ from the saved ones; a removed case counts, and the page lists no removed case. A case is the same case by its id, so a renamed case counts as one new case and one removed case. While the form has an error, the page cannot compare, and it says so.
- Reset to saved puts the saved config and the saved cases back, also after an error in the form. It keeps the selection of the cases that still exist, and the trial count. It works when nothing changed, replaces an in-flight edit without waiting for it, and discards an edit or Run queued behind that request. It is off only while an action runs.
- The Trials area updates every second while a run is active, and an open
<details>stays open. The status line is one live region (aria-live="polite") that changes its content: it reports the result of each action, for example Saved, and a change of phase (from the saved config to your changes) once. - Stop requests cancellation of a load, a run or a save. Load and save callbacks receive no signal; after one second the studio kills their process tree. A save callback must use an atomic write or transaction because Stop or the deadline can interrupt it mid-write. During Run and compare Stop ends the running side, and the other side does not start. Core ends a running trial in
error, and setsstoppedBy: 'signal'only when the stop kept work from starting. - Your changes live in the memory of the server until a save. Save calls
config.saveand thencases.save, each in a separate child and only when its value changed. These are not one transaction: the config can be saved even when the cases are not. After a save, or a refused save, the studio loads the stored values again and keeps your changes. When a load or save is stopped or times out, retry the load before saving again. When a reload fails, the page keeps your changes, reports the failure, and shows the number of calls as unknown. - Prepare trial detail, then Open trial detail, shows the HTML report of
trialkit reportfor a run on its own page; a comparison has one for each side. The link works once.
Run and compare
Section titled “Run and compare”Run and compare runs your changes and compares them with the saved config. Run without comparing runs only your changes, and the Compare area then says that no comparison was asked for.
- The saved side. The page reuses a run of the saved config from
eval-studio/only when it completed without stopping or an error case, and its eval name, config, complete trial policy, scorer names and versions, and dataset hash match the saved config and the exact selected rows. The selected rows must also equal the loaded rows, and their recorded input hashes must match. A run with additional cases cannot be reused. Test-file datasets do not record full row values and therefore cannot prove a match. The latest matching run wins. Without one, the page runs the saved config first, on the same rows, and then your changes. The Compare area saysSaved config: reused the run from <local time>orSaved config: ran now. - One snapshot. The page snapshots the selected rows (edited or not), the trial count, the saved config and your changes. It runs the saved side, when it needs one, and then your changes, each in its own child, as two ordinary runs in
eval-studio/. A saved side that fails does not stop your changes. - The comparison.
compareRunsgives the flips and the rate changes for exactly these two runs. The page says how many cases it compared. Edited cases are comparable, because both sides ran the same rows. - The verdict. The page says No gate regression, or Regression with its reasons, only when every case ran on both sides, the rows equal the loaded ones, and both recorded dataset hashes match those rows. A selection, rows that differ from the loaded ones, or a run that cannot prove the row values shows the flips and rates without a verdict. The gate of
compareRunsfails a case that goes from pass to fail only when the case has a gated tag (regression), so the page can show a flip next to No gate regression; it names the rule. - A side that fails, stops early or has a case in
error, or two runs with different eval names, give no comparison. The page says why, and it uses no other run. - Save needs a completed run of your current changes with no error case or trial. The run of your changes in a comparison counts.
Trajectories
Section titled “Trajectories”A trial’s recorded spans (ctx.span, spansOfAiSdk, or OpenTelemetry’s spansOf) are its trajectory. The page shows them without a new event field.
- Each trial in the grid has a Trajectory disclosure: the number of model calls, the number of steps, and the first five steps in the order that the trial recorded them. A tool step shows its name, short arguments and its result or error. A model step shows the model, and another span shows its name. A duration shows only when the span has both times.
- A trial without spans says No trajectory recorded. That is not the same as no tool calls.
- All steps opens a panel through an authenticated request: 16 steps for each page, each value clipped to 1 KiB, and at most 64 KiB for each response. More steps loads the next page.
- Every text of a step goes through the escaping helper of the page.
Each run writes one JSON Lines file to eval-studio/ in the folder where you started the studio. A test-file run uses a UUID-only filename because the runner needs an exclusive path before it knows the eval name or run id. Run and compare writes one for your changes, and one for the saved config when it cannot reuse a run. Each run records the config of its side (your changes, or the saved config for the saved side) with its hash, and the label studio: 'true'. The server reads existing history at load, keeps it in memory, and reads each new run’s file directly. Restart the studio, or complete a save that reloads the store, to see files added by another process. Before reuse it reloads the saved definition and reads the candidate file again. No documented gate reads this folder, so a studio run never becomes the head of a pull-request comparison. Add eval-studio/ to .gitignore, or keep it as history. trialkit report eval-studio --by config.model reads the runs.
Security
Section titled “Security”The server is reachable only from the local machine, and only the page that it served can use it.
- It binds to
127.0.0.1and refuses a request whoseHostheader is not its own address, so open the address as printed:localhostand other names get 403. - The address holds a random token of 64 hexadecimal characters. Only a GET of the page takes it from the URL, so a reload or a second tab works. The page then sends it in the
X-Studio-Tokenheader of every request, and the server compares it in constant time. There is no cookie: a cookie on127.0.0.1is shared across ports. A POST also needs anOriginequal to the address andSec-Fetch-Site: same-origin. Each form has a revision; a stale tab gets an error and must reload rather than overwrite a newer edit. A per-tab identifier lets Reset replace that tab’s own in-flight edit without accepting another tab’s stale revision. - Two kinds of request need no token. The static files of the page (
/static/htmx.min.js,/static/studio.jsand/static/studio.css) are served by an exact list, for GET only. A trial report opens through a random id that an authenticated request creates; the id works once. The server keeps at most 32 unopened report links. - Model output is untrusted. Every fragment of the page escapes through one helper. Responses send
Content-Security-Policy: script-src 'self',Referrer-Policy: no-referrerandCache-Control: no-store, and htmx runs withallowEvalandallowScriptTagsoff. - The page sends data, never code or file paths. Writes go only through your
savefunctions, and runs go toeval-studio/. - Remote use. Keep the server on
127.0.0.1and forward the port: start it with--port 4100, runssh -L 4100:127.0.0.1:4100 host, and open the printed address. The ports must be equal, because of theHostcheck. Each person runs their own studio. A shared server needs sign-in, changes for each person and a run lock, which belong to a platform.
TypeScript files
Section titled “TypeScript files”The child runs with the Node.js flags and the environment of the command that started the studio. Node.js 22.23.4 strips types by itself, with these rules: imports name their .ts extension, type-only imports use import type, and Node.js does not strip types in files under node_modules. Syntax that needs a transform fails: enum, namespace, parameter properties, decorators, and paths aliases. Start the studio through your loader for them:
node --import tsx node_modules/@trialkit/studio/dist/esm/bin.js evals/triage.studio.tsNODE_OPTIONS='--import tsx' npx trialkit-studio evals/triage.studio.tsLimits
Section titled “Limits”- One run at a time, one person for each server, and your changes only in memory: a restart loses them.
- The server caps what a child sends. A stdout line over 2 MiB (a trial at core’s 1 MiB span limit fits), more than 10,000 events, or more than 16 MiB of events ends the action with an error. The page keeps up to 64 KiB of stderr and other stdout, including the first 16 KiB and the tail when output is omitted. Load and save children have a 60-second deadline. A run has 10 minutes from the start of its child, which bounds the studio file’s import and
runOptions. WhenrunOptionsresolves, the child announces itsrunTimeoutMs(10 minutes by default), and the deadline becomes that limit plus 5 seconds, counted from the announcement. The firstRunStartedre-arms the same deadline so a slow runner still has time to finish its run. Core’s own limit ends the run first and keeps the partial run; the deadline is the backstop for a child that does not stop. A child that is stopped or reaches its deadline gets one second before the server ends its process tree. A form post is limited to 16 MiB; a load that would produce a larger form is refused. The server owns and removes the temporary folder that holds a test-file run’s unsaved config, cases and options, including when it kills the child. - Not in the studio: a shared server, sign-in or roles, editing code, a store of its own, run history beyond the files of
eval-studio/, running in CI, and promotion. The test runner andtrialkit comparestay the gate.
Page files
Section titled “Page files”The page uses htmx 2.0.11 (0BSD) from static/htmx.min.js, with its license in static/htmx.LICENSE, static/studio.js for request errors, and static/studio.css. A test pins the SHA-256 of the htmx file. Biome excludes static/htmx.min.js so that the vendored bytes stay exact.