Skip to content

@trialkit/studio

Alpha, parked: it works and is tested, but its API and page may change, or the package may be removed. Do not build a workflow that depends on it yet.

A local page to change an eval’s config and cases, run a few trials, compare your changes with the saved config, and save a candidate. Use it when the config or the cases of an eval live outside the code and you want to try a change before you save it. Your app keeps the config and the cases; the studio stores nothing.

Terminal window
npm install --save-dev @trialkit/studio @trialkit/core trialkit

Peers: @trialkit/core and trialkit. No other dependencies. ESM, and a CommonJS build of @trialkit/studio/draft for test files; Node.js 22.23.4+. No other trialkit package imports the studio. The studio is a development tool; a test file can use studioDraft both inside and outside the studio.

A studio file connects an eval that takes a config to functions that your app already has:

evals/triage.studio.ts
import { studio } from '@trialkit/studio';
import { type Config, queryTickets, readConfig, writeCandidate, writeTickets } from '../src/db.ts';
import { triageEval } from './triage.eval.ts';
export default studio({
eval: (config: Config) => triageEval(client, config),
runOptions: (config) => ({ labels: { model: config.model } }),
config: {
load: () => readConfig('candidate'),
save: (next, { loaded }) => writeCandidate(next, { expect: loaded }),
},
cases: {
load: async () => (await queryTickets()).rows,
save: (next, { loaded }) => writeTickets(next, { expect: loaded }),
},
});

Start the server with the file, then open the address that it prints:

Terminal window
npx trialkit-studio evals/triage.studio.ts
# http://127.0.0.1:41873/?token=<64 hexadecimal characters>
  • A run makes real model calls. Before you start it, the page shows about how many calls it makes. Run and compare runs your changes and compares them with the saved config (Run and compare). Run without comparing runs only your changes.
  • Save writes through your save functions, after a completed run of your changes without an error case or trial. Save a candidate, not production: promotion stays with trialkit compare in CI or in a promotion script.
  • The server listens on 127.0.0.1 only. Use ssh -L to reach it on another machine.

A complete app is examples/config-from-db. The design says why it works this way.

trialkit-studio <file> [--port <n>] starts the server, prints the address of the page, and runs until Ctrl+C.

Argument Default Meaning
<file> Required The studio file: a module whose default export is the result of studio(). A relative path starts in the current folder.
--port <n> A free port An integer from 0 to 65535. 0 lets the system choose.
--help, -h None Prints the usage and exits with code 0.

A bad argument exits with code 2. A server that cannot start exits with code 1.

studio(definition) checks the definition when you create it and returns the same object. A missing field, a field that is not a function, or a field that it does not know throws a TypeError that names the field. The default export of the file must be this object: any other value fails the load with studio file must default-export a value returned by studio().

Export Contract
studio(definition) Returns definition. The generic arguments follow from eval: the config type C is the type of its parameter, so config.load, config.save and runOptions must use the same type.
StudioDefinition<C, I, O, E> The shape of definition: the four parts below.
StudioRunOptions The run options that runOptions can return: labels, tokenBudget, maxErrors, runTimeoutMs and concurrency of core’s RunOptions. See Run options.
StudioTestDefinition<C> The shape of a definition that has test in place of eval. See Test files.
studioDraft(path) Exported from @trialkit/studio/draft. Reads the config, the cases and the options of a studio run in a test file. See Test files.
Part of StudioDefinition Required Rule
eval(config) Yes, or test Builds the eval for one config: an EvalDefinition whose task reads the config from this argument. A run replaces dataset.cases with the cases that the page selects and trials with the trial count of the page. dataset.id and the rest of the trial policy stay as eval sets them. Core validates the result before any model call, so an empty selection, or a verdict that the trial count cannot reach, is refused.
test Or eval Runs a describeEval test file in your own runner. See Test files.
runOptions(config) No Returns the StudioRunOptions of a run of this config.
config.load() Yes Returns the saved config, or a promise of it: a plain object with JSON values.
config.save(next, { loaded }) Yes Writes next, the config of a completed run. loaded is the value that config.load returned. Throw to refuse a stale write: when the store no longer holds loaded, nothing must be written.
cases.load() No Returns the rows of your case store, or a promise of them. The rows go through casesFromRows, with the rules of readCases: select the case fields, because a column that is not a case field is an error. Without cases, the page shows the dataset of eval and cannot edit it.
cases.save(next, { loaded }) No Writes next, the checked cases of the page (EvalCase<unknown, unknown>[]). loaded is the checked cases that cases.load returned. Without it, the cases are read-only. Throw to refuse a stale write.
  • The studio starts a child process for each action (load, run, save), and the child imports your file again. A code edit applies to the next action, and a crash or a hang in your code cannot stop the server. Top-level await works in the file.
  • The child writes its protocol on stdout, so console.log in your code goes to stderr. Loader output written before the redirect is retained with stderr unless it is a JSON object with type: "timeout", type: "event" or type: "result". When an action fails, the page shows the error, the exit code and the output. The protocol is language-neutral.
  • The config is shown on the page and recorded in each run. Do not put a secret in it. eval(config) must check what it uses: a config field can name an endpoint or a tool.

runOptions(config) returns options for one run. The studio adds its defaults, and the label studio: 'true' replaces any label of that name. As in core, a field set to null or undefined is not supplied.

Field Default Meaning
labels { studio: 'true' } String run labels. Reports group runs by them (trialkit report eval-studio --by model).
tokenBudget No budget Positive limit for the input and output tokens that tasks report with ctx.addUsage. Running trials finish; later work does not start. The page says so when no trial reported usage.
maxErrors 10 Stop the run after this number of error trials. Core has no limit by default.
runTimeoutMs 600000 Limit for the whole run, in milliseconds. Core has no limit by default.
concurrency 1 The number of cases that run at the same time (core’s default). Trials of one case stay in sequence.

Other options are not available: config comes from the page, signal is the Stop button, the trial count comes from the page, and trialTimeoutMs keeps core’s default of 120 seconds. The reporters write the run to eval-studio/ and send its events to the page.

A run that tokenBudget, maxErrors or runTimeoutMs stops ends as a partial run. It stays in eval-studio/, and the page names the limit that stopped it. A partial run is not a completed run: you cannot save it or compare it.

Use test in place of eval to run an existing describeEval test file in your own runner. The runner config, setup files, hooks, fixtures and assertions apply.

evals/triage.studio.ts
import { studio } from '@trialkit/studio';
import { queryTickets, readConfig, writeCandidate, writeTickets } from '../src/db.ts';
export default studio({
test: { runner: 'vitest', file: './triage.eval.test.ts', args: ['--project', 'evals'] },
config: {
load: () => readConfig('candidate'),
save: (next, { loaded }) => writeCandidate(next, { expect: loaded }),
},
cases: {
load: async () => (await queryTickets()).rows,
save: (next, { loaded }) => writeTickets(next, { expect: loaded }),
},
});
evals/triage.eval.test.ts
import { studioDraft } from '@trialkit/studio/draft';
import { describeEval } from '@trialkit/vitest';
import { expect } from 'vitest';
import { type Config, readConfigSync, readTicketsSync } from '../src/db.ts';
import { triage } from '../src/triage.ts';
const studioRun = studioDraft<Config, string, string>(process.env['TRIALKIT_STUDIO_DRAFT']); // undefined outside the studio
const config = studioRun?.config ?? readConfigSync('production');
const cases = studioRun?.cases ?? readTicketsSync();
describeEval(
'triage',
{
task: (input: string) => triage(client, config, input),
trials: 3,
...studioRun?.options,
config,
},
(it) => {
it.for(cases)('$id', async ({ input, expected }, { run }) =>
expect(await run(input)).toBe(expected),
);
},
);
  • test.runner is vitest, jest or node-test. test.file is relative to the folder of the studio file. For node-test, test.args are Node.js options before the file. For Vitest and Jest, they are runner options after the file; use NODE_OPTIONS for Node.js flags. test and eval exclude each other, and runOptions may not set a non-null concurrency with test.
  • The studio starts the runner in the folder where you started the studio, from your installed runner. The runner gets your environment and TRIALKIT_STUDIO_DRAFT, the path of the file that holds the config, the cases and the options of the run. This is the one variable that a trialkit package sets, and only for the process that it starts. No package reads a variable; your test file does.
  • studioDraft(path) reads that file synchronously, so Jest in CommonJS can call it where it declares the tests. It returns undefined for undefined, and it throws for a file that it cannot read. Otherwise it returns { config, cases, options }. cases are rows as casesFromRows returns them, or undefined when the studio file has no cases and the test file declares its own. options holds only describeEval options: trials, labels, maxErrors, runTimeoutMs, tokenBudget (when set), filter (when the page selected cases) and reporters (one jsonlFile at an absolute path). Spread options after the defaults of the file. The labels replace the labels of the file, because the spread is shallow; they are studio: 'true' and the labels of runOptions. The @trialkit/studio/draft entry has an ESM build and a CommonJS build.
  • The runner runs the whole file, and the studio selects cases by id, not by name: runners join test names in different ways. The other cases end in skip without a call of the task, and a set check over them is partial.
  • The studio checks the recorded config, labels and trial count as the run starts, then checks each completed case’s inputHash as it arrives. It stops the runner on a mismatch, including a selected case skipped without a reason matching one of the configured limits. A skip whose reason matches a limit remains provisional until RunCompleted confirms that the same limit stopped the run. It requires exactly one completed run; a selected case missing from the events is an error. A runner that exits with code 1 and recorded failures completed a run, and so does a runner that exits with code 1 after tokenBudget, maxErrors or runTimeoutMs stopped the run: a selected case that did not start is then skip, with no inputHash. A crash or no run is an error, and so is exit code 1 with no recorded failure and no such stop. The server’s deadline bounds the studio file’s import and setup, then lasts runTimeoutMs plus 5 seconds from the limit announcement and is re-armed once at the first RunStarted. The runner has its own cleanup deadline of runTimeoutMs plus one second, counted from that event, because core starts the limit there; before that event, the server’s deadline bounds the start of the runner. The event stream does not record the budget and error limit, so the studio cannot verify that the test file applied those options. Stop signals the runner, and the studio ends its process tree after the grace period.
  • Edited and new cases run only when the test is row-driven: it.for(cases) that passes the input of the row to run. The studio cannot change a hard-coded it('case', …).
  • Without cases, the case list is read-only and comes from the last run. Before the first run, the cases, the policy and the number of calls are unknown, and Run without comparing starts a first run of the whole file. Run and compare needs a selected case.

One screen in two columns; below 900 px wide, in one. The server renders it as HTML, and htmx swaps in the parts that change.

Area Does
Action bar Stays at the top of the window: the selected cases, the trial count, about how many calls a run makes, Run and compare, Run without comparing, Stop, and one status line for the result of every action. The Run buttons are off when no case is selected or the form is not valid. Before a test file has run once, the page has no cases yet: Run without comparing runs the whole file and lists them.
Your changes Left column. The heading of the column, a status line, and Reset to saved. The status line says Same as saved, or how many config fields and how many cases changed from the saved ones.
Config Left column. A form that follows the value types: a text field (a text area when the value has a line break), a number field or a checkbox. Arrays, objects and null use a JSON text area. A field that you changed shows changed and its saved value.
Cases Left column. The trials for each case, and one row for each case: a checkbox, the id and a short input. A row that you edited shows edited, and a row that you added shows new. With cases.save, a row opens an editor for id, input, expected and tags (a string as text, any other value as JSON), and you can add and remove cases.
Trials Right column. Each case with its trials: the outcome and the scores first, then the output and the trajectory in <details>, clipped. A label says when you changed the config or the cases after the run shown. A comparison shows the trials of the saved config and of your changes side by side, and marks the cases that flipped.
Compare Right column, with the heading Your changes vs saved. It says which run of the saved config the page used, then No gate regression, or the reasons; the flips first, then the rates. See Run and compare.
Save Right column. The changed lines of the config and of the cases, or No changes. Save needs a change and a completed run of your current changes without an error case or trial.
  • The estimate is cases × trials × (1 + errorRetries) from the policy of the saved config. For a test file the policy comes from its last run. The page says About N calls when a run of the saved config can be reused, and About N calls (2N with the saved config) when Run and compare has to run it too. A config change can change the policy, and scorers and judges can make more calls. One run at a time.
  • Changes in the config and the cases show where you made them. The status line of Your changes counts the config fields and the cases that differ from the saved ones; a removed case counts, and the page lists no removed case. A case is the same case by its id, so a renamed case counts as one new case and one removed case. While the form has an error, the page cannot compare, and it says so.
  • Reset to saved puts the saved config and the saved cases back, also after an error in the form. It keeps the selection of the cases that still exist, and the trial count. It works when nothing changed, replaces an in-flight edit without waiting for it, and discards an edit or Run queued behind that request. It is off only while an action runs.
  • The Trials area updates every second while a run is active, and an open <details> stays open. The status line is one live region (aria-live="polite") that changes its content: it reports the result of each action, for example Saved, and a change of phase (from the saved config to your changes) once.
  • Stop requests cancellation of a load, a run or a save. Load and save callbacks receive no signal; after one second the studio kills their process tree. A save callback must use an atomic write or transaction because Stop or the deadline can interrupt it mid-write. During Run and compare Stop ends the running side, and the other side does not start. Core ends a running trial in error, and sets stoppedBy: 'signal' only when the stop kept work from starting.
  • Your changes live in the memory of the server until a save. Save calls config.save and then cases.save, each in a separate child and only when its value changed. These are not one transaction: the config can be saved even when the cases are not. After a save, or a refused save, the studio loads the stored values again and keeps your changes. When a load or save is stopped or times out, retry the load before saving again. When a reload fails, the page keeps your changes, reports the failure, and shows the number of calls as unknown.
  • Prepare trial detail, then Open trial detail, shows the HTML report of trialkit report for a run on its own page; a comparison has one for each side. The link works once.

Run and compare runs your changes and compares them with the saved config. Run without comparing runs only your changes, and the Compare area then says that no comparison was asked for.

  • The saved side. The page reuses a run of the saved config from eval-studio/ only when it completed without stopping or an error case, and its eval name, config, complete trial policy, scorer names and versions, and dataset hash match the saved config and the exact selected rows. The selected rows must also equal the loaded rows, and their recorded input hashes must match. A run with additional cases cannot be reused. Test-file datasets do not record full row values and therefore cannot prove a match. The latest matching run wins. Without one, the page runs the saved config first, on the same rows, and then your changes. The Compare area says Saved config: reused the run from <local time> or Saved config: ran now.
  • One snapshot. The page snapshots the selected rows (edited or not), the trial count, the saved config and your changes. It runs the saved side, when it needs one, and then your changes, each in its own child, as two ordinary runs in eval-studio/. A saved side that fails does not stop your changes.
  • The comparison. compareRuns gives the flips and the rate changes for exactly these two runs. The page says how many cases it compared. Edited cases are comparable, because both sides ran the same rows.
  • The verdict. The page says No gate regression, or Regression with its reasons, only when every case ran on both sides, the rows equal the loaded ones, and both recorded dataset hashes match those rows. A selection, rows that differ from the loaded ones, or a run that cannot prove the row values shows the flips and rates without a verdict. The gate of compareRuns fails a case that goes from pass to fail only when the case has a gated tag (regression), so the page can show a flip next to No gate regression; it names the rule.
  • A side that fails, stops early or has a case in error, or two runs with different eval names, give no comparison. The page says why, and it uses no other run.
  • Save needs a completed run of your current changes with no error case or trial. The run of your changes in a comparison counts.

A trial’s recorded spans (ctx.span, spansOfAiSdk, or OpenTelemetry’s spansOf) are its trajectory. The page shows them without a new event field.

  • Each trial in the grid has a Trajectory disclosure: the number of model calls, the number of steps, and the first five steps in the order that the trial recorded them. A tool step shows its name, short arguments and its result or error. A model step shows the model, and another span shows its name. A duration shows only when the span has both times.
  • A trial without spans says No trajectory recorded. That is not the same as no tool calls.
  • All steps opens a panel through an authenticated request: 16 steps for each page, each value clipped to 1 KiB, and at most 64 KiB for each response. More steps loads the next page.
  • Every text of a step goes through the escaping helper of the page.

Each run writes one JSON Lines file to eval-studio/ in the folder where you started the studio. A test-file run uses a UUID-only filename because the runner needs an exclusive path before it knows the eval name or run id. Run and compare writes one for your changes, and one for the saved config when it cannot reuse a run. Each run records the config of its side (your changes, or the saved config for the saved side) with its hash, and the label studio: 'true'. The server reads existing history at load, keeps it in memory, and reads each new run’s file directly. Restart the studio, or complete a save that reloads the store, to see files added by another process. Before reuse it reloads the saved definition and reads the candidate file again. No documented gate reads this folder, so a studio run never becomes the head of a pull-request comparison. Add eval-studio/ to .gitignore, or keep it as history. trialkit report eval-studio --by config.model reads the runs.

The server is reachable only from the local machine, and only the page that it served can use it.

  • It binds to 127.0.0.1 and refuses a request whose Host header is not its own address, so open the address as printed: localhost and other names get 403.
  • The address holds a random token of 64 hexadecimal characters. Only a GET of the page takes it from the URL, so a reload or a second tab works. The page then sends it in the X-Studio-Token header of every request, and the server compares it in constant time. There is no cookie: a cookie on 127.0.0.1 is shared across ports. A POST also needs an Origin equal to the address and Sec-Fetch-Site: same-origin. Each form has a revision; a stale tab gets an error and must reload rather than overwrite a newer edit. A per-tab identifier lets Reset replace that tab’s own in-flight edit without accepting another tab’s stale revision.
  • Two kinds of request need no token. The static files of the page (/static/htmx.min.js, /static/studio.js and /static/studio.css) are served by an exact list, for GET only. A trial report opens through a random id that an authenticated request creates; the id works once. The server keeps at most 32 unopened report links.
  • Model output is untrusted. Every fragment of the page escapes through one helper. Responses send Content-Security-Policy: script-src 'self', Referrer-Policy: no-referrer and Cache-Control: no-store, and htmx runs with allowEval and allowScriptTags off.
  • The page sends data, never code or file paths. Writes go only through your save functions, and runs go to eval-studio/.
  • Remote use. Keep the server on 127.0.0.1 and forward the port: start it with --port 4100, run ssh -L 4100:127.0.0.1:4100 host, and open the printed address. The ports must be equal, because of the Host check. Each person runs their own studio. A shared server needs sign-in, changes for each person and a run lock, which belong to a platform.

The child runs with the Node.js flags and the environment of the command that started the studio. Node.js 22.23.4 strips types by itself, with these rules: imports name their .ts extension, type-only imports use import type, and Node.js does not strip types in files under node_modules. Syntax that needs a transform fails: enum, namespace, parameter properties, decorators, and paths aliases. Start the studio through your loader for them:

Terminal window
node --import tsx node_modules/@trialkit/studio/dist/esm/bin.js evals/triage.studio.ts
NODE_OPTIONS='--import tsx' npx trialkit-studio evals/triage.studio.ts
  • One run at a time, one person for each server, and your changes only in memory: a restart loses them.
  • The server caps what a child sends. A stdout line over 2 MiB (a trial at core’s 1 MiB span limit fits), more than 10,000 events, or more than 16 MiB of events ends the action with an error. The page keeps up to 64 KiB of stderr and other stdout, including the first 16 KiB and the tail when output is omitted. Load and save children have a 60-second deadline. A run has 10 minutes from the start of its child, which bounds the studio file’s import and runOptions. When runOptions resolves, the child announces its runTimeoutMs (10 minutes by default), and the deadline becomes that limit plus 5 seconds, counted from the announcement. The first RunStarted re-arms the same deadline so a slow runner still has time to finish its run. Core’s own limit ends the run first and keeps the partial run; the deadline is the backstop for a child that does not stop. A child that is stopped or reaches its deadline gets one second before the server ends its process tree. A form post is limited to 16 MiB; a load that would produce a larger form is refused. The server owns and removes the temporary folder that holds a test-file run’s unsaved config, cases and options, including when it kills the child.
  • Not in the studio: a shared server, sign-in or roles, editing code, a store of its own, run history beyond the files of eval-studio/, running in CI, and promotion. The test runner and trialkit compare stay the gate.

The page uses htmx 2.0.11 (0BSD) from static/htmx.min.js, with its license in static/htmx.LICENSE, static/studio.js for request errors, and static/studio.css. A test pins the SHA-256 of the htmx file. Biome excludes static/htmx.min.js so that the vendored bytes stay exact.