Skip to content

15. Studio (alpha, parked)

The studio is a local page for the inner loop: change the model, the prompt or another config value, edit cases, run trials of a few cases, compare with the saved config, and save a candidate. It is what hosted eval platforms sell, without the platform: the app keeps the config and the cases, and trialkit stores nothing. Status: alpha, parked. It gets no new features until its use is clear, and it may change or be removed. It has no shared server, sign-in, run history beyond its files, CI mode or promotion: the test runner and trialkit compare stay the gate. The package README holds the contract, and examples/config-from-db shows a studio file.

A studio file connects functions that an app already has: eval(config) builds the eval for one config, config.load and config.save read and write it, and optional cases.load and cases.save do the same for cases. Why functions, not paths: the data stays where the app keeps it, so the studio does not become a second store. save writes a candidate, not production, and promotion stays with the existing gate, trialkit compare in CI or a promotion script.

A small server renders the page, and each action runs the user’s code in a fresh child process. Why: a code edit applies to the next action, and a crash or a hang in user code cannot stop the server. The protocol between the server and the child is JSON Lines, and it is in spec/ before the code.

“Run and compare” runs your changes and compares them with the saved config. The page reuses a saved run when it fits (conditions), and otherwise runs the saved config now, on the same rows. The comparison is compareRuns of exactly those two runs, and it claims only what they support: edited cases are comparable, because both sides ran the same rows, while a selection, or rows that differ from the loaded ones, give flips and rates but no verdict. If the saved side fails, stops, has an error case or reports another eval name, there is no comparison, and the page never falls back to another run.

A studio over a describeEval file runs the runner on the whole file and selects cases by id, not by name. Why: runners join test names differently (Vitest 4 with a space, Vitest 5 with >, Jest with a space and case-insensitively), so the studio builds no -t patterns. The child passes the draft in a file that TRIALKIT_STUDIO_DRAFT names, and the test file reads it with studioDraft. This is the one place where a trialkit package sets an environment variable, and only for a process that it starts.

One screen in two columns, rendered by the server as HTML fragments that htmx swaps in. Why htmx: the server renders the page, so the logic stays in tested TypeScript and the page has no state to keep in sync. Why polling: it needs no second vendored file, no resume logic and no proxy settings.

The server is reachable only from the local machine, and only the page that it served can use it: it binds to 127.0.0.1, checks Host, and needs a random token on every request, with no cookie, because a cookie on 127.0.0.1 is shared across ports. Model output is untrusted and escapes through one tested helper. A shared, multi-user server is not in this design: it needs sign-in, per-user drafts and a run lock, which belong to a platform, so each person runs their own studio and forwards the port. Why: Vitest’s API server had a critical advisory for remote code execution from a malicious web page while the server was listening. The studio must not repeat that.