← trialkit docs: History and reports

trialkit report

1 eval, 4 runs, from 2026-11-02 05:03 to 2026-11-02 05:03 UTC.

Series by config · configName.

ticket-category

Dataset: ticket-category 663d8221 (7 cases).

Trial pass rate over time

100%80%60%40% trial pass rate (the axis starts at 40%) 05:03:2505:03:2805:03:3105:03:3405:03:37 sha256:3224dc6fbb33… · candidate 2026-11-02 05:03 UTC, run 81fb8293-356d-4f61-bf80-af3a09fb5632 trial pass rate 71.4% (50.0–86.2%), 15/21 trialssha256:3224dc6fbb33… · candidate 2026-11-02 05:03 UTC, run f6552169-59ac-4fa4-ba1b-5d2e6f4b6c04 trial pass rate 71.4% (50.0–86.2%), 15/21 trials sha256:e45aace9b6d9… · production 2026-11-02 05:03 UTC, run 05b9d3af-aafc-4e90-86c3-c6db4465a9ea trial pass rate 100.0% (84.5–100.0%), 21/21 trialssha256:e45aace9b6d9… · production 2026-11-02 05:03 UTC, run e81bb33f-e10b-45ed-b49e-3da2bb2d39c5 trial pass rate 95.2% (77.3–99.2%), 20/21 trials
  • sha256:3224dc6fbb33… · candidate (2 runs)
  • sha256:e45aace9b6d9… · production (2 runs)
X: start time of the run (UTC). Y: passed / (passed + failed) trials of the run. Whisker: 95% Wilson interval. Hollow diamond, dashed line and a break in the line: the dataset or the trial policy differs from the previous run of the series, so the rates do not compare. Dotted line with a value: the run where prompt changed to that value.

Latest completed run per series

Select the start time of a run to see its cases and trials.

SeriespromptRunsCasesCase pass rate (95% CI)Trial pass rate (95% CI)TrendError trialsMean trialLatest run (UTC)
sha256:3224dc6fbb33… · candidate-25 pass, 2 fail71.4% 5/7
35.9–91.8%
71.4% 15/21
50.0–86.2%
▆▆0447 ms2026-11-02 05:03
sha256:e45aace9b6d9… · production-27 pass100.0% 7/7
64.6–100.0%
95.2% 20/21
77.3–99.2%
██0648 ms2026-11-02 05:03

Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.

All 4 runs in the window
SeriespromptStarted (UTC)Trial pass rate (95% CI)Case pass rate (95% CI)Error trialsCommitNotes
sha256:3224dc6fbb33… · candidate- 2026-11-02 05:03 71.4% 15/21
50.0–86.2%
71.4% 5/7
35.9–91.8%
0 –
- 2026-11-02 05:03 71.4% 15/21
50.0–86.2%
71.4% 5/7
35.9–91.8%
0 –
sha256:e45aace9b6d9… · production- 2026-11-02 05:03 100.0% 21/21
84.5–100.0%
100.0% 7/7
64.6–100.0%
0 –
- 2026-11-02 05:03 95.2% 20/21
77.3–99.2%
100.0% 7/7
64.6–100.0%
0 –

Case consistency

Per case, the trial pass rate over the completed runs in the window. At 0% and 100%, the task gives the same result each time; the bins between hold the cases where it is not consistent.

sha256:3224dc6fbb33… · candidate
7 cases, 2 runs
0%: 2 of 7 cases (28.6%)2(0, 25%]: 0 of 7 cases (0.0%)(25%, 50%]: 0 of 7 cases (0.0%)(50%, 75%]: 0 of 7 cases (0.0%)(75%, 100%): 0 of 7 cases (0.0%)100%: 5 of 7 cases (71.4%)5 0%≤25%≤50%≤75%<100%100%
sha256:e45aace9b6d9… · production
7 cases, 2 runs
0%: 0 of 7 cases (0.0%)(0, 25%]: 0 of 7 cases (0.0%)(25%, 50%]: 0 of 7 cases (0.0%)(50%, 75%]: 0 of 7 cases (0.0%)(75%, 100%): 1 of 7 cases (14.3%)1100%: 6 of 7 cases (85.7%)6 0%≤25%≤50%≤75%<100%100%

Score distributions

Per scorer, the score values of all trials of the completed runs in the window, in 10 bins from 0 to 1.

category@1

sha256:3224dc6fbb33… · candidate
42 scores
[0.0, 0.1): 12 of 42 scores (28.6%)12[0.1, 0.2): 0 of 42 scores (0.0%)[0.2, 0.3): 0 of 42 scores (0.0%)[0.3, 0.4): 0 of 42 scores (0.0%)[0.4, 0.5): 0 of 42 scores (0.0%)[0.5, 0.6): 0 of 42 scores (0.0%)[0.6, 0.7): 0 of 42 scores (0.0%)[0.7, 0.8): 0 of 42 scores (0.0%)[0.8, 0.9): 0 of 42 scores (0.0%)[0.9, 1.0]: 30 of 42 scores (71.4%)30 0.1.2.3.4.5.6.7.8.91
sha256:e45aace9b6d9… · production
42 scores
[0.0, 0.1): 1 of 42 scores (2.4%)1[0.1, 0.2): 0 of 42 scores (0.0%)[0.2, 0.3): 0 of 42 scores (0.0%)[0.3, 0.4): 0 of 42 scores (0.0%)[0.4, 0.5): 0 of 42 scores (0.0%)[0.5, 0.6): 0 of 42 scores (0.0%)[0.6, 0.7): 0 of 42 scores (0.0%)[0.7, 0.8): 0 of 42 scores (0.0%)[0.8, 0.9): 0 of 42 scores (0.0%)[0.9, 1.0]: 41 of 42 scores (97.6%)41 0.1.2.3.4.5.6.7.8.91

Flips

No flips since the previous completed run of each series.

← Back to the report

ticket-category

sha256:3224dc6fbb33… · candidate

Started 2026-11-02 05:03 UTC · run 81fb8293-356d-4f61-bf80-af3a09fb5632

Source JSONL: examples/config-from-db/history/ticket-category.81fb8293-356d-4f61-bf80-af3a09fb5632.jsonl

5 pass, 2 fail

Tokens in / out / cached / reasoning: unknown / unknown / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=z-ai/glm-5.3-flash configName=candidate

Config sha256:3224dc6fbb33…
model
z-ai/glm-5.3-flash
effort
low
systemPrompt
You triage customer support tickets for a file-sharing app. Reply with the category of the ticket, one of: - billing: a charge, a refund, an invoice, a price or a payment method. - account: login, password, profile, team members or the plan. - bug: an error, a crash or a wrong result in the app. - o…Show all 471 charactersShow less
You triage customer support tickets for a file-sharing app. Reply with the category of the ticket, one of: - billing: a charge, a refund, an invoice, a price or a payment method. - account: login, password, profile, team members or the plan. - bug: an error, a crash or a wrong result in the app. - other: praise, a feature request, a question about how to use the app, or anything else. When a ticket states two problems, use the category of the first one in the ticket.
firstPrompt
Classify this support ticket: {{ticket}}
responseSchema
{ "type": "object", "properties": { "category": { "type": "string", "enum": [ "billing", "account", "bug", "other" ] } }, "required": [ "category" ], "additionalProperties": false }

Cases

crash-and-charge fail (0/3)

Tags: regression

0 of 3 trials passed, needs 2; category: expected "billing", got "bug"
Trial 1 fail
Input
The export to CSV crashes every time. Also, you charged my card twice this month.
Output
{ "category": "bug" }
Scores
  • category@1 fail 0.00
    expected "billing", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 2 fail
Input
The export to CSV crashes every time. Also, you charged my card twice this month.
Output
{ "category": "bug" }
Scores
  • category@1 fail 0.00
    expected "billing", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input
The export to CSV crashes every time. Also, you charged my card twice this month.
Output
{ "category": "bug" }
Scores
  • category@1 fail 0.00
    expected "billing", got "bug"
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: plain

Trial 1 pass
Input
I was charged twice for my March subscription. Please refund one of the charges.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
I was charged twice for my March subscription. Please refund one of the charges.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
I was charged twice for my March subscription. Please refund one of the charges.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

feature-and-login fail (0/3)

Tags: regression

0 of 3 trials passed, needs 2; category: expected "account", got "other"
Trial 1 fail
Input
Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output
{ "category": "other" }
Scores
  • category@1 fail 0.00
    expected "account", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 fail
Input
Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output
{ "category": "other" }
Scores
  • category@1 fail 0.00
    expected "account", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input
Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output
{ "category": "other" }
Scores
  • category@1 fail 0.00
    expected "account", got "other"
Trace

Not recorded

Logs

Not recorded

login-after-update pass (3/3)

Tags: regression

Trial 1 pass
Input
Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

paid-still-free pass (3/3)

Tags: regression

Trial 1 pass
Input
You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

praise pass (3/3)

Tags: plain

Trial 1 pass
Input
Thanks, the new search is great!
Output
{ "category": "other" }
Scores
  • category@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Thanks, the new search is great!
Output
{ "category": "other" }
Scores
  • category@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Thanks, the new search is great!
Output
{ "category": "other" }
Scores
  • category@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

share-error pass (3/3)

Tags: plain

Trial 1 pass
Input
When I click Share, the page shows an error 500 and no link appears.
Output
{ "category": "bug" }
Scores
  • category@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
When I click Share, the page shows an error 500 and no link appears.
Output
{ "category": "bug" }
Scores
  • category@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
When I click Share, the page shows an error 500 and no link appears.
Output
{ "category": "bug" }
Scores
  • category@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

← Back to the report

ticket-category

sha256:3224dc6fbb33… · candidate

Started 2026-11-02 05:03 UTC · run f6552169-59ac-4fa4-ba1b-5d2e6f4b6c04

Source JSONL: examples/config-from-db/history/ticket-category.f6552169-59ac-4fa4-ba1b-5d2e6f4b6c04.jsonl

5 pass, 2 fail

Tokens in / out / cached / reasoning: unknown / unknown / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=z-ai/glm-5.3-flash configName=candidate

Config sha256:3224dc6fbb33…
model
z-ai/glm-5.3-flash
effort
low
systemPrompt
You triage customer support tickets for a file-sharing app. Reply with the category of the ticket, one of: - billing: a charge, a refund, an invoice, a price or a payment method. - account: login, password, profile, team members or the plan. - bug: an error, a crash or a wrong result in the app. - o…Show all 471 charactersShow less
You triage customer support tickets for a file-sharing app. Reply with the category of the ticket, one of: - billing: a charge, a refund, an invoice, a price or a payment method. - account: login, password, profile, team members or the plan. - bug: an error, a crash or a wrong result in the app. - other: praise, a feature request, a question about how to use the app, or anything else. When a ticket states two problems, use the category of the first one in the ticket.
firstPrompt
Classify this support ticket: {{ticket}}
responseSchema
{ "type": "object", "properties": { "category": { "type": "string", "enum": [ "billing", "account", "bug", "other" ] } }, "required": [ "category" ], "additionalProperties": false }

Cases

crash-and-charge fail (0/3)

Tags: regression

0 of 3 trials passed, needs 2; category: expected "billing", got "bug"
Trial 1 fail
Input
The export to CSV crashes every time. Also, you charged my card twice this month.
Output
{ "category": "bug" }
Scores
  • category@1 fail 0.00
    expected "billing", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 2 fail
Input
The export to CSV crashes every time. Also, you charged my card twice this month.
Output
{ "category": "bug" }
Scores
  • category@1 fail 0.00
    expected "billing", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input
The export to CSV crashes every time. Also, you charged my card twice this month.
Output
{ "category": "bug" }
Scores
  • category@1 fail 0.00
    expected "billing", got "bug"
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: plain

Trial 1 pass
Input
I was charged twice for my March subscription. Please refund one of the charges.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
I was charged twice for my March subscription. Please refund one of the charges.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
I was charged twice for my March subscription. Please refund one of the charges.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

feature-and-login fail (0/3)

Tags: regression

0 of 3 trials passed, needs 2; category: expected "account", got "other"
Trial 1 fail
Input
Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output
{ "category": "other" }
Scores
  • category@1 fail 0.00
    expected "account", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 fail
Input
Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output
{ "category": "other" }
Scores
  • category@1 fail 0.00
    expected "account", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input
Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output
{ "category": "other" }
Scores
  • category@1 fail 0.00
    expected "account", got "other"
Trace

Not recorded

Logs

Not recorded

login-after-update pass (3/3)

Tags: regression

Trial 1 pass
Input
Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

paid-still-free pass (3/3)

Tags: regression

Trial 1 pass
Input
You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

praise pass (3/3)

Tags: plain

Trial 1 pass
Input
Thanks, the new search is great!
Output
{ "category": "other" }
Scores
  • category@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Thanks, the new search is great!
Output
{ "category": "other" }
Scores
  • category@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Thanks, the new search is great!
Output
{ "category": "other" }
Scores
  • category@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

share-error pass (3/3)

Tags: plain

Trial 1 pass
Input
When I click Share, the page shows an error 500 and no link appears.
Output
{ "category": "bug" }
Scores
  • category@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
When I click Share, the page shows an error 500 and no link appears.
Output
{ "category": "bug" }
Scores
  • category@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
When I click Share, the page shows an error 500 and no link appears.
Output
{ "category": "bug" }
Scores
  • category@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

← Back to the report

ticket-category

sha256:e45aace9b6d9… · production

Started 2026-11-02 05:03 UTC · run 05b9d3af-aafc-4e90-86c3-c6db4465a9ea

Source JSONL: examples/config-from-db/history/ticket-category.05b9d3af-aafc-4e90-86c3-c6db4465a9ea.jsonl

7 pass

Tokens in / out / cached / reasoning: unknown / unknown / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=z-ai/glm-5.3-flash configName=production

Config sha256:e45aace9b6d9…
model
z-ai/glm-5.3-flash
effort
medium
systemPrompt
You triage customer support tickets for a file-sharing app. Reply with the category of the ticket, one of: - billing: a charge, a refund, an invoice, a price or a payment method. - account: login, password, profile, team members or the plan. - bug: an error, a crash or a wrong result in the app. - o…Show all 512 charactersShow less
You triage customer support tickets for a file-sharing app. Reply with the category of the ticket, one of: - billing: a charge, a refund, an invoice, a price or a payment method. - account: login, password, profile, team members or the plan. - bug: an error, a crash or a wrong result in the app. - other: praise, a feature request, a question about how to use the app, or anything else. When a ticket states two problems, or one problem fits two categories, use the first of these: billing, account, bug, other.
firstPrompt
Classify this support ticket: {{ticket}}
responseSchema
{ "type": "object", "properties": { "category": { "type": "string", "enum": [ "billing", "account", "bug", "other" ] } }, "required": [ "category" ], "additionalProperties": false }

Cases

crash-and-charge pass (3/3)

Tags: regression

Trial 1 pass
Input
The export to CSV crashes every time. Also, you charged my card twice this month.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
The export to CSV crashes every time. Also, you charged my card twice this month.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
The export to CSV crashes every time. Also, you charged my card twice this month.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: plain

Trial 1 pass
Input
I was charged twice for my March subscription. Please refund one of the charges.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
I was charged twice for my March subscription. Please refund one of the charges.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
I was charged twice for my March subscription. Please refund one of the charges.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

feature-and-login pass (3/3)

Tags: regression

Trial 1 pass
Input
Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

login-after-update pass (3/3)

Tags: regression

Trial 1 pass
Input
Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

paid-still-free pass (3/3)

Tags: regression

Trial 1 pass
Input
You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

praise pass (3/3)

Tags: plain

Trial 1 pass
Input
Thanks, the new search is great!
Output
{ "category": "other" }
Scores
  • category@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Thanks, the new search is great!
Output
{ "category": "other" }
Scores
  • category@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Thanks, the new search is great!
Output
{ "category": "other" }
Scores
  • category@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

share-error pass (3/3)

Tags: plain

Trial 1 pass
Input
When I click Share, the page shows an error 500 and no link appears.
Output
{ "category": "bug" }
Scores
  • category@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
When I click Share, the page shows an error 500 and no link appears.
Output
{ "category": "bug" }
Scores
  • category@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
When I click Share, the page shows an error 500 and no link appears.
Output
{ "category": "bug" }
Scores
  • category@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

← Back to the report

ticket-category

sha256:e45aace9b6d9… · production

Started 2026-11-02 05:03 UTC · run e81bb33f-e10b-45ed-b49e-3da2bb2d39c5

Source JSONL: examples/config-from-db/history/ticket-category.e81bb33f-e10b-45ed-b49e-3da2bb2d39c5.jsonl

7 pass

Tokens in / out / cached / reasoning: unknown / unknown / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=z-ai/glm-5.3-flash configName=production

Config sha256:e45aace9b6d9…
model
z-ai/glm-5.3-flash
effort
medium
systemPrompt
You triage customer support tickets for a file-sharing app. Reply with the category of the ticket, one of: - billing: a charge, a refund, an invoice, a price or a payment method. - account: login, password, profile, team members or the plan. - bug: an error, a crash or a wrong result in the app. - o…Show all 512 charactersShow less
You triage customer support tickets for a file-sharing app. Reply with the category of the ticket, one of: - billing: a charge, a refund, an invoice, a price or a payment method. - account: login, password, profile, team members or the plan. - bug: an error, a crash or a wrong result in the app. - other: praise, a feature request, a question about how to use the app, or anything else. When a ticket states two problems, or one problem fits two categories, use the first of these: billing, account, bug, other.
firstPrompt
Classify this support ticket: {{ticket}}
responseSchema
{ "type": "object", "properties": { "category": { "type": "string", "enum": [ "billing", "account", "bug", "other" ] } }, "required": [ "category" ], "additionalProperties": false }

Cases

crash-and-charge pass (3/3)

Tags: regression

Trial 1 pass
Input
The export to CSV crashes every time. Also, you charged my card twice this month.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
The export to CSV crashes every time. Also, you charged my card twice this month.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
The export to CSV crashes every time. Also, you charged my card twice this month.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: plain

Trial 1 pass
Input
I was charged twice for my March subscription. Please refund one of the charges.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
I was charged twice for my March subscription. Please refund one of the charges.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
I was charged twice for my March subscription. Please refund one of the charges.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

feature-and-login pass (2/3)

Tags: regression

Trial 1 fail
Input
Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output
{ "category": "other" }
Scores
  • category@1 fail 0.00
    expected "account", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

login-after-update pass (3/3)

Tags: regression

Trial 1 pass
Input
Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output
{ "category": "account" }
Scores
  • category@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

paid-still-free pass (3/3)

Tags: regression

Trial 1 pass
Input
You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output
{ "category": "billing" }
Scores
  • category@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

praise pass (3/3)

Tags: plain

Trial 1 pass
Input
Thanks, the new search is great!
Output
{ "category": "other" }
Scores
  • category@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Thanks, the new search is great!
Output
{ "category": "other" }
Scores
  • category@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Thanks, the new search is great!
Output
{ "category": "other" }
Scores
  • category@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

share-error pass (3/3)

Tags: plain

Trial 1 pass
Input
When I click Share, the page shows an error 500 and no link appears.
Output
{ "category": "bug" }
Scores
  • category@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
When I click Share, the page shows an error 500 and no link appears.
Output
{ "category": "bug" }
Scores
  • category@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
When I click Share, the page shows an error 500 and no link appears.
Output
{ "category": "bug" }
Scores
  • category@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded