trialkit report
1 eval, 4 runs, from 2026-11-02 05:03 to 2026-11-02 05:03 UTC.
Series by config · configName.
ticket-category
Dataset: ticket-category 663d8221 (7 cases).
Trial pass rate over time
100% 80% 60% 40%
trial pass rate (the axis starts at 40%)
05:03:25 05:03:28 05:03:31 05:03:34 05:03:37
sha256:3224dc6fbb33… · candidate
2026-11-02 05:03 UTC, run 81fb8293-356d-4f61-bf80-af3a09fb5632
trial pass rate 71.4% (50.0–86.2%), 15/21 trials sha256:3224dc6fbb33… · candidate
2026-11-02 05:03 UTC, run f6552169-59ac-4fa4-ba1b-5d2e6f4b6c04
trial pass rate 71.4% (50.0–86.2%), 15/21 trials
sha256:e45aace9b6d9… · production
2026-11-02 05:03 UTC, run 05b9d3af-aafc-4e90-86c3-c6db4465a9ea
trial pass rate 100.0% (84.5–100.0%), 21/21 trials sha256:e45aace9b6d9… · production
2026-11-02 05:03 UTC, run e81bb33f-e10b-45ed-b49e-3da2bb2d39c5
trial pass rate 95.2% (77.3–99.2%), 20/21 trials
sha256:3224dc6fbb33… · candidate (2 runs) sha256:e45aace9b6d9… · production (2 runs)
X: start time of the run (UTC). Y: passed / (passed + failed) trials of the run. Whisker: 95% Wilson interval. Hollow diamond, dashed line and a break in the line: the dataset or the trial policy differs from the previous run of the series, so the rates do not compare. Dotted line with a value: the run where prompt changed to that value.
Latest completed run per series
Select the start time of a run to see its cases and trials.
Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.
All 4 runs in the window
Case consistency
Per case, the trial pass rate over the completed runs in the window. At 0% and 100%, the task gives the same result each time; the bins between hold the cases where it is not consistent.
sha256:3224dc6fbb33… · candidate
7 cases, 2 runs
0%: 2 of 7 cases (28.6%) 2 (0, 25%]: 0 of 7 cases (0.0%) (25%, 50%]: 0 of 7 cases (0.0%) (50%, 75%]: 0 of 7 cases (0.0%) (75%, 100%): 0 of 7 cases (0.0%) 100%: 5 of 7 cases (71.4%) 5
0% ≤25% ≤50% ≤75% <100% 100%
sha256:e45aace9b6d9… · production
7 cases, 2 runs
0%: 0 of 7 cases (0.0%) (0, 25%]: 0 of 7 cases (0.0%) (25%, 50%]: 0 of 7 cases (0.0%) (50%, 75%]: 0 of 7 cases (0.0%) (75%, 100%): 1 of 7 cases (14.3%) 1 100%: 6 of 7 cases (85.7%) 6
0% ≤25% ≤50% ≤75% <100% 100%
Score distributions
Per scorer, the score values of all trials of the completed runs in the window, in 10 bins from 0 to 1.
category@1
sha256:3224dc6fbb33… · candidate
42 scores
[0.0, 0.1): 12 of 42 scores (28.6%) 12 [0.1, 0.2): 0 of 42 scores (0.0%) [0.2, 0.3): 0 of 42 scores (0.0%) [0.3, 0.4): 0 of 42 scores (0.0%) [0.4, 0.5): 0 of 42 scores (0.0%) [0.5, 0.6): 0 of 42 scores (0.0%) [0.6, 0.7): 0 of 42 scores (0.0%) [0.7, 0.8): 0 of 42 scores (0.0%) [0.8, 0.9): 0 of 42 scores (0.0%) [0.9, 1.0]: 30 of 42 scores (71.4%) 30
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
sha256:e45aace9b6d9… · production
42 scores
[0.0, 0.1): 1 of 42 scores (2.4%) 1 [0.1, 0.2): 0 of 42 scores (0.0%) [0.2, 0.3): 0 of 42 scores (0.0%) [0.3, 0.4): 0 of 42 scores (0.0%) [0.4, 0.5): 0 of 42 scores (0.0%) [0.5, 0.6): 0 of 42 scores (0.0%) [0.6, 0.7): 0 of 42 scores (0.0%) [0.7, 0.8): 0 of 42 scores (0.0%) [0.8, 0.9): 0 of 42 scores (0.0%) [0.9, 1.0]: 41 of 42 scores (97.6%) 41
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
Flips
No flips since the previous completed run of each series.
← Back to the report
ticket-category
sha256:3224dc6fbb33… · candidate
Started 2026-11-02 05:03 UTC · run 81fb8293-356d-4f61-bf80-af3a09fb5632
Source JSONL: examples/config-from-db/history/ticket-category.81fb8293-356d-4f61-bf80-af3a09fb5632.jsonl
5 pass , 2 fail
Tokens in / out / cached / reasoning: unknown / unknown / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=z-ai/glm-5.3-flash configName=candidate
Config sha256:3224dc6fbb33…
modelz-ai/glm-5.3-flash
effortlow
systemPrompt
You triage customer support tickets for a file-sharing app.
Reply with the category of the ticket, one of:
- billing: a charge, a refund, an invoice, a price or a payment method.
- account: login, password, profile, team members or the plan.
- bug: an error, a crash or a wrong result in the app.
- o… Show all 471 characters Show less
You triage customer support tickets for a file-sharing app.
Reply with the category of the ticket, one of:
- billing: a charge, a refund, an invoice, a price or a payment method.
- account: login, password, profile, team members or the plan.
- bug: an error, a crash or a wrong result in the app.
- other: praise, a feature request, a question about how to use the app, or anything else.
When a ticket states two problems, use the category of the first one in the ticket.
firstPromptClassify this support ticket:
{{ticket}}
responseSchema{
"type": "object",
"properties": {
"category": {
"type": "string",
"enum": [
"billing",
"account",
"bug",
"other"
]
}
},
"required": [
"category"
],
"additionalProperties": false
}
Cases
crash-and-charge fail (0/3)
Tags: regression
0 of 3 trials passed, needs 2; category: expected "billing", got "bug"
Trial 1 fail
Input The export to CSV crashes every time. Also, you charged my card twice this month.
Output {
"category": "bug"
}
Scores category@1 fail 0.00
expected "billing", got "bug"
Trace Not recorded
Logs Not recorded
Trial 2 fail
Input The export to CSV crashes every time. Also, you charged my card twice this month.
Output {
"category": "bug"
}
Scores category@1 fail 0.00
expected "billing", got "bug"
Trace Not recorded
Logs Not recorded
Trial 3 fail
Input The export to CSV crashes every time. Also, you charged my card twice this month.
Output {
"category": "bug"
}
Scores category@1 fail 0.00
expected "billing", got "bug"
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: plain
Trial 1 pass
Input I was charged twice for my March subscription. Please refund one of the charges.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input I was charged twice for my March subscription. Please refund one of the charges.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input I was charged twice for my March subscription. Please refund one of the charges.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
feature-and-login fail (0/3)
Tags: regression
0 of 3 trials passed, needs 2; category: expected "account", got "other"
Trial 1 fail
Input Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output {
"category": "other"
}
Scores category@1 fail 0.00
expected "account", got "other"
Trace Not recorded
Logs Not recorded
Trial 2 fail
Input Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output {
"category": "other"
}
Scores category@1 fail 0.00
expected "account", got "other"
Trace Not recorded
Logs Not recorded
Trial 3 fail
Input Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output {
"category": "other"
}
Scores category@1 fail 0.00
expected "account", got "other"
Trace Not recorded
Logs Not recorded
login-after-update pass (3/3)
Tags: regression
Trial 1 pass
Input Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
paid-still-free pass (3/3)
Tags: regression
Trial 1 pass
Input You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
praise pass (3/3)
Tags: plain
Trial 1 pass
Input Thanks, the new search is great!
Output {
"category": "other"
}
Scores category@1 pass 1.00
expected "other", got "other"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input Thanks, the new search is great!
Output {
"category": "other"
}
Scores category@1 pass 1.00
expected "other", got "other"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input Thanks, the new search is great!
Output {
"category": "other"
}
Scores category@1 pass 1.00
expected "other", got "other"
Trace Not recorded
Logs Not recorded
share-error pass (3/3)
Tags: plain
Trial 1 pass
Input When I click Share, the page shows an error 500 and no link appears.
Output {
"category": "bug"
}
Scores category@1 pass 1.00
expected "bug", got "bug"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input When I click Share, the page shows an error 500 and no link appears.
Output {
"category": "bug"
}
Scores category@1 pass 1.00
expected "bug", got "bug"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input When I click Share, the page shows an error 500 and no link appears.
Output {
"category": "bug"
}
Scores category@1 pass 1.00
expected "bug", got "bug"
Trace Not recorded
Logs Not recorded
← Back to the report
ticket-category
sha256:3224dc6fbb33… · candidate
Started 2026-11-02 05:03 UTC · run f6552169-59ac-4fa4-ba1b-5d2e6f4b6c04
Source JSONL: examples/config-from-db/history/ticket-category.f6552169-59ac-4fa4-ba1b-5d2e6f4b6c04.jsonl
5 pass , 2 fail
Tokens in / out / cached / reasoning: unknown / unknown / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=z-ai/glm-5.3-flash configName=candidate
Config sha256:3224dc6fbb33…
modelz-ai/glm-5.3-flash
effortlow
systemPrompt
You triage customer support tickets for a file-sharing app.
Reply with the category of the ticket, one of:
- billing: a charge, a refund, an invoice, a price or a payment method.
- account: login, password, profile, team members or the plan.
- bug: an error, a crash or a wrong result in the app.
- o… Show all 471 characters Show less
You triage customer support tickets for a file-sharing app.
Reply with the category of the ticket, one of:
- billing: a charge, a refund, an invoice, a price or a payment method.
- account: login, password, profile, team members or the plan.
- bug: an error, a crash or a wrong result in the app.
- other: praise, a feature request, a question about how to use the app, or anything else.
When a ticket states two problems, use the category of the first one in the ticket.
firstPromptClassify this support ticket:
{{ticket}}
responseSchema{
"type": "object",
"properties": {
"category": {
"type": "string",
"enum": [
"billing",
"account",
"bug",
"other"
]
}
},
"required": [
"category"
],
"additionalProperties": false
}
Cases
crash-and-charge fail (0/3)
Tags: regression
0 of 3 trials passed, needs 2; category: expected "billing", got "bug"
Trial 1 fail
Input The export to CSV crashes every time. Also, you charged my card twice this month.
Output {
"category": "bug"
}
Scores category@1 fail 0.00
expected "billing", got "bug"
Trace Not recorded
Logs Not recorded
Trial 2 fail
Input The export to CSV crashes every time. Also, you charged my card twice this month.
Output {
"category": "bug"
}
Scores category@1 fail 0.00
expected "billing", got "bug"
Trace Not recorded
Logs Not recorded
Trial 3 fail
Input The export to CSV crashes every time. Also, you charged my card twice this month.
Output {
"category": "bug"
}
Scores category@1 fail 0.00
expected "billing", got "bug"
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: plain
Trial 1 pass
Input I was charged twice for my March subscription. Please refund one of the charges.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input I was charged twice for my March subscription. Please refund one of the charges.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input I was charged twice for my March subscription. Please refund one of the charges.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
feature-and-login fail (0/3)
Tags: regression
0 of 3 trials passed, needs 2; category: expected "account", got "other"
Trial 1 fail
Input Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output {
"category": "other"
}
Scores category@1 fail 0.00
expected "account", got "other"
Trace Not recorded
Logs Not recorded
Trial 2 fail
Input Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output {
"category": "other"
}
Scores category@1 fail 0.00
expected "account", got "other"
Trace Not recorded
Logs Not recorded
Trial 3 fail
Input Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output {
"category": "other"
}
Scores category@1 fail 0.00
expected "account", got "other"
Trace Not recorded
Logs Not recorded
login-after-update pass (3/3)
Tags: regression
Trial 1 pass
Input Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
paid-still-free pass (3/3)
Tags: regression
Trial 1 pass
Input You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
praise pass (3/3)
Tags: plain
Trial 1 pass
Input Thanks, the new search is great!
Output {
"category": "other"
}
Scores category@1 pass 1.00
expected "other", got "other"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input Thanks, the new search is great!
Output {
"category": "other"
}
Scores category@1 pass 1.00
expected "other", got "other"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input Thanks, the new search is great!
Output {
"category": "other"
}
Scores category@1 pass 1.00
expected "other", got "other"
Trace Not recorded
Logs Not recorded
share-error pass (3/3)
Tags: plain
Trial 1 pass
Input When I click Share, the page shows an error 500 and no link appears.
Output {
"category": "bug"
}
Scores category@1 pass 1.00
expected "bug", got "bug"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input When I click Share, the page shows an error 500 and no link appears.
Output {
"category": "bug"
}
Scores category@1 pass 1.00
expected "bug", got "bug"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input When I click Share, the page shows an error 500 and no link appears.
Output {
"category": "bug"
}
Scores category@1 pass 1.00
expected "bug", got "bug"
Trace Not recorded
Logs Not recorded
← Back to the report
ticket-category
sha256:e45aace9b6d9… · production
Started 2026-11-02 05:03 UTC · run 05b9d3af-aafc-4e90-86c3-c6db4465a9ea
Source JSONL: examples/config-from-db/history/ticket-category.05b9d3af-aafc-4e90-86c3-c6db4465a9ea.jsonl
7 pass
Tokens in / out / cached / reasoning: unknown / unknown / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=z-ai/glm-5.3-flash configName=production
Config sha256:e45aace9b6d9…
modelz-ai/glm-5.3-flash
effortmedium
systemPrompt
You triage customer support tickets for a file-sharing app.
Reply with the category of the ticket, one of:
- billing: a charge, a refund, an invoice, a price or a payment method.
- account: login, password, profile, team members or the plan.
- bug: an error, a crash or a wrong result in the app.
- o… Show all 512 characters Show less
You triage customer support tickets for a file-sharing app.
Reply with the category of the ticket, one of:
- billing: a charge, a refund, an invoice, a price or a payment method.
- account: login, password, profile, team members or the plan.
- bug: an error, a crash or a wrong result in the app.
- other: praise, a feature request, a question about how to use the app, or anything else.
When a ticket states two problems, or one problem fits two categories, use the first of these: billing, account, bug, other.
firstPromptClassify this support ticket:
{{ticket}}
responseSchema{
"type": "object",
"properties": {
"category": {
"type": "string",
"enum": [
"billing",
"account",
"bug",
"other"
]
}
},
"required": [
"category"
],
"additionalProperties": false
}
Cases
crash-and-charge pass (3/3)
Tags: regression
Trial 1 pass
Input The export to CSV crashes every time. Also, you charged my card twice this month.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input The export to CSV crashes every time. Also, you charged my card twice this month.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input The export to CSV crashes every time. Also, you charged my card twice this month.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: plain
Trial 1 pass
Input I was charged twice for my March subscription. Please refund one of the charges.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input I was charged twice for my March subscription. Please refund one of the charges.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input I was charged twice for my March subscription. Please refund one of the charges.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
feature-and-login pass (3/3)
Tags: regression
Trial 1 pass
Input Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
login-after-update pass (3/3)
Tags: regression
Trial 1 pass
Input Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
paid-still-free pass (3/3)
Tags: regression
Trial 1 pass
Input You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
praise pass (3/3)
Tags: plain
Trial 1 pass
Input Thanks, the new search is great!
Output {
"category": "other"
}
Scores category@1 pass 1.00
expected "other", got "other"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input Thanks, the new search is great!
Output {
"category": "other"
}
Scores category@1 pass 1.00
expected "other", got "other"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input Thanks, the new search is great!
Output {
"category": "other"
}
Scores category@1 pass 1.00
expected "other", got "other"
Trace Not recorded
Logs Not recorded
share-error pass (3/3)
Tags: plain
Trial 1 pass
Input When I click Share, the page shows an error 500 and no link appears.
Output {
"category": "bug"
}
Scores category@1 pass 1.00
expected "bug", got "bug"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input When I click Share, the page shows an error 500 and no link appears.
Output {
"category": "bug"
}
Scores category@1 pass 1.00
expected "bug", got "bug"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input When I click Share, the page shows an error 500 and no link appears.
Output {
"category": "bug"
}
Scores category@1 pass 1.00
expected "bug", got "bug"
Trace Not recorded
Logs Not recorded
← Back to the report
ticket-category
sha256:e45aace9b6d9… · production
Started 2026-11-02 05:03 UTC · run e81bb33f-e10b-45ed-b49e-3da2bb2d39c5
Source JSONL: examples/config-from-db/history/ticket-category.e81bb33f-e10b-45ed-b49e-3da2bb2d39c5.jsonl
7 pass
Tokens in / out / cached / reasoning: unknown / unknown / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=z-ai/glm-5.3-flash configName=production
Config sha256:e45aace9b6d9…
modelz-ai/glm-5.3-flash
effortmedium
systemPrompt
You triage customer support tickets for a file-sharing app.
Reply with the category of the ticket, one of:
- billing: a charge, a refund, an invoice, a price or a payment method.
- account: login, password, profile, team members or the plan.
- bug: an error, a crash or a wrong result in the app.
- o… Show all 512 characters Show less
You triage customer support tickets for a file-sharing app.
Reply with the category of the ticket, one of:
- billing: a charge, a refund, an invoice, a price or a payment method.
- account: login, password, profile, team members or the plan.
- bug: an error, a crash or a wrong result in the app.
- other: praise, a feature request, a question about how to use the app, or anything else.
When a ticket states two problems, or one problem fits two categories, use the first of these: billing, account, bug, other.
firstPromptClassify this support ticket:
{{ticket}}
responseSchema{
"type": "object",
"properties": {
"category": {
"type": "string",
"enum": [
"billing",
"account",
"bug",
"other"
]
}
},
"required": [
"category"
],
"additionalProperties": false
}
Cases
crash-and-charge pass (3/3)
Tags: regression
Trial 1 pass
Input The export to CSV crashes every time. Also, you charged my card twice this month.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input The export to CSV crashes every time. Also, you charged my card twice this month.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input The export to CSV crashes every time. Also, you charged my card twice this month.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: plain
Trial 1 pass
Input I was charged twice for my March subscription. Please refund one of the charges.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input I was charged twice for my March subscription. Please refund one of the charges.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input I was charged twice for my March subscription. Please refund one of the charges.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
feature-and-login pass (2/3)
Tags: regression
Trial 1 fail
Input Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output {
"category": "other"
}
Scores category@1 fail 0.00
expected "account", got "other"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input Hi! A dark mode would be great for working at night. Also, since this morning the app does not let me log in: it shows 'session expired' every time.
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
login-after-update pass (3/3)
Tags: regression
Trial 1 pass
Input Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input Since your last update I cannot log in. It says my password is wrong, but it is not. Is this a bug?
Output {
"category": "account"
}
Scores category@1 pass 1.00
expected "account", got "account"
Trace Not recorded
Logs Not recorded
paid-still-free pass (3/3)
Tags: regression
Trial 1 pass
Input You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input You charged my card $29 for Pro an hour ago, but the app still shows the Free plan.
Output {
"category": "billing"
}
Scores category@1 pass 1.00
expected "billing", got "billing"
Trace Not recorded
Logs Not recorded
praise pass (3/3)
Tags: plain
Trial 1 pass
Input Thanks, the new search is great!
Output {
"category": "other"
}
Scores category@1 pass 1.00
expected "other", got "other"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input Thanks, the new search is great!
Output {
"category": "other"
}
Scores category@1 pass 1.00
expected "other", got "other"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input Thanks, the new search is great!
Output {
"category": "other"
}
Scores category@1 pass 1.00
expected "other", got "other"
Trace Not recorded
Logs Not recorded
share-error pass (3/3)
Tags: plain
Trial 1 pass
Input When I click Share, the page shows an error 500 and no link appears.
Output {
"category": "bug"
}
Scores category@1 pass 1.00
expected "bug", got "bug"
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input When I click Share, the page shows an error 500 and no link appears.
Output {
"category": "bug"
}
Scores category@1 pass 1.00
expected "bug", got "bug"
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input When I click Share, the page shows an error 500 and no link appears.
Output {
"category": "bug"
}
Scores category@1 pass 1.00
expected "bug", got "bug"
Trace Not recorded
Logs Not recorded