← trialkit docs: History and reports

trialkit report

1 eval, 18 runs, from 2026-10-31 14:36 to 2026-10-31 14:42 UTC.

Series by model · prompt.

support-triage

Dataset: support-tickets-v1 dd2dcb11 (16 cases).

Trial pass rate over time

100%90%80%70%60% trial pass rate (the axis starts at 60%) 14:36:1614:37:5014:39:2414:40:5714:42:31 openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb 2026-10-31 14:37 UTC, run 75a75702-874d-41a0-965f-ea935521e996 trial pass rate 100.0% (92.6–100.0%), 48/48 trialsopenrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb 2026-10-31 14:39 UTC, run 9b0035e6-4312-42e2-8b11-4fc0cf4caa01 trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6 2026-10-31 14:40 UTC, run d03d1344-15f2-4477-a509-22ee13950699 trial pass rate 100.0% (92.6–100.0%), 48/48 trialsopenrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6 2026-10-31 14:41 UTC, run 03378131-9f85-483d-9096-fb35368d1e08 trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1 2026-10-31 14:41 UTC, run b108cdab-a644-4e7b-a8a6-a7428b36490d trial pass rate 100.0% (92.6–100.0%), 48/48 trialsopenrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1 2026-10-31 14:42 UTC, run 581f294d-9557-4ab6-a412-250db41fab6d trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb 2026-10-31 14:38 UTC, run e00d49cb-a385-4d81-b929-51c9ceb67f69 trial pass rate 100.0% (92.6–100.0%), 48/48 trialsopenrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb 2026-10-31 14:39 UTC, run 533881ac-803e-41c8-8075-e4569cc2258a trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6 2026-10-31 14:40 UTC, run 6686e7b7-8372-4b3b-b42f-29db62cd13a9 trial pass rate 100.0% (92.6–100.0%), 48/48 trialsopenrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6 2026-10-31 14:41 UTC, run 0cc6b085-9194-4d9f-893d-aae81e227695 trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1 2026-10-31 14:41 UTC, run 17fa6703-f6c5-486c-8355-b41482a566d3 trial pass rate 100.0% (92.6–100.0%), 48/48 trialsopenrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1 2026-10-31 14:42 UTC, run 49b37ff7-f799-4c85-9e3d-f2aeccc2f4b3 trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:z-ai/glm-5.3-flash · triage@1+b590cb 2026-10-31 14:36 UTC, run 19c1db5c-c3a9-4dd7-a0bc-54a027ee2ef9 trial pass rate 87.5% (75.3–94.1%), 42/48 trialsopenrouter:z-ai/glm-5.3-flash · triage@1+b590cb 2026-10-31 14:39 UTC, run 04afa8f6-4ef2-46fc-85bd-aef41ae5358c trial pass rate 87.5% (75.3–94.1%), 42/48 trials openrouter:z-ai/glm-5.3-flash · triage@2+1721e6 2026-10-31 14:40 UTC, run 95e215e0-380f-4982-9c55-4475c1229d2b trial pass rate 93.8% (83.2–97.9%), 45/48 trialsopenrouter:z-ai/glm-5.3-flash · triage@2+1721e6 2026-10-31 14:41 UTC, run 36fd9f0d-411c-4a0d-848d-7f5d4a5afd43 trial pass rate 91.7% (80.4–96.7%), 44/48 trials openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1 2026-10-31 14:41 UTC, run f1daf85e-fe94-45c1-bb51-bbbc600943ff trial pass rate 100.0% (92.6–100.0%), 48/48 trialsopenrouter:z-ai/glm-5.3-flash · triage@3+6de1a1 2026-10-31 14:42 UTC, run 029fb86b-4958-4fc2-b187-c6406017f867 trial pass rate 97.9% (89.1–99.6%), 47/48 trials
  • openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb (2 runs)
  • openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6 (2 runs)
  • openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1 (2 runs)
  • openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb (2 runs)
  • openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6 (2 runs)
  • openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1 (2 runs)
  • openrouter:z-ai/glm-5.3-flash · triage@1+b590cb (2 runs)
  • openrouter:z-ai/glm-5.3-flash · triage@2+1721e6 (2 runs)
  • openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1 (2 runs)
X: start time of the run (UTC). Y: passed / (passed + failed) trials of the run. Whisker: 95% Wilson interval. Hollow diamond, dashed line and a break in the line: the dataset or the trial policy differs from the previous run of the series, so the rates do not compare.

Latest completed run per series

Select the start time of a run to see its cases and trials.

SeriesRunsCasesCase pass rate (95% CI)Trial pass rate (95% CI)TrendError trialsMean trialLatest run (UTC)
openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb216 pass100.0% 16/16
80.6–100.0%
100.0% 48/48
92.6–100.0%
██01.3 s2026-10-31 14:39
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6216 pass100.0% 16/16
80.6–100.0%
100.0% 48/48
92.6–100.0%
██01.4 s2026-10-31 14:41
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1216 pass100.0% 16/16
80.6–100.0%
100.0% 48/48
92.6–100.0%
██01.4 s2026-10-31 14:42
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb216 pass100.0% 16/16
80.6–100.0%
100.0% 48/48
92.6–100.0%
██01.2 s2026-10-31 14:39
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6216 pass100.0% 16/16
80.6–100.0%
100.0% 48/48
92.6–100.0%
██01.2 s2026-10-31 14:41
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1216 pass100.0% 16/16
80.6–100.0%
100.0% 48/48
92.6–100.0%
██0980 ms2026-10-31 14:42
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb214 pass, 2 fail87.5% 14/16
64.0–96.5%
87.5% 42/48
75.3–94.1%
▇▇01.7 s2026-10-31 14:39
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6215 pass, 1 fail93.8% 15/16
71.7–98.9%
91.7% 44/48
80.4–96.7%
█▇01.9 s2026-10-31 14:41
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1216 pass100.0% 16/16
80.6–100.0%
97.9% 47/48
89.1–99.6%
██01.8 s2026-10-31 14:42

Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.

All 18 runs in the window
SeriesStarted (UTC)Trial pass rate (95% CI)Case pass rate (95% CI)Error trialsCommitNotes
openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb 2026-10-31 14:37 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
2026-10-31 14:39 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6 2026-10-31 14:40 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
2026-10-31 14:41 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1 2026-10-31 14:41 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
2026-10-31 14:42 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb 2026-10-31 14:38 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
2026-10-31 14:39 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6 2026-10-31 14:40 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
2026-10-31 14:41 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1 2026-10-31 14:41 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
2026-10-31 14:42 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb 2026-10-31 14:36 87.5% 42/48
75.3–94.1%
87.5% 14/16
64.0–96.5%
0 –
2026-10-31 14:39 87.5% 42/48
75.3–94.1%
87.5% 14/16
64.0–96.5%
0 –
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6 2026-10-31 14:40 93.8% 45/48
83.2–97.9%
93.8% 15/16
71.7–98.9%
0 –
2026-10-31 14:41 91.7% 44/48
80.4–96.7%
93.8% 15/16
71.7–98.9%
0 –
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1 2026-10-31 14:41 100.0% 48/48
92.6–100.0%
100.0% 16/16
80.6–100.0%
0 –
2026-10-31 14:42 97.9% 47/48
89.1–99.6%
100.0% 16/16
80.6–100.0%
0 –

Usage and metric means

SeriesTokens inTokens out
openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb4839010667
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e65127012268
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a15443814237
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb1836423641
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e61792222438
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a11985425991
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb1408218683
openrouter:z-ai/glm-5.3-flash · triage@2+1721e61652426971
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a11839025833

Case consistency

Per case, the trial pass rate over the completed runs in the window. At 0% and 100%, the task gives the same result each time; the bins between hold the cases where it is not consistent.

openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb
16 cases, 2 runs
0%: 0 of 16 cases (0.0%)(0, 25%]: 0 of 16 cases (0.0%)(25%, 50%]: 0 of 16 cases (0.0%)(50%, 75%]: 0 of 16 cases (0.0%)(75%, 100%): 0 of 16 cases (0.0%)100%: 16 of 16 cases (100.0%)16 0%≤25%≤50%≤75%<100%100%
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6
16 cases, 2 runs
0%: 0 of 16 cases (0.0%)(0, 25%]: 0 of 16 cases (0.0%)(25%, 50%]: 0 of 16 cases (0.0%)(50%, 75%]: 0 of 16 cases (0.0%)(75%, 100%): 0 of 16 cases (0.0%)100%: 16 of 16 cases (100.0%)16 0%≤25%≤50%≤75%<100%100%
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1
16 cases, 2 runs
0%: 0 of 16 cases (0.0%)(0, 25%]: 0 of 16 cases (0.0%)(25%, 50%]: 0 of 16 cases (0.0%)(50%, 75%]: 0 of 16 cases (0.0%)(75%, 100%): 0 of 16 cases (0.0%)100%: 16 of 16 cases (100.0%)16 0%≤25%≤50%≤75%<100%100%
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb
16 cases, 2 runs
0%: 0 of 16 cases (0.0%)(0, 25%]: 0 of 16 cases (0.0%)(25%, 50%]: 0 of 16 cases (0.0%)(50%, 75%]: 0 of 16 cases (0.0%)(75%, 100%): 0 of 16 cases (0.0%)100%: 16 of 16 cases (100.0%)16 0%≤25%≤50%≤75%<100%100%
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6
16 cases, 2 runs
0%: 0 of 16 cases (0.0%)(0, 25%]: 0 of 16 cases (0.0%)(25%, 50%]: 0 of 16 cases (0.0%)(50%, 75%]: 0 of 16 cases (0.0%)(75%, 100%): 0 of 16 cases (0.0%)100%: 16 of 16 cases (100.0%)16 0%≤25%≤50%≤75%<100%100%
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1
16 cases, 2 runs
0%: 0 of 16 cases (0.0%)(0, 25%]: 0 of 16 cases (0.0%)(25%, 50%]: 0 of 16 cases (0.0%)(50%, 75%]: 0 of 16 cases (0.0%)(75%, 100%): 0 of 16 cases (0.0%)100%: 16 of 16 cases (100.0%)16 0%≤25%≤50%≤75%<100%100%
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb
16 cases, 2 runs
0%: 2 of 16 cases (12.5%)2(0, 25%]: 0 of 16 cases (0.0%)(25%, 50%]: 0 of 16 cases (0.0%)(50%, 75%]: 0 of 16 cases (0.0%)(75%, 100%): 0 of 16 cases (0.0%)100%: 14 of 16 cases (87.5%)14 0%≤25%≤50%≤75%<100%100%
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6
16 cases, 2 runs
0%: 1 of 16 cases (6.3%)1(0, 25%]: 0 of 16 cases (0.0%)(25%, 50%]: 0 of 16 cases (0.0%)(50%, 75%]: 0 of 16 cases (0.0%)(75%, 100%): 1 of 16 cases (6.3%)1100%: 14 of 16 cases (87.5%)14 0%≤25%≤50%≤75%<100%100%
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1
16 cases, 2 runs
0%: 0 of 16 cases (0.0%)(0, 25%]: 0 of 16 cases (0.0%)(25%, 50%]: 0 of 16 cases (0.0%)(50%, 75%]: 0 of 16 cases (0.0%)(75%, 100%): 1 of 16 cases (6.3%)1100%: 15 of 16 cases (93.8%)15 0%≤25%≤50%≤75%<100%100%

Score distributions

Per scorer, the score values of all trials of the completed runs in the window, in 10 bins from 0 to 1.

category@1

openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 0 of 96 scores (0.0%)[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 96 of 96 scores (100.0%)96 0.1.2.3.4.5.6.7.8.91
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 0 of 96 scores (0.0%)[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 96 of 96 scores (100.0%)96 0.1.2.3.4.5.6.7.8.91
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 0 of 96 scores (0.0%)[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 96 of 96 scores (100.0%)96 0.1.2.3.4.5.6.7.8.91
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 0 of 96 scores (0.0%)[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 96 of 96 scores (100.0%)96 0.1.2.3.4.5.6.7.8.91
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 0 of 96 scores (0.0%)[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 96 of 96 scores (100.0%)96 0.1.2.3.4.5.6.7.8.91
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 0 of 96 scores (0.0%)[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 96 of 96 scores (100.0%)96 0.1.2.3.4.5.6.7.8.91
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb
96 scores
[0.0, 0.1): 12 of 96 scores (12.5%)12[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 0 of 96 scores (0.0%)[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 84 of 96 scores (87.5%)84 0.1.2.3.4.5.6.7.8.91
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6
96 scores
[0.0, 0.1): 7 of 96 scores (7.3%)7[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 0 of 96 scores (0.0%)[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 89 of 96 scores (92.7%)89 0.1.2.3.4.5.6.7.8.91
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1
96 scores
[0.0, 0.1): 1 of 96 scores (1.0%)1[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 0 of 96 scores (0.0%)[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 95 of 96 scores (99.0%)95 0.1.2.3.4.5.6.7.8.91

priority@1

openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 13 of 96 scores (13.5%)13[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 83 of 96 scores (86.5%)83 0.1.2.3.4.5.6.7.8.91
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 12 of 96 scores (12.5%)12[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 84 of 96 scores (87.5%)84 0.1.2.3.4.5.6.7.8.91
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 11 of 96 scores (11.5%)11[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 85 of 96 scores (88.5%)85 0.1.2.3.4.5.6.7.8.91
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 5 of 96 scores (5.2%)5[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 91 of 96 scores (94.8%)91 0.1.2.3.4.5.6.7.8.91
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 6 of 96 scores (6.3%)6[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 90 of 96 scores (93.8%)90 0.1.2.3.4.5.6.7.8.91
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 7 of 96 scores (7.3%)7[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 89 of 96 scores (92.7%)89 0.1.2.3.4.5.6.7.8.91
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 1 of 96 scores (1.0%)1[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 95 of 96 scores (99.0%)95 0.1.2.3.4.5.6.7.8.91
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 8 of 96 scores (8.3%)8[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 88 of 96 scores (91.7%)88 0.1.2.3.4.5.6.7.8.91
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%)[0.1, 0.2): 0 of 96 scores (0.0%)[0.2, 0.3): 0 of 96 scores (0.0%)[0.3, 0.4): 0 of 96 scores (0.0%)[0.4, 0.5): 0 of 96 scores (0.0%)[0.5, 0.6): 0 of 96 scores (0.0%)[0.6, 0.7): 3 of 96 scores (3.1%)3[0.7, 0.8): 0 of 96 scores (0.0%)[0.8, 0.9): 0 of 96 scores (0.0%)[0.9, 1.0]: 93 of 96 scores (96.9%)93 0.1.2.3.4.5.6.7.8.91

Flips

No flips since the previous completed run of each series.

← Back to the report

support-triage

openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb

Started 2026-10-31 14:37 UTC · run 75a75702-874d-41a0-965f-ea935521e996

Source JSONL: examples/support-triage/history-sample/support-triage.75a75702-874d-41a0-965f-ea935521e996.jsonl

16 pass

Tokens in / out / cached / reasoning: 24195 / 5778 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:anthropic/claude-haiku-5.5 prompt=triage@1+b590cb env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb

Started 2026-10-31 14:39 UTC · run 9b0035e6-4312-42e2-8b11-4fc0cf4caa01

Source JSONL: examples/support-triage/history-sample/support-triage.9b0035e6-4312-42e2-8b11-4fc0cf4caa01.jsonl

16 pass

Tokens in / out / cached / reasoning: 24195 / 4889 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:anthropic/claude-haiku-5.5 prompt=triage@1+b590cb env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6

Started 2026-10-31 14:40 UTC · run d03d1344-15f2-4477-a509-22ee13950699

Source JSONL: examples/support-triage/history-sample/support-triage.d03d1344-15f2-4477-a509-22ee13950699.jsonl

16 pass

Tokens in / out / cached / reasoning: 25635 / 6197 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:anthropic/claude-haiku-5.5 prompt=triage@2+1721e6 env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6

Started 2026-10-31 14:41 UTC · run 03378131-9f85-483d-9096-fb35368d1e08

Source JSONL: examples/support-triage/history-sample/support-triage.03378131-9f85-483d-9096-fb35368d1e08.jsonl

16 pass

Tokens in / out / cached / reasoning: 25635 / 6071 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:anthropic/claude-haiku-5.5 prompt=triage@2+1721e6 env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1

Started 2026-10-31 14:41 UTC · run b108cdab-a644-4e7b-a8a6-a7428b36490d

Source JSONL: examples/support-triage/history-sample/support-triage.b108cdab-a644-4e7b-a8a6-a7428b36490d.jsonl

16 pass

Tokens in / out / cached / reasoning: 27219 / 7350 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:anthropic/claude-haiku-5.5 prompt=triage@3+6de1a1 env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1

Started 2026-10-31 14:42 UTC · run 581f294d-9557-4ab6-a412-250db41fab6d

Source JSONL: examples/support-triage/history-sample/support-triage.581f294d-9557-4ab6-a412-250db41fab6d.jsonl

16 pass

Tokens in / out / cached / reasoning: 27219 / 6887 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:anthropic/claude-haiku-5.5 prompt=triage@3+6de1a1 env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb

Started 2026-10-31 14:38 UTC · run e00d49cb-a385-4d81-b929-51c9ceb67f69

Source JSONL: examples/support-triage/history-sample/support-triage.e00d49cb-a385-4d81-b929-51c9ceb67f69.jsonl

16 pass

Tokens in / out / cached / reasoning: 9184 / 11375 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:deepseek/deepseek-v4.1-flash prompt=triage@1+b590cb env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb

Started 2026-10-31 14:39 UTC · run 533881ac-803e-41c8-8075-e4569cc2258a

Source JSONL: examples/support-triage/history-sample/support-triage.533881ac-803e-41c8-8075-e4569cc2258a.jsonl

16 pass

Tokens in / out / cached / reasoning: 9180 / 12266 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:deepseek/deepseek-v4.1-flash prompt=triage@1+b590cb env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6

Started 2026-10-31 14:40 UTC · run 6686e7b7-8372-4b3b-b42f-29db62cd13a9

Source JSONL: examples/support-triage/history-sample/support-triage.6686e7b7-8372-4b3b-b42f-29db62cd13a9.jsonl

16 pass

Tokens in / out / cached / reasoning: 8961 / 11164 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:deepseek/deepseek-v4.1-flash prompt=triage@2+1721e6 env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6

Started 2026-10-31 14:41 UTC · run 0cc6b085-9194-4d9f-893d-aae81e227695

Source JSONL: examples/support-triage/history-sample/support-triage.0cc6b085-9194-4d9f-893d-aae81e227695.jsonl

16 pass

Tokens in / out / cached / reasoning: 8961 / 11274 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:deepseek/deepseek-v4.1-flash prompt=triage@2+1721e6 env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1

Started 2026-10-31 14:41 UTC · run 17fa6703-f6c5-486c-8355-b41482a566d3

Source JSONL: examples/support-triage/history-sample/support-triage.17fa6703-f6c5-486c-8355-b41482a566d3.jsonl

16 pass

Tokens in / out / cached / reasoning: 9792 / 14538 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:deepseek/deepseek-v4.1-flash prompt=triage@3+6de1a1 env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1

Started 2026-10-31 14:42 UTC · run 49b37ff7-f799-4c85-9e3d-f2aeccc2f4b3

Source JSONL: examples/support-triage/history-sample/support-triage.49b37ff7-f799-4c85-9e3d-f2aeccc2f4b3.jsonl

16 pass

Tokens in / out / cached / reasoning: 10062 / 11453 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:deepseek/deepseek-v4.1-flash prompt=triage@3+6de1a1 env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:z-ai/glm-5.3-flash · triage@1+b590cb

Started 2026-10-31 14:36 UTC · run 19c1db5c-c3a9-4dd7-a0bc-54a027ee2ef9

Source JSONL: examples/support-triage/history-sample/support-triage.19c1db5c-c3a9-4dd7-a0bc-54a027ee2ef9.jsonl

14 pass, 2 fail

Tokens in / out / cached / reasoning: 7053 / 9394 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:z-ai/glm-5.3-flash prompt=triage@1+b590cb env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent fail (0/3)

Tags: bug ambiguous

0 of 3 trials passed, needs 2; category: expected bug, got billing
Trial 1 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing fail (0/3)

Tags: bug ambiguous

0 of 3 trials passed, needs 2; category: expected bug, got billing
Trial 1 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:z-ai/glm-5.3-flash · triage@1+b590cb

Started 2026-10-31 14:39 UTC · run 04afa8f6-4ef2-46fc-85bd-aef41ae5358c

Source JSONL: examples/support-triage/history-sample/support-triage.04afa8f6-4ef2-46fc-85bd-aef41ae5358c.jsonl

14 pass, 2 fail

Tokens in / out / cached / reasoning: 7029 / 9289 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:z-ai/glm-5.3-flash prompt=triage@1+b590cb env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent fail (0/3)

Tags: bug ambiguous

0 of 3 trials passed, needs 2; category: expected bug, got billing
Trial 1 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing fail (0/3)

Tags: bug ambiguous

0 of 3 trials passed, needs 2; category: expected bug, got billing
Trial 1 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:z-ai/glm-5.3-flash · triage@2+1721e6

Started 2026-10-31 14:40 UTC · run 95e215e0-380f-4982-9c55-4475c1229d2b

Source JSONL: examples/support-triage/history-sample/support-triage.95e215e0-380f-4982-9c55-4475c1229d2b.jsonl

15 pass, 1 fail

Tokens in / out / cached / reasoning: 8253 / 13253 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:z-ai/glm-5.3-flash prompt=triage@2+1721e6 env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent fail (0/3)

Tags: bug ambiguous

0 of 3 trials passed, needs 2; category: expected bug, got billing
Trial 1 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:z-ai/glm-5.3-flash · triage@2+1721e6

Started 2026-10-31 14:41 UTC · run 36fd9f0d-411c-4a0d-848d-7f5d4a5afd43

Source JSONL: examples/support-triage/history-sample/support-triage.36fd9f0d-411c-4a0d-848d-7f5d4a5afd43.jsonl

15 pass, 1 fail

Tokens in / out / cached / reasoning: 8271 / 13718 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:z-ai/glm-5.3-flash prompt=triage@2+1721e6 env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 0.67
    expected P4, got P3
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent fail (0/3)

Tags: bug ambiguous

0 of 3 trials passed, needs 2; category: expected bug, got billing
Trial 1 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (2/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1

Started 2026-10-31 14:41 UTC · run f1daf85e-fe94-45c1-bb51-bbbc600943ff

Source JSONL: examples/support-triage/history-sample/support-triage.f1daf85e-fe94-45c1-bb51-bbbc600943ff.jsonl

16 pass

Tokens in / out / cached / reasoning: 9189 / 13074 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:z-ai/glm-5.3-flash prompt=triage@3+6de1a1 env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 0.67
    expected P1, got P2
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

← Back to the report

support-triage

openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1

Started 2026-10-31 14:42 UTC · run 029fb86b-4958-4fc2-b187-c6406017f867

Source JSONL: examples/support-triage/history-sample/support-triage.029fb86b-4958-4fc2-b187-c6406017f867.jsonl

16 pass

Tokens in / out / cached / reasoning: 9201 / 12759 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=openrouter:z-ai/glm-5.3-flash prompt=triage@3+6de1a1 env=local

Cases

api-webhooks pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

card-declined-upgrade pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

change-email pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

charged-after-cancel pass (3/3)

Tags: billing ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: feature_request

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected feature_request, got feature_request
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

delete-account pass (3/3)

Tags: account

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

double-charge pass (3/3)

Tags: billing regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

locked-out-2fa pass (3/3)

Tags: account regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

outage pass (3/3)

Tags: bug regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

partnership pass (3/3)

Tags: other

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected other, got other
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

pdf-export-wrong-total pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P2, got P2
Trace

Not recorded

Logs

Not recorded

recurring-not-sent pass (2/3)

Tags: bug ambiguous

Trial 1 fail
Input and output
Not recorded.
Scores
  • category@1 fail 0.00
    expected bug, got billing
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

refund-policy pass (3/3)

Tags: billing

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected billing, got billing
  • priority@1 pass 1.00
    expected P4, got P4
Trace

Not recorded

Logs

Not recorded

slow-search pass (3/3)

Tags: bug

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

suspicious-login pass (3/3)

Tags: account security regression

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected account, got account
  • priority@1 pass 1.00
    expected P1, got P1
Trace

Not recorded

Logs

Not recorded

vat-missing pass (3/3)

Tags: bug ambiguous

Trial 1 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input and output
Not recorded.
Scores
  • category@1 pass 1.00
    expected bug, got bug
  • priority@1 pass 1.00
    expected P3, got P3
Trace

Not recorded

Logs

Not recorded