trialkit report
1 eval, 18 runs, from 2026-10-31 14:36 to 2026-10-31 14:42 UTC.
Series by model · prompt.
support-triage
Dataset: support-tickets-v1 dd2dcb11 (16 cases).
Trial pass rate over time
100% 90% 80% 70% 60%
trial pass rate (the axis starts at 60%)
14:36:16 14:37:50 14:39:24 14:40:57 14:42:31
openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb
2026-10-31 14:37 UTC, run 75a75702-874d-41a0-965f-ea935521e996
trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb
2026-10-31 14:39 UTC, run 9b0035e6-4312-42e2-8b11-4fc0cf4caa01
trial pass rate 100.0% (92.6–100.0%), 48/48 trials
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6
2026-10-31 14:40 UTC, run d03d1344-15f2-4477-a509-22ee13950699
trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6
2026-10-31 14:41 UTC, run 03378131-9f85-483d-9096-fb35368d1e08
trial pass rate 100.0% (92.6–100.0%), 48/48 trials
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1
2026-10-31 14:41 UTC, run b108cdab-a644-4e7b-a8a6-a7428b36490d
trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1
2026-10-31 14:42 UTC, run 581f294d-9557-4ab6-a412-250db41fab6d
trial pass rate 100.0% (92.6–100.0%), 48/48 trials
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb
2026-10-31 14:38 UTC, run e00d49cb-a385-4d81-b929-51c9ceb67f69
trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb
2026-10-31 14:39 UTC, run 533881ac-803e-41c8-8075-e4569cc2258a
trial pass rate 100.0% (92.6–100.0%), 48/48 trials
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6
2026-10-31 14:40 UTC, run 6686e7b7-8372-4b3b-b42f-29db62cd13a9
trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6
2026-10-31 14:41 UTC, run 0cc6b085-9194-4d9f-893d-aae81e227695
trial pass rate 100.0% (92.6–100.0%), 48/48 trials
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1
2026-10-31 14:41 UTC, run 17fa6703-f6c5-486c-8355-b41482a566d3
trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1
2026-10-31 14:42 UTC, run 49b37ff7-f799-4c85-9e3d-f2aeccc2f4b3
trial pass rate 100.0% (92.6–100.0%), 48/48 trials
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb
2026-10-31 14:36 UTC, run 19c1db5c-c3a9-4dd7-a0bc-54a027ee2ef9
trial pass rate 87.5% (75.3–94.1%), 42/48 trials openrouter:z-ai/glm-5.3-flash · triage@1+b590cb
2026-10-31 14:39 UTC, run 04afa8f6-4ef2-46fc-85bd-aef41ae5358c
trial pass rate 87.5% (75.3–94.1%), 42/48 trials
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6
2026-10-31 14:40 UTC, run 95e215e0-380f-4982-9c55-4475c1229d2b
trial pass rate 93.8% (83.2–97.9%), 45/48 trials openrouter:z-ai/glm-5.3-flash · triage@2+1721e6
2026-10-31 14:41 UTC, run 36fd9f0d-411c-4a0d-848d-7f5d4a5afd43
trial pass rate 91.7% (80.4–96.7%), 44/48 trials
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1
2026-10-31 14:41 UTC, run f1daf85e-fe94-45c1-bb51-bbbc600943ff
trial pass rate 100.0% (92.6–100.0%), 48/48 trials openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1
2026-10-31 14:42 UTC, run 029fb86b-4958-4fc2-b187-c6406017f867
trial pass rate 97.9% (89.1–99.6%), 47/48 trials
openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb (2 runs) openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6 (2 runs) openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1 (2 runs) openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb (2 runs) openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6 (2 runs) openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1 (2 runs) openrouter:z-ai/glm-5.3-flash · triage@1+b590cb (2 runs) openrouter:z-ai/glm-5.3-flash · triage@2+1721e6 (2 runs) openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1 (2 runs)
X: start time of the run (UTC). Y: passed / (passed + failed) trials of the run. Whisker: 95% Wilson interval. Hollow diamond, dashed line and a break in the line: the dataset or the trial policy differs from the previous run of the series, so the rates do not compare.
Latest completed run per series
Select the start time of a run to see its cases and trials.
Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.
All 18 runs in the window
Usage and metric means
Case consistency
Per case, the trial pass rate over the completed runs in the window. At 0% and 100%, the task gives the same result each time; the bins between hold the cases where it is not consistent.
openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb
16 cases, 2 runs
0%: 0 of 16 cases (0.0%) (0, 25%]: 0 of 16 cases (0.0%) (25%, 50%]: 0 of 16 cases (0.0%) (50%, 75%]: 0 of 16 cases (0.0%) (75%, 100%): 0 of 16 cases (0.0%) 100%: 16 of 16 cases (100.0%) 16
0% ≤25% ≤50% ≤75% <100% 100%
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6
16 cases, 2 runs
0%: 0 of 16 cases (0.0%) (0, 25%]: 0 of 16 cases (0.0%) (25%, 50%]: 0 of 16 cases (0.0%) (50%, 75%]: 0 of 16 cases (0.0%) (75%, 100%): 0 of 16 cases (0.0%) 100%: 16 of 16 cases (100.0%) 16
0% ≤25% ≤50% ≤75% <100% 100%
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1
16 cases, 2 runs
0%: 0 of 16 cases (0.0%) (0, 25%]: 0 of 16 cases (0.0%) (25%, 50%]: 0 of 16 cases (0.0%) (50%, 75%]: 0 of 16 cases (0.0%) (75%, 100%): 0 of 16 cases (0.0%) 100%: 16 of 16 cases (100.0%) 16
0% ≤25% ≤50% ≤75% <100% 100%
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb
16 cases, 2 runs
0%: 0 of 16 cases (0.0%) (0, 25%]: 0 of 16 cases (0.0%) (25%, 50%]: 0 of 16 cases (0.0%) (50%, 75%]: 0 of 16 cases (0.0%) (75%, 100%): 0 of 16 cases (0.0%) 100%: 16 of 16 cases (100.0%) 16
0% ≤25% ≤50% ≤75% <100% 100%
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6
16 cases, 2 runs
0%: 0 of 16 cases (0.0%) (0, 25%]: 0 of 16 cases (0.0%) (25%, 50%]: 0 of 16 cases (0.0%) (50%, 75%]: 0 of 16 cases (0.0%) (75%, 100%): 0 of 16 cases (0.0%) 100%: 16 of 16 cases (100.0%) 16
0% ≤25% ≤50% ≤75% <100% 100%
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1
16 cases, 2 runs
0%: 0 of 16 cases (0.0%) (0, 25%]: 0 of 16 cases (0.0%) (25%, 50%]: 0 of 16 cases (0.0%) (50%, 75%]: 0 of 16 cases (0.0%) (75%, 100%): 0 of 16 cases (0.0%) 100%: 16 of 16 cases (100.0%) 16
0% ≤25% ≤50% ≤75% <100% 100%
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb
16 cases, 2 runs
0%: 2 of 16 cases (12.5%) 2 (0, 25%]: 0 of 16 cases (0.0%) (25%, 50%]: 0 of 16 cases (0.0%) (50%, 75%]: 0 of 16 cases (0.0%) (75%, 100%): 0 of 16 cases (0.0%) 100%: 14 of 16 cases (87.5%) 14
0% ≤25% ≤50% ≤75% <100% 100%
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6
16 cases, 2 runs
0%: 1 of 16 cases (6.3%) 1 (0, 25%]: 0 of 16 cases (0.0%) (25%, 50%]: 0 of 16 cases (0.0%) (50%, 75%]: 0 of 16 cases (0.0%) (75%, 100%): 1 of 16 cases (6.3%) 1 100%: 14 of 16 cases (87.5%) 14
0% ≤25% ≤50% ≤75% <100% 100%
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1
16 cases, 2 runs
0%: 0 of 16 cases (0.0%) (0, 25%]: 0 of 16 cases (0.0%) (25%, 50%]: 0 of 16 cases (0.0%) (50%, 75%]: 0 of 16 cases (0.0%) (75%, 100%): 1 of 16 cases (6.3%) 1 100%: 15 of 16 cases (93.8%) 15
0% ≤25% ≤50% ≤75% <100% 100%
Score distributions
Per scorer, the score values of all trials of the completed runs in the window, in 10 bins from 0 to 1.
category@1
openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 0 of 96 scores (0.0%) [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 96 of 96 scores (100.0%) 96
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 0 of 96 scores (0.0%) [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 96 of 96 scores (100.0%) 96
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 0 of 96 scores (0.0%) [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 96 of 96 scores (100.0%) 96
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 0 of 96 scores (0.0%) [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 96 of 96 scores (100.0%) 96
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 0 of 96 scores (0.0%) [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 96 of 96 scores (100.0%) 96
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 0 of 96 scores (0.0%) [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 96 of 96 scores (100.0%) 96
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb
96 scores
[0.0, 0.1): 12 of 96 scores (12.5%) 12 [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 0 of 96 scores (0.0%) [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 84 of 96 scores (87.5%) 84
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6
96 scores
[0.0, 0.1): 7 of 96 scores (7.3%) 7 [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 0 of 96 scores (0.0%) [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 89 of 96 scores (92.7%) 89
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1
96 scores
[0.0, 0.1): 1 of 96 scores (1.0%) 1 [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 0 of 96 scores (0.0%) [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 95 of 96 scores (99.0%) 95
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
priority@1
openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 13 of 96 scores (13.5%) 13 [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 83 of 96 scores (86.5%) 83
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 12 of 96 scores (12.5%) 12 [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 84 of 96 scores (87.5%) 84
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 11 of 96 scores (11.5%) 11 [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 85 of 96 scores (88.5%) 85
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 5 of 96 scores (5.2%) 5 [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 91 of 96 scores (94.8%) 91
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 6 of 96 scores (6.3%) 6 [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 90 of 96 scores (93.8%) 90
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 7 of 96 scores (7.3%) 7 [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 89 of 96 scores (92.7%) 89
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 1 of 96 scores (1.0%) 1 [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 95 of 96 scores (99.0%) 95
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 8 of 96 scores (8.3%) 8 [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 88 of 96 scores (91.7%) 88
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1
96 scores
[0.0, 0.1): 0 of 96 scores (0.0%) [0.1, 0.2): 0 of 96 scores (0.0%) [0.2, 0.3): 0 of 96 scores (0.0%) [0.3, 0.4): 0 of 96 scores (0.0%) [0.4, 0.5): 0 of 96 scores (0.0%) [0.5, 0.6): 0 of 96 scores (0.0%) [0.6, 0.7): 3 of 96 scores (3.1%) 3 [0.7, 0.8): 0 of 96 scores (0.0%) [0.8, 0.9): 0 of 96 scores (0.0%) [0.9, 1.0]: 93 of 96 scores (96.9%) 93
0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1
Flips
No flips since the previous completed run of each series.
← Back to the report
support-triage
openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb
Started 2026-10-31 14:37 UTC · run 75a75702-874d-41a0-965f-ea935521e996
Source JSONL: examples/support-triage/history-sample/support-triage.75a75702-874d-41a0-965f-ea935521e996.jsonl
16 pass
Tokens in / out / cached / reasoning: 24195 / 5778 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:anthropic/claude-haiku-5.5 prompt=triage@1+b590cb env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:anthropic/claude-haiku-5.5 · triage@1+b590cb
Started 2026-10-31 14:39 UTC · run 9b0035e6-4312-42e2-8b11-4fc0cf4caa01
Source JSONL: examples/support-triage/history-sample/support-triage.9b0035e6-4312-42e2-8b11-4fc0cf4caa01.jsonl
16 pass
Tokens in / out / cached / reasoning: 24195 / 4889 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:anthropic/claude-haiku-5.5 prompt=triage@1+b590cb env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6
Started 2026-10-31 14:40 UTC · run d03d1344-15f2-4477-a509-22ee13950699
Source JSONL: examples/support-triage/history-sample/support-triage.d03d1344-15f2-4477-a509-22ee13950699.jsonl
16 pass
Tokens in / out / cached / reasoning: 25635 / 6197 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:anthropic/claude-haiku-5.5 prompt=triage@2+1721e6 env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:anthropic/claude-haiku-5.5 · triage@2+1721e6
Started 2026-10-31 14:41 UTC · run 03378131-9f85-483d-9096-fb35368d1e08
Source JSONL: examples/support-triage/history-sample/support-triage.03378131-9f85-483d-9096-fb35368d1e08.jsonl
16 pass
Tokens in / out / cached / reasoning: 25635 / 6071 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:anthropic/claude-haiku-5.5 prompt=triage@2+1721e6 env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1
Started 2026-10-31 14:41 UTC · run b108cdab-a644-4e7b-a8a6-a7428b36490d
Source JSONL: examples/support-triage/history-sample/support-triage.b108cdab-a644-4e7b-a8a6-a7428b36490d.jsonl
16 pass
Tokens in / out / cached / reasoning: 27219 / 7350 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:anthropic/claude-haiku-5.5 prompt=triage@3+6de1a1 env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:anthropic/claude-haiku-5.5 · triage@3+6de1a1
Started 2026-10-31 14:42 UTC · run 581f294d-9557-4ab6-a412-250db41fab6d
Source JSONL: examples/support-triage/history-sample/support-triage.581f294d-9557-4ab6-a412-250db41fab6d.jsonl
16 pass
Tokens in / out / cached / reasoning: 27219 / 6887 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:anthropic/claude-haiku-5.5 prompt=triage@3+6de1a1 env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb
Started 2026-10-31 14:38 UTC · run e00d49cb-a385-4d81-b929-51c9ceb67f69
Source JSONL: examples/support-triage/history-sample/support-triage.e00d49cb-a385-4d81-b929-51c9ceb67f69.jsonl
16 pass
Tokens in / out / cached / reasoning: 9184 / 11375 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:deepseek/deepseek-v4.1-flash prompt=triage@1+b590cb env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:deepseek/deepseek-v4.1-flash · triage@1+b590cb
Started 2026-10-31 14:39 UTC · run 533881ac-803e-41c8-8075-e4569cc2258a
Source JSONL: examples/support-triage/history-sample/support-triage.533881ac-803e-41c8-8075-e4569cc2258a.jsonl
16 pass
Tokens in / out / cached / reasoning: 9180 / 12266 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:deepseek/deepseek-v4.1-flash prompt=triage@1+b590cb env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6
Started 2026-10-31 14:40 UTC · run 6686e7b7-8372-4b3b-b42f-29db62cd13a9
Source JSONL: examples/support-triage/history-sample/support-triage.6686e7b7-8372-4b3b-b42f-29db62cd13a9.jsonl
16 pass
Tokens in / out / cached / reasoning: 8961 / 11164 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:deepseek/deepseek-v4.1-flash prompt=triage@2+1721e6 env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:deepseek/deepseek-v4.1-flash · triage@2+1721e6
Started 2026-10-31 14:41 UTC · run 0cc6b085-9194-4d9f-893d-aae81e227695
Source JSONL: examples/support-triage/history-sample/support-triage.0cc6b085-9194-4d9f-893d-aae81e227695.jsonl
16 pass
Tokens in / out / cached / reasoning: 8961 / 11274 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:deepseek/deepseek-v4.1-flash prompt=triage@2+1721e6 env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1
Started 2026-10-31 14:41 UTC · run 17fa6703-f6c5-486c-8355-b41482a566d3
Source JSONL: examples/support-triage/history-sample/support-triage.17fa6703-f6c5-486c-8355-b41482a566d3.jsonl
16 pass
Tokens in / out / cached / reasoning: 9792 / 14538 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:deepseek/deepseek-v4.1-flash prompt=triage@3+6de1a1 env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:deepseek/deepseek-v4.1-flash · triage@3+6de1a1
Started 2026-10-31 14:42 UTC · run 49b37ff7-f799-4c85-9e3d-f2aeccc2f4b3
Source JSONL: examples/support-triage/history-sample/support-triage.49b37ff7-f799-4c85-9e3d-f2aeccc2f4b3.jsonl
16 pass
Tokens in / out / cached / reasoning: 10062 / 11453 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:deepseek/deepseek-v4.1-flash prompt=triage@3+6de1a1 env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb
Started 2026-10-31 14:36 UTC · run 19c1db5c-c3a9-4dd7-a0bc-54a027ee2ef9
Source JSONL: examples/support-triage/history-sample/support-triage.19c1db5c-c3a9-4dd7-a0bc-54a027ee2ef9.jsonl
14 pass , 2 fail
Tokens in / out / cached / reasoning: 7053 / 9394 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:z-ai/glm-5.3-flash prompt=triage@1+b590cb env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent fail (0/3)
Tags: bug ambiguous
0 of 3 trials passed, needs 2; category: expected bug, got billing
Trial 1 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing fail (0/3)
Tags: bug ambiguous
0 of 3 trials passed, needs 2; category: expected bug, got billing
Trial 1 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:z-ai/glm-5.3-flash · triage@1+b590cb
Started 2026-10-31 14:39 UTC · run 04afa8f6-4ef2-46fc-85bd-aef41ae5358c
Source JSONL: examples/support-triage/history-sample/support-triage.04afa8f6-4ef2-46fc-85bd-aef41ae5358c.jsonl
14 pass , 2 fail
Tokens in / out / cached / reasoning: 7029 / 9289 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:z-ai/glm-5.3-flash prompt=triage@1+b590cb env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent fail (0/3)
Tags: bug ambiguous
0 of 3 trials passed, needs 2; category: expected bug, got billing
Trial 1 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing fail (0/3)
Tags: bug ambiguous
0 of 3 trials passed, needs 2; category: expected bug, got billing
Trial 1 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6
Started 2026-10-31 14:40 UTC · run 95e215e0-380f-4982-9c55-4475c1229d2b
Source JSONL: examples/support-triage/history-sample/support-triage.95e215e0-380f-4982-9c55-4475c1229d2b.jsonl
15 pass , 1 fail
Tokens in / out / cached / reasoning: 8253 / 13253 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:z-ai/glm-5.3-flash prompt=triage@2+1721e6 env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent fail (0/3)
Tags: bug ambiguous
0 of 3 trials passed, needs 2; category: expected bug, got billing
Trial 1 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:z-ai/glm-5.3-flash · triage@2+1721e6
Started 2026-10-31 14:41 UTC · run 36fd9f0d-411c-4a0d-848d-7f5d4a5afd43
Source JSONL: examples/support-triage/history-sample/support-triage.36fd9f0d-411c-4a0d-848d-7f5d4a5afd43.jsonl
15 pass , 1 fail
Tokens in / out / cached / reasoning: 8271 / 13718 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:z-ai/glm-5.3-flash prompt=triage@2+1721e6 env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent fail (0/3)
Tags: bug ambiguous
0 of 3 trials passed, needs 2; category: expected bug, got billing
Trial 1 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (2/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1
Started 2026-10-31 14:41 UTC · run f1daf85e-fe94-45c1-bb51-bbbc600943ff
Source JSONL: examples/support-triage/history-sample/support-triage.f1daf85e-fe94-45c1-bb51-bbbc600943ff.jsonl
16 pass
Tokens in / out / cached / reasoning: 9189 / 13074 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:z-ai/glm-5.3-flash prompt=triage@3+6de1a1 env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 0.67
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
← Back to the report
support-triage
openrouter:z-ai/glm-5.3-flash · triage@3+6de1a1
Started 2026-10-31 14:42 UTC · run 029fb86b-4958-4fc2-b187-c6406017f867
Source JSONL: examples/support-triage/history-sample/support-triage.029fb86b-4958-4fc2-b187-c6406017f867.jsonl
16 pass
Tokens in / out / cached / reasoning: 9201 / 12759 / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost
Labels: model=openrouter:z-ai/glm-5.3-flash prompt=triage@3+6de1a1 env=local
Cases
api-webhooks pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
card-declined-upgrade pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
change-email pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
charged-after-cancel pass (3/3)
Tags: billing ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
dark-mode pass (3/3)
Tags: feature_request
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected feature_request, got feature_request
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
delete-account pass (3/3)
Tags: account
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
double-charge pass (3/3)
Tags: billing regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
locked-out-2fa pass (3/3)
Tags: account regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
outage pass (3/3)
Tags: bug regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
partnership pass (3/3)
Tags: other
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected other, got other
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
pdf-export-wrong-total pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
recurring-not-sent pass (2/3)
Tags: bug ambiguous
Trial 1 fail
Input and output Not recorded.
Scores category@1 fail 0.00
expected bug, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
refund-policy pass (3/3)
Tags: billing
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected billing, got billing
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
slow-search pass (3/3)
Tags: bug
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
suspicious-login pass (3/3)
Tags: account security regression
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
expected account, got account
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
vat-missing pass (3/3)
Tags: bug ambiguous
Trial 1 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 2 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded
Trial 3 pass
Input and output Not recorded.
Scores category@1 pass 1.00
priority@1 pass 1.00
Trace Not recorded
Logs Not recorded