← trialkit docs: History and reports

trialkit report

1 eval, 4 runs, from 2026-11-02 04:40 to 2026-11-02 04:42 UTC.

Series by model.

routing

Dataset: routing 3c73416b (10 cases).

Trial pass rate over time

100%90%80%70%60% trial pass rate (the axis starts at 60%) 04:40:5604:41:1504:41:3404:41:5404:42:13 typesafe/jev-router 2026-11-02 04:40 UTC, run c1fd9ee0-26e5-47b9-b0b7-a3d482c454bb trial pass rate 90.0% (74.4–96.5%), 27/30 trialstypesafe/jev-router 2026-11-02 04:42 UTC, run e6510e32-6e14-4556-9b3f-a090e42a655e trial pass rate 90.0% (74.4–96.5%), 27/30 trials z-ai/glm-5.3-flash 2026-11-02 04:40 UTC, run 172b1821-1e76-43a0-81fe-90be27286b01 trial pass rate 100.0% (88.6–100.0%), 30/30 trialsz-ai/glm-5.3-flash 2026-11-02 04:42 UTC, run a1d65af4-5763-40ca-a1ce-322e7462cee2 trial pass rate 93.3% (78.7–98.2%), 28/30 trials
  • typesafe/jev-router (2 runs)
  • z-ai/glm-5.3-flash (2 runs)
X: start time of the run (UTC). Y: passed / (passed + failed) trials of the run. Whisker: 95% Wilson interval. Hollow diamond, dashed line and a break in the line: the dataset or the trial policy differs from the previous run of the series, so the rates do not compare. Dotted line with a value: the run where prompt changed to that value.

Latest completed run per series

Select the start time of a run to see its cases and trials.

SeriespromptRunsCasesCase pass rate (95% CI)Trial pass rate (95% CI)TrendError trialsMean trialLatest run (UTC)
typesafe/jev-router-29 pass, 1 fail90.0% 9/10
59.6–98.2%
90.0% 27/30
74.4–96.5%
▇▇02.3 s2026-11-02 04:42
z-ai/glm-5.3-flash-210 pass100.0% 10/10
72.2–100.0%
93.3% 28/30
78.7–98.2%
██01.6 s2026-11-02 04:42

Trend: the trial pass rate of each completed run in the window, oldest first, on a fixed scale from ▁ 0% to █ 100%, so rows compare.

All 4 runs in the window
SeriespromptStarted (UTC)Trial pass rate (95% CI)Case pass rate (95% CI)Error trialsCommitNotes
typesafe/jev-router- 2026-11-02 04:40 90.0% 27/30
74.4–96.5%
90.0% 9/10
59.6–98.2%
0 –
- 2026-11-02 04:42 90.0% 27/30
74.4–96.5%
90.0% 9/10
59.6–98.2%
0 –
z-ai/glm-5.3-flash- 2026-11-02 04:40 100.0% 30/30
88.6–100.0%
100.0% 10/10
72.2–100.0%
0 –
- 2026-11-02 04:42 93.3% 28/30
78.7–98.2%
100.0% 10/10
72.2–100.0%
0 –

Usage and metric means

SeriesMetric means
typesafe/jev-routercostUsd: 0.0009284423333
z-ai/glm-5.3-flashcostUsd: 0.00007655463333

Case consistency

Per case, the trial pass rate over the completed runs in the window. At 0% and 100%, the task gives the same result each time; the bins between hold the cases where it is not consistent.

typesafe/jev-router
10 cases, 2 runs
0%: 0 of 10 cases (0.0%)(0, 25%]: 1 of 10 cases (10.0%)1(25%, 50%]: 0 of 10 cases (0.0%)(50%, 75%]: 0 of 10 cases (0.0%)(75%, 100%): 1 of 10 cases (10.0%)1100%: 8 of 10 cases (80.0%)8 0%≤25%≤50%≤75%<100%100%
z-ai/glm-5.3-flash
10 cases, 2 runs
0%: 0 of 10 cases (0.0%)(0, 25%]: 0 of 10 cases (0.0%)(25%, 50%]: 0 of 10 cases (0.0%)(50%, 75%]: 0 of 10 cases (0.0%)(75%, 100%): 2 of 10 cases (20.0%)2100%: 8 of 10 cases (80.0%)8 0%≤25%≤50%≤75%<100%100%

Score distributions

Per scorer, the score values of all trials of the completed runs in the window, in 10 bins from 0 to 1.

answer@1

typesafe/jev-router
60 scores
[0.0, 0.1): 6 of 60 scores (10.0%)6[0.1, 0.2): 0 of 60 scores (0.0%)[0.2, 0.3): 0 of 60 scores (0.0%)[0.3, 0.4): 0 of 60 scores (0.0%)[0.4, 0.5): 0 of 60 scores (0.0%)[0.5, 0.6): 0 of 60 scores (0.0%)[0.6, 0.7): 0 of 60 scores (0.0%)[0.7, 0.8): 0 of 60 scores (0.0%)[0.8, 0.9): 0 of 60 scores (0.0%)[0.9, 1.0]: 54 of 60 scores (90.0%)54 0.1.2.3.4.5.6.7.8.91
z-ai/glm-5.3-flash
60 scores
[0.0, 0.1): 2 of 60 scores (3.3%)2[0.1, 0.2): 0 of 60 scores (0.0%)[0.2, 0.3): 0 of 60 scores (0.0%)[0.3, 0.4): 0 of 60 scores (0.0%)[0.4, 0.5): 0 of 60 scores (0.0%)[0.5, 0.6): 0 of 60 scores (0.0%)[0.6, 0.7): 0 of 60 scores (0.0%)[0.7, 0.8): 0 of 60 scores (0.0%)[0.8, 0.9): 0 of 60 scores (0.0%)[0.9, 1.0]: 58 of 60 scores (96.7%)58 0.1.2.3.4.5.6.7.8.91

Flips

No flips since the previous completed run of each series.

← Back to the report

routing

typesafe/jev-router

Started 2026-11-02 04:40 UTC · run c1fd9ee0-26e5-47b9-b0b7-a3d482c454bb

Source JSONL: examples/compare-models/history/routing/typesafe-jev-router/routing.c1fd9ee0-26e5-47b9-b0b7-a3d482c454bb.jsonl

9 pass, 1 fail

Tokens in / out / cached / reasoning: unknown / unknown / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=typesafe/jev-router env=local

Cases

clause-month-days pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund. Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
  • answer@1 pass 1.00
    expected "4.4", got "4.4"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund. Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
  • answer@1 pass 1.00
    expected "4.4", got "4.4"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund. Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
  • answer@1 pass 1.00
    expected "4.4", got "4.4"
Trace

Not recorded

Logs

Not recorded

clause-upgrade fail (1/3)

Tags: hard

1 of 3 trials passed, needs 2; answer: expected "4.3", got "Yearly plan began 20 January. 15 February is 26 days later, which is within 30 days, so a full refund under 4.3.\n\nBut the question asks which clause gives the amount: 4.3 (with 4.5 applying the yea…
Trial 1 fail
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund. Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
Yearly plan began 20 January; 15 February is 26 days later, so within 30 days. Full refund under 4.3. **4.3**
Scores
  • answer@1 fail 0.00
    expected "4.3", got "Yearly plan began 20 January; 15 February is 26 days later, so within 30 days. Full refund under 4.3.\n\n**4.3**"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund. Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
  • answer@1 pass 1.00
    expected "4.3", got "4.3"
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund. Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
Yearly plan began 20 January. 15 February is 26 days later, which is within 30 days, so a full refund under 4.3. But the question asks which clause gives the amount: 4.3 (with 4.5 applying the yearly rules). The amount is given by 4.3. **4.3**
Scores
  • answer@1 fail 0.00
    expected "4.3", got "Yearly plan began 20 January. 15 February is 26 days later, which is within 30 days, so a full refund under 4.3.\n\nBut the question asks which clause gives the amount: 4.3 (with 4.5 applying the yea…
Trace

Not recorded

Logs

Not recorded

crash pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
bug
Scores
  • answer@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
bug
Scores
  • answer@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
bug
Scores
  • answer@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

invoice pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
billing
Scores
  • answer@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
billing
Scores
  • answer@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
billing
Scores
  • answer@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

login pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
account
Scores
  • answer@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
account
Scores
  • answer@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
account
Scores
  • answer@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

praise pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

refund-added-line pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): 20.00 - Setup fee: 40.00 The customer cancels at the end of 20 June. How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
  • answer@1 pass 1.00
    expected "4000", got "4000"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): 20.00 - Setup fee: 40.00 The customer cancels at the end of 20 June. How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
  • answer@1 pass 1.00
    expected "4000", got "4000"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): 20.00 - Setup fee: 40.00 The customer cancels at the end of 20 June. How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
  • answer@1 pass 1.00
    expected "4000", got "4000"
Trace

Not recorded

Logs

Not recorded

refund-credit pass (2/3)

Tags: hard

Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): …Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): - Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00 - Archive add-on: 15.00 - Setup fee: 30.00 The customer cancels at the end of 12 April. How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
  • answer@1 pass 1.00
    expected "3060", got "3060"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): …Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): - Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00 - Archive add-on: 15.00 - Setup fee: 30.00 The customer cancels at the end of 12 April. How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
  • answer@1 pass 1.00
    expected "3060", got "3060"
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): …Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): - Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00 - Archive add-on: 15.00 - Setup fee: 30.00 The customer cancels at the end of 12 April. How much is refunded? Reply with the amount in pence, digits only.
Output
3160
Scores
  • answer@1 fail 0.00
    expected "3060", got "3160"
Trace

Not recorded

Logs

Not recorded

refund-currency pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July (31 days): - Business plan, 4 seats × 25.00: 100.00 - Priority support: 31.00 - Setup fee: 50.00 The customer cancels at the end of 9 July. How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
  • answer@1 pass 1.00
    expected "8460", got "8460"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July (31 days): - Business plan, 4 seats × 25.00: 100.00 - Priority support: 31.00 - Setup fee: 50.00 The customer cancels at the end of 9 July. How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
  • answer@1 pass 1.00
    expected "8460", got "8460"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July (31 days): - Business plan, 4 seats × 25.00: 100.00 - Priority support: 31.00 - Setup fee: 50.00 The customer cancels at the end of 9 July. How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
  • answer@1 pass 1.00
    expected "8460", got "8460"
Trace

Not recorded

Logs

Not recorded

← Back to the report

routing

typesafe/jev-router

Started 2026-11-02 04:42 UTC · run e6510e32-6e14-4556-9b3f-a090e42a655e

Source JSONL: examples/compare-models/history/routing/typesafe-jev-router/routing.e6510e32-6e14-4556-9b3f-a090e42a655e.jsonl

9 pass, 1 fail

Tokens in / out / cached / reasoning: unknown / unknown / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=typesafe/jev-router env=local

Cases

clause-month-days pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund. Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
  • answer@1 pass 1.00
    expected "4.4", got "4.4"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund. Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
  • answer@1 pass 1.00
    expected "4.4", got "4.4"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund. Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
  • answer@1 pass 1.00
    expected "4.4", got "4.4"
Trace

Not recorded

Logs

Not recorded

clause-upgrade fail (0/3)

Tags: hard

0 of 3 trials passed, needs 2; answer: expected "4.3", got "Yearly plan began 20 January; 15 February is 26 days later, which is within 30 days, so a full refund applies under 4.3.\n\n**4.3**"
Trial 1 fail
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund. Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
Yearly plan began 20 January. 15 February is 26 days later, which is within 30 days, so a full refund under 4.3. But the question asks which clause gives the amount: 4.3 (with 4.5 applying the start date). The amount is given by 4.3. **4.3**
Scores
  • answer@1 fail 0.00
    expected "4.3", got "Yearly plan began 20 January. 15 February is 26 days later, which is within 30 days, so a full refund under 4.3.\n\nBut the question asks which clause gives the amount: 4.3 (with 4.5 applying the sta…
Trace

Not recorded

Logs

Not recorded

Trial 2 fail
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund. Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
The yearly plan began on 20 January (clause 4.5). On 15 February, 26 days have passed (11 days left in January after the 20th, plus 15 days in February), which is within 30 days. A full refund therefore applies under clause 4.3. **4.3**
Scores
  • answer@1 fail 0.00
    expected "4.3", got "The yearly plan began on 20 January (clause 4.5). On 15 February, 26 days have passed (11 days left in January after the 20th, plus 15 days in February), which is within 30 days. A full refund theref…
Trace

Not recorded

Logs

Not recorded

Trial 3 fail
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund. Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
Yearly plan began 20 January; 15 February is 26 days later, which is within 30 days, so a full refund applies under 4.3. **4.3**
Scores
  • answer@1 fail 0.00
    expected "4.3", got "Yearly plan began 20 January; 15 February is 26 days later, which is within 30 days, so a full refund applies under 4.3.\n\n**4.3**"
Trace

Not recorded

Logs

Not recorded

crash pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
bug
Scores
  • answer@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
bug
Scores
  • answer@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
bug
Scores
  • answer@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

invoice pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
billing
Scores
  • answer@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
billing
Scores
  • answer@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
billing
Scores
  • answer@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

login pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
account
Scores
  • answer@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
account
Scores
  • answer@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
account
Scores
  • answer@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

praise pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

refund-added-line pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): 20.00 - Setup fee: 40.00 The customer cancels at the end of 20 June. How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
  • answer@1 pass 1.00
    expected "4000", got "4000"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): 20.00 - Setup fee: 40.00 The customer cancels at the end of 20 June. How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
  • answer@1 pass 1.00
    expected "4000", got "4000"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): 20.00 - Setup fee: 40.00 The customer cancels at the end of 20 June. How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
  • answer@1 pass 1.00
    expected "4000", got "4000"
Trace

Not recorded

Logs

Not recorded

refund-credit pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): …Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): - Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00 - Archive add-on: 15.00 - Setup fee: 30.00 The customer cancels at the end of 12 April. How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
  • answer@1 pass 1.00
    expected "3060", got "3060"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): …Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): - Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00 - Archive add-on: 15.00 - Setup fee: 30.00 The customer cancels at the end of 12 April. How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
  • answer@1 pass 1.00
    expected "3060", got "3060"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): …Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): - Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00 - Archive add-on: 15.00 - Setup fee: 30.00 The customer cancels at the end of 12 April. How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
  • answer@1 pass 1.00
    expected "3060", got "3060"
Trace

Not recorded

Logs

Not recorded

refund-currency pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July (31 days): - Business plan, 4 seats × 25.00: 100.00 - Priority support: 31.00 - Setup fee: 50.00 The customer cancels at the end of 9 July. How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
  • answer@1 pass 1.00
    expected "8460", got "8460"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July (31 days): - Business plan, 4 seats × 25.00: 100.00 - Priority support: 31.00 - Setup fee: 50.00 The customer cancels at the end of 9 July. How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
  • answer@1 pass 1.00
    expected "8460", got "8460"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July (31 days): - Business plan, 4 seats × 25.00: 100.00 - Priority support: 31.00 - Setup fee: 50.00 The customer cancels at the end of 9 July. How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
  • answer@1 pass 1.00
    expected "8460", got "8460"
Trace

Not recorded

Logs

Not recorded

← Back to the report

routing

z-ai/glm-5.3-flash

Started 2026-11-02 04:40 UTC · run 172b1821-1e76-43a0-81fe-90be27286b01

Source JSONL: examples/compare-models/history/routing/z-ai-glm-5.3-flash/routing.172b1821-1e76-43a0-81fe-90be27286b01.jsonl

10 pass

Tokens in / out / cached / reasoning: unknown / unknown / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=z-ai/glm-5.3-flash env=local

Cases

clause-month-days pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund. Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
  • answer@1 pass 1.00
    expected "4.4", got "4.4"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund. Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
  • answer@1 pass 1.00
    expected "4.4", got "4.4"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund. Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
  • answer@1 pass 1.00
    expected "4.4", got "4.4"
Trace

Not recorded

Logs

Not recorded

clause-upgrade pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund. Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
  • answer@1 pass 1.00
    expected "4.3", got "4.3"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund. Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
  • answer@1 pass 1.00
    expected "4.3", got "4.3"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund. Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
  • answer@1 pass 1.00
    expected "4.3", got "4.3"
Trace

Not recorded

Logs

Not recorded

crash pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
bug
Scores
  • answer@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
bug
Scores
  • answer@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
bug
Scores
  • answer@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

invoice pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
billing
Scores
  • answer@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
billing
Scores
  • answer@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
billing
Scores
  • answer@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

login pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
account
Scores
  • answer@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
account
Scores
  • answer@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
account
Scores
  • answer@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

praise pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

refund-added-line pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): 20.00 - Setup fee: 40.00 The customer cancels at the end of 20 June. How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
  • answer@1 pass 1.00
    expected "4000", got "4000"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): 20.00 - Setup fee: 40.00 The customer cancels at the end of 20 June. How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
  • answer@1 pass 1.00
    expected "4000", got "4000"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): 20.00 - Setup fee: 40.00 The customer cancels at the end of 20 June. How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
  • answer@1 pass 1.00
    expected "4000", got "4000"
Trace

Not recorded

Logs

Not recorded

refund-credit pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): …Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): - Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00 - Archive add-on: 15.00 - Setup fee: 30.00 The customer cancels at the end of 12 April. How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
  • answer@1 pass 1.00
    expected "3060", got "3060"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): …Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): - Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00 - Archive add-on: 15.00 - Setup fee: 30.00 The customer cancels at the end of 12 April. How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
  • answer@1 pass 1.00
    expected "3060", got "3060"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): …Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): - Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00 - Archive add-on: 15.00 - Setup fee: 30.00 The customer cancels at the end of 12 April. How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
  • answer@1 pass 1.00
    expected "3060", got "3060"
Trace

Not recorded

Logs

Not recorded

refund-currency pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July (31 days): - Business plan, 4 seats × 25.00: 100.00 - Priority support: 31.00 - Setup fee: 50.00 The customer cancels at the end of 9 July. How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
  • answer@1 pass 1.00
    expected "8460", got "8460"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July (31 days): - Business plan, 4 seats × 25.00: 100.00 - Priority support: 31.00 - Setup fee: 50.00 The customer cancels at the end of 9 July. How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
  • answer@1 pass 1.00
    expected "8460", got "8460"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July (31 days): - Business plan, 4 seats × 25.00: 100.00 - Priority support: 31.00 - Setup fee: 50.00 The customer cancels at the end of 9 July. How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
  • answer@1 pass 1.00
    expected "8460", got "8460"
Trace

Not recorded

Logs

Not recorded

← Back to the report

routing

z-ai/glm-5.3-flash

Started 2026-11-02 04:42 UTC · run a1d65af4-5763-40ca-a1ce-322e7462cee2

Source JSONL: examples/compare-models/history/routing/z-ai-glm-5.3-flash/routing.a1d65af4-5763-40ca-a1ce-322e7462cee2.jsonl

10 pass

Tokens in / out / cached / reasoning: unknown / unknown / unknown / unknown; cost (USD): unknown; judge share: unknown tokens, unknown cost

Labels: model=z-ai/glm-5.3-flash env=local

Cases

clause-month-days pass (2/3)

Tags: hard

Trial 1 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund. Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
  • answer@1 pass 1.00
    expected "4.4", got "4.4"
Trace

Not recorded

Logs

Not recorded

Trial 2 fail
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund. Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
  • answer@1 fail 0.00
    expected "4.4", got "4.3"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund. Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
  • answer@1 pass 1.00
    expected "4.4", got "4.4"
Trace

Not recorded

Logs

Not recorded

clause-upgrade pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund. Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
  • answer@1 pass 1.00
    expected "4.3", got "4.3"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund. Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
  • answer@1 pass 1.00
    expected "4.3", got "4.3"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract): 4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB. 4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month. 4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use. 4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price. 4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day. A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund. Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
  • answer@1 pass 1.00
    expected "4.3", got "4.3"
Trace

Not recorded

Logs

Not recorded

crash pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
bug
Scores
  • answer@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
bug
Scores
  • answer@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
bug
Scores
  • answer@1 pass 1.00
    expected "bug", got "bug"
Trace

Not recorded

Logs

Not recorded

dark-mode pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

invoice pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
billing
Scores
  • answer@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
billing
Scores
  • answer@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
billing
Scores
  • answer@1 pass 1.00
    expected "billing", got "billing"
Trace

Not recorded

Logs

Not recorded

login pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
account
Scores
  • answer@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
account
Scores
  • answer@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password." billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
account
Scores
  • answer@1 pass 1.00
    expected "account", got "account"
Trace

Not recorded

Logs

Not recorded

praise pass (3/3)

Tags: easy

Trial 1 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!" billing: a charge, a refund, an invoice, a price or a payment method. account: login, password, profile, team members or the plan. bug: an error, a crash or a wrong result in the app. other: praise, a feature request, a question about how to use the app, or anything else. Reply with the category only.
Output
other
Scores
  • answer@1 pass 1.00
    expected "other", got "other"
Trace

Not recorded

Logs

Not recorded

refund-added-line pass (2/3)

Tags: hard

Trial 1 fail
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): 20.00 - Setup fee: 40.00 The customer cancels at the end of 20 June. How much is refunded? Reply with the amount in euro cents, digits only.
Output
9000
Scores
  • answer@1 fail 0.00
    expected "4000", got "9000"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): 20.00 - Setup fee: 40.00 The customer cancels at the end of 20 June. How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
  • answer@1 pass 1.00
    expected "4000", got "4000"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded. Invoice in EUR, 1 to 30 June: - Team plan, 1 to 30 June (30 days): 90.00 - Extra storage, 11 to 30 June (20 days): 20.00 - Setup fee: 40.00 The customer cancels at the end of 20 June. How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
  • answer@1 pass 1.00
    expected "4000", got "4000"
Trace

Not recorded

Logs

Not recorded

refund-credit pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): …Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): - Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00 - Archive add-on: 15.00 - Setup fee: 30.00 The customer cancels at the end of 12 April. How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
  • answer@1 pass 1.00
    expected "3060", got "3060"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): …Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): - Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00 - Archive add-on: 15.00 - Setup fee: 30.00 The customer cancels at the end of 12 April. How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
  • answer@1 pass 1.00
    expected "3060", got "3060"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): …Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded. Invoice in GBP, 1 to 30 April (30 days): - Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00 - Archive add-on: 15.00 - Setup fee: 30.00 The customer cancels at the end of 12 April. How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
  • answer@1 pass 1.00
    expected "3060", got "3060"
Trace

Not recorded

Logs

Not recorded

refund-currency pass (3/3)

Tags: hard

Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July (31 days): - Business plan, 4 seats × 25.00: 100.00 - Priority support: 31.00 - Setup fee: 50.00 The customer cancels at the end of 9 July. How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
  • answer@1 pass 1.00
    expected "8460", got "8460"
Trace

Not recorded

Logs

Not recorded

Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July (31 days): - Business plan, 4 seats × 25.00: 100.00 - Priority support: 31.00 - Setup fee: 50.00 The customer cancels at the end of 9 July. How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
  • answer@1 pass 1.00
    expected "8460", got "8460"
Trace

Not recorded

Logs

Not recorded

Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total. Invoice in USD, 1 to 31 July (31 days): - Business plan, 4 seats × 25.00: 100.00 - Priority support: 31.00 - Setup fee: 50.00 The customer cancels at the end of 9 July. How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
  • answer@1 pass 1.00
    expected "8460", got "8460"
Trace

Not recorded

Logs

Not recorded