1 eval, 4 runs, from 2026-11-02 04:40 to 2026-11-02 04:42 UTC.
Series by model.
routing
Dataset: routing 3c73416b (10 cases).
Trial pass rate over time
typesafe/jev-router(2 runs)
z-ai/glm-5.3-flash(2 runs)
X: start time of the run (UTC). Y: passed / (passed + failed) trials of the run. Whisker: 95% Wilson interval. Hollow diamond, dashed line and a break in the line: the dataset or the trial policy differs from the previous run of the series, so the rates do not compare. Dotted line with a value: the run where prompt changed to that value.
Latest completed run per series
Select the start time of a run to see its cases and trials.
Per case, the trial pass rate over the completed runs in the window. At 0% and 100%, the task gives the same result each time; the bins between hold the cases where it is not consistent.
typesafe/jev-router
10 cases, 2 runs
z-ai/glm-5.3-flash
10 cases, 2 runs
Score distributions
Per scorer, the score values of all trials of the completed runs in the window, in 10 bins from 0 to 1.
answer@1
typesafe/jev-router
60 scores
z-ai/glm-5.3-flash
60 scores
Flips
No flips since the previous completed run of each series.
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund.
Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
answer@1pass 1.00
expected "4.4", got "4.4"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund.
Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
answer@1pass 1.00
expected "4.4", got "4.4"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund.
Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
answer@1pass 1.00
expected "4.4", got "4.4"
Trace
Not recorded
Logs
Not recorded
clause-upgradefail (1/3)
Tags: hard
1 of 3 trials passed, needs 2; answer: expected "4.3", got "Yearly plan began 20 January. 15 February is 26 days later, which is within 30 days, so a full refund under 4.3.\n\nBut the question asks which clause gives the amount: 4.3 (with 4.5 applying the yea…
Trial 1 fail
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund.
Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
Yearly plan began 20 January; 15 February is 26 days later, so within 30 days. Full refund under 4.3.
**4.3**
Scores
answer@1fail 0.00
expected "4.3", got "Yearly plan began 20 January; 15 February is 26 days later, so within 30 days. Full refund under 4.3.\n\n**4.3**"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund.
Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
answer@1pass 1.00
expected "4.3", got "4.3"
Trace
Not recorded
Logs
Not recorded
Trial 3 fail
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund.
Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
Yearly plan began 20 January. 15 February is 26 days later, which is within 30 days, so a full refund under 4.3.
But the question asks which clause gives the amount: 4.3 (with 4.5 applying the yearly rules). The amount is given by 4.3.
**4.3**
Scores
answer@1fail 0.00
expected "4.3", got "Yearly plan began 20 January. 15 February is 26 days later, which is within 30 days, so a full refund under 4.3.\n\nBut the question asks which clause gives the amount: 4.3 (with 4.5 applying the yea…
Trace
Not recorded
Logs
Not recorded
crashpass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
bug
Scores
answer@1pass 1.00
expected "bug", got "bug"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
bug
Scores
answer@1pass 1.00
expected "bug", got "bug"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
bug
Scores
answer@1pass 1.00
expected "bug", got "bug"
Trace
Not recorded
Logs
Not recorded
dark-modepass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
invoicepass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
billing
Scores
answer@1pass 1.00
expected "billing", got "billing"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
billing
Scores
answer@1pass 1.00
expected "billing", got "billing"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
billing
Scores
answer@1pass 1.00
expected "billing", got "billing"
Trace
Not recorded
Logs
Not recorded
loginpass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
account
Scores
answer@1pass 1.00
expected "account", got "account"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
account
Scores
answer@1pass 1.00
expected "account", got "account"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
account
Scores
answer@1pass 1.00
expected "account", got "account"
Trace
Not recorded
Logs
Not recorded
praisepass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
refund-added-linepass (3/3)
Tags: hard
Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): 20.00
- Setup fee: 40.00
The customer cancels at the end of 20 June.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
answer@1pass 1.00
expected "4000", got "4000"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): 20.00
- Setup fee: 40.00
The customer cancels at the end of 20 June.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
answer@1pass 1.00
expected "4000", got "4000"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): 20.00
- Setup fee: 40.00
The customer cancels at the end of 20 June.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
answer@1pass 1.00
expected "4000", got "4000"
Trace
Not recorded
Logs
Not recorded
refund-creditpass (2/3)
Tags: hard
Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
…Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
- Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00
- Archive add-on: 15.00
- Setup fee: 30.00
The customer cancels at the end of 12 April.
How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
answer@1pass 1.00
expected "3060", got "3060"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
…Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
- Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00
- Archive add-on: 15.00
- Setup fee: 30.00
The customer cancels at the end of 12 April.
How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
answer@1pass 1.00
expected "3060", got "3060"
Trace
Not recorded
Logs
Not recorded
Trial 3 fail
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
…Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
- Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00
- Archive add-on: 15.00
- Setup fee: 30.00
The customer cancels at the end of 12 April.
How much is refunded? Reply with the amount in pence, digits only.
Output
3160
Scores
answer@1fail 0.00
expected "3060", got "3160"
Trace
Not recorded
Logs
Not recorded
refund-currencypass (3/3)
Tags: hard
Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July (31 days):
- Business plan, 4 seats × 25.00: 100.00
- Priority support: 31.00
- Setup fee: 50.00
The customer cancels at the end of 9 July.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
answer@1pass 1.00
expected "8460", got "8460"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July (31 days):
- Business plan, 4 seats × 25.00: 100.00
- Priority support: 31.00
- Setup fee: 50.00
The customer cancels at the end of 9 July.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
answer@1pass 1.00
expected "8460", got "8460"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July (31 days):
- Business plan, 4 seats × 25.00: 100.00
- Priority support: 31.00
- Setup fee: 50.00
The customer cancels at the end of 9 July.
How much is refunded? Reply with the amount in euro cents, digits only.
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund.
Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
answer@1pass 1.00
expected "4.4", got "4.4"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund.
Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
answer@1pass 1.00
expected "4.4", got "4.4"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund.
Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
answer@1pass 1.00
expected "4.4", got "4.4"
Trace
Not recorded
Logs
Not recorded
clause-upgradefail (0/3)
Tags: hard
0 of 3 trials passed, needs 2; answer: expected "4.3", got "Yearly plan began 20 January; 15 February is 26 days later, which is within 30 days, so a full refund applies under 4.3.\n\n**4.3**"
Trial 1 fail
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund.
Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
Yearly plan began 20 January. 15 February is 26 days later, which is within 30 days, so a full refund under 4.3.
But the question asks which clause gives the amount: 4.3 (with 4.5 applying the start date). The amount is given by 4.3.
**4.3**
Scores
answer@1fail 0.00
expected "4.3", got "Yearly plan began 20 January. 15 February is 26 days later, which is within 30 days, so a full refund under 4.3.\n\nBut the question asks which clause gives the amount: 4.3 (with 4.5 applying the sta…
Trace
Not recorded
Logs
Not recorded
Trial 2 fail
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund.
Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
The yearly plan began on 20 January (clause 4.5). On 15 February, 26 days have passed (11 days left in January after the 20th, plus 15 days in February), which is within 30 days. A full refund therefore applies under clause 4.3.
**4.3**
Scores
answer@1fail 0.00
expected "4.3", got "The yearly plan began on 20 January (clause 4.5). On 15 February, 26 days have passed (11 days left in January after the 20th, plus 15 days in February), which is within 30 days. A full refund theref…
Trace
Not recorded
Logs
Not recorded
Trial 3 fail
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund.
Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
Yearly plan began 20 January; 15 February is 26 days later, which is within 30 days, so a full refund applies under 4.3.
**4.3**
Scores
answer@1fail 0.00
expected "4.3", got "Yearly plan began 20 January; 15 February is 26 days later, which is within 30 days, so a full refund applies under 4.3.\n\n**4.3**"
Trace
Not recorded
Logs
Not recorded
crashpass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
bug
Scores
answer@1pass 1.00
expected "bug", got "bug"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
bug
Scores
answer@1pass 1.00
expected "bug", got "bug"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
bug
Scores
answer@1pass 1.00
expected "bug", got "bug"
Trace
Not recorded
Logs
Not recorded
dark-modepass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
invoicepass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
billing
Scores
answer@1pass 1.00
expected "billing", got "billing"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
billing
Scores
answer@1pass 1.00
expected "billing", got "billing"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
billing
Scores
answer@1pass 1.00
expected "billing", got "billing"
Trace
Not recorded
Logs
Not recorded
loginpass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
account
Scores
answer@1pass 1.00
expected "account", got "account"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
account
Scores
answer@1pass 1.00
expected "account", got "account"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
account
Scores
answer@1pass 1.00
expected "account", got "account"
Trace
Not recorded
Logs
Not recorded
praisepass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
refund-added-linepass (3/3)
Tags: hard
Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): 20.00
- Setup fee: 40.00
The customer cancels at the end of 20 June.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
answer@1pass 1.00
expected "4000", got "4000"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): 20.00
- Setup fee: 40.00
The customer cancels at the end of 20 June.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
answer@1pass 1.00
expected "4000", got "4000"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): 20.00
- Setup fee: 40.00
The customer cancels at the end of 20 June.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
answer@1pass 1.00
expected "4000", got "4000"
Trace
Not recorded
Logs
Not recorded
refund-creditpass (3/3)
Tags: hard
Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
…Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
- Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00
- Archive add-on: 15.00
- Setup fee: 30.00
The customer cancels at the end of 12 April.
How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
answer@1pass 1.00
expected "3060", got "3060"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
…Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
- Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00
- Archive add-on: 15.00
- Setup fee: 30.00
The customer cancels at the end of 12 April.
How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
answer@1pass 1.00
expected "3060", got "3060"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
…Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
- Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00
- Archive add-on: 15.00
- Setup fee: 30.00
The customer cancels at the end of 12 April.
How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
answer@1pass 1.00
expected "3060", got "3060"
Trace
Not recorded
Logs
Not recorded
refund-currencypass (3/3)
Tags: hard
Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July (31 days):
- Business plan, 4 seats × 25.00: 100.00
- Priority support: 31.00
- Setup fee: 50.00
The customer cancels at the end of 9 July.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
answer@1pass 1.00
expected "8460", got "8460"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July (31 days):
- Business plan, 4 seats × 25.00: 100.00
- Priority support: 31.00
- Setup fee: 50.00
The customer cancels at the end of 9 July.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
answer@1pass 1.00
expected "8460", got "8460"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July (31 days):
- Business plan, 4 seats × 25.00: 100.00
- Priority support: 31.00
- Setup fee: 50.00
The customer cancels at the end of 9 July.
How much is refunded? Reply with the amount in euro cents, digits only.
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund.
Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
answer@1pass 1.00
expected "4.4", got "4.4"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund.
Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
answer@1pass 1.00
expected "4.4", got "4.4"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund.
Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
answer@1pass 1.00
expected "4.4", got "4.4"
Trace
Not recorded
Logs
Not recorded
clause-upgradepass (3/3)
Tags: hard
Trial 1 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund.
Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
answer@1pass 1.00
expected "4.3", got "4.3"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund.
Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
answer@1pass 1.00
expected "4.3", got "4.3"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund.
Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
answer@1pass 1.00
expected "4.3", got "4.3"
Trace
Not recorded
Logs
Not recorded
crashpass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
bug
Scores
answer@1pass 1.00
expected "bug", got "bug"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
bug
Scores
answer@1pass 1.00
expected "bug", got "bug"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
bug
Scores
answer@1pass 1.00
expected "bug", got "bug"
Trace
Not recorded
Logs
Not recorded
dark-modepass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
invoicepass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
billing
Scores
answer@1pass 1.00
expected "billing", got "billing"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
billing
Scores
answer@1pass 1.00
expected "billing", got "billing"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
billing
Scores
answer@1pass 1.00
expected "billing", got "billing"
Trace
Not recorded
Logs
Not recorded
loginpass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
account
Scores
answer@1pass 1.00
expected "account", got "account"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
account
Scores
answer@1pass 1.00
expected "account", got "account"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
account
Scores
answer@1pass 1.00
expected "account", got "account"
Trace
Not recorded
Logs
Not recorded
praisepass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
refund-added-linepass (3/3)
Tags: hard
Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): 20.00
- Setup fee: 40.00
The customer cancels at the end of 20 June.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
answer@1pass 1.00
expected "4000", got "4000"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): 20.00
- Setup fee: 40.00
The customer cancels at the end of 20 June.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
answer@1pass 1.00
expected "4000", got "4000"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): 20.00
- Setup fee: 40.00
The customer cancels at the end of 20 June.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
answer@1pass 1.00
expected "4000", got "4000"
Trace
Not recorded
Logs
Not recorded
refund-creditpass (3/3)
Tags: hard
Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
…Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
- Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00
- Archive add-on: 15.00
- Setup fee: 30.00
The customer cancels at the end of 12 April.
How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
answer@1pass 1.00
expected "3060", got "3060"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
…Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
- Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00
- Archive add-on: 15.00
- Setup fee: 30.00
The customer cancels at the end of 12 April.
How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
answer@1pass 1.00
expected "3060", got "3060"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
…Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
- Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00
- Archive add-on: 15.00
- Setup fee: 30.00
The customer cancels at the end of 12 April.
How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
answer@1pass 1.00
expected "3060", got "3060"
Trace
Not recorded
Logs
Not recorded
refund-currencypass (3/3)
Tags: hard
Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July (31 days):
- Business plan, 4 seats × 25.00: 100.00
- Priority support: 31.00
- Setup fee: 50.00
The customer cancels at the end of 9 July.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
answer@1pass 1.00
expected "8460", got "8460"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July (31 days):
- Business plan, 4 seats × 25.00: 100.00
- Priority support: 31.00
- Setup fee: 50.00
The customer cancels at the end of 9 July.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
answer@1pass 1.00
expected "8460", got "8460"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July (31 days):
- Business plan, 4 seats × 25.00: 100.00
- Priority support: 31.00
- Setup fee: 50.00
The customer cancels at the end of 9 July.
How much is refunded? Reply with the amount in euro cents, digits only.
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund.
Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
answer@1pass 1.00
expected "4.4", got "4.4"
Trace
Not recorded
Logs
Not recorded
Trial 2 fail
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund.
Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
answer@1fail 0.00
expected "4.4", got "4.3"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 848 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 10 March and upgraded to a yearly plan on 5 April. On 6 May he asks for a refund.
Which clause gives the amount of his refund? Reply with the clause number only, for example 4.2.
Output
4.4
Scores
answer@1pass 1.00
expected "4.4", got "4.4"
Trace
Not recorded
Logs
Not recorded
clause-upgradepass (3/3)
Tags: hard
Trial 1 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund.
Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
answer@1pass 1.00
expected "4.3", got "4.3"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund.
Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
answer@1pass 1.00
expected "4.3", got "4.3"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refun…Show all 859 charactersShow less
Refund rules of a file-sharing service (extract):
4.1 Monthly plan: a refund is made within 7 days of the first payment, if the customer used less than 1 GB.
4.2 Monthly plan: after 7 days, or after 1 GB of use, there is no refund. The plan runs to the end of its month.
4.3 Yearly plan: a full refund is made within 30 days of the first payment, whatever the use.
4.4 Yearly plan: after 30 days, the refund is the unused full months, minus a fee of 10% of the yearly price.
4.5 When a customer upgrades from a monthly plan to a yearly plan, the yearly rules apply from the day of the upgrade, as if the yearly plan began that day.
A customer paid for a monthly plan on 2 January and upgraded to a yearly plan on 20 January. On 15 February she asks for a refund.
Which clause gives the amount of her refund? Reply with the clause number only, for example 4.2.
Output
4.3
Scores
answer@1pass 1.00
expected "4.3", got "4.3"
Trace
Not recorded
Logs
Not recorded
crashpass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
bug
Scores
answer@1pass 1.00
expected "bug", got "bug"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
bug
Scores
answer@1pass 1.00
expected "bug", got "bug"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a ques…Show all 378 charactersShow less
Category of this support ticket: "The app crashes every time I open a PDF."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
bug
Scores
answer@1pass 1.00
expected "bug", got "bug"
Trace
Not recorded
Logs
Not recorded
dark-modepass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question …Show all 373 charactersShow less
Category of this support ticket: "Can you add a dark mode to the app?"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
invoicepass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
billing
Scores
answer@1pass 1.00
expected "billing", got "billing"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
billing
Scores
answer@1pass 1.00
expected "billing", got "billing"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, …Show all 384 charactersShow less
Category of this support ticket: "Please send me an invoice for my last payment."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
billing
Scores
answer@1pass 1.00
expected "billing", got "billing"
Trace
Not recorded
Logs
Not recorded
loginpass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
account
Scores
answer@1pass 1.00
expected "account", got "account"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
account
Scores
answer@1pass 1.00
expected "account", got "account"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a…Show all 383 charactersShow less
Category of this support ticket: "I cannot log in because I forgot my password."
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
account
Scores
answer@1pass 1.00
expected "account", got "account"
Trace
Not recorded
Logs
Not recorded
praisepass (3/3)
Tags: easy
Trial 1 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a questi…Show all 376 charactersShow less
Category of this support ticket: "Thanks, the new search is really fast!"
billing: a charge, a refund, an invoice, a price or a payment method.
account: login, password, profile, team members or the plan.
bug: an error, a crash or a wrong result in the app.
other: praise, a feature request, a question about how to use the app, or anything else.
Reply with the category only.
Output
other
Scores
answer@1pass 1.00
expected "other", got "other"
Trace
Not recorded
Logs
Not recorded
refund-added-linepass (2/3)
Tags: hard
Trial 1 fail
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): 20.00
- Setup fee: 40.00
The customer cancels at the end of 20 June.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
9000
Scores
answer@1fail 0.00
expected "4000", got "9000"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): 20.00
- Setup fee: 40.00
The customer cancels at the end of 20 June.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
answer@1pass 1.00
expected "4000", got "4000"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): …Show all 440 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period of the line. A setup fee is not refunded.
Invoice in EUR, 1 to 30 June:
- Team plan, 1 to 30 June (30 days): 90.00
- Extra storage, 11 to 30 June (20 days): 20.00
- Setup fee: 40.00
The customer cancels at the end of 20 June.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
4000
Scores
answer@1pass 1.00
expected "4000", got "4000"
Trace
Not recorded
Logs
Not recorded
refund-creditpass (3/3)
Tags: hard
Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
…Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
- Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00
- Archive add-on: 15.00
- Setup fee: 30.00
The customer cancels at the end of 12 April.
How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
answer@1pass 1.00
expected "3060", got "3060"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
…Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
- Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00
- Archive add-on: 15.00
- Setup fee: 30.00
The customer cancels at the end of 12 April.
How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
answer@1pass 1.00
expected "3060", got "3060"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
…Show all 512 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: net price × unused days ÷ days of the period. The net price is the price minus the credit that the invoice applied to the line. A setup fee is not refunded.
Invoice in GBP, 1 to 30 April (30 days):
- Team plan, 6 seats × 8.00: 48.00, credit applied: 12.00
- Archive add-on: 15.00
- Setup fee: 30.00
The customer cancels at the end of 12 April.
How much is refunded? Reply with the amount in pence, digits only.
Output
3060
Scores
answer@1pass 1.00
expected "3060", got "3060"
Trace
Not recorded
Logs
Not recorded
refund-currencypass (3/3)
Tags: hard
Trial 1 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July (31 days):
- Business plan, 4 seats × 25.00: 100.00
- Priority support: 31.00
- Setup fee: 50.00
The customer cancels at the end of 9 July.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
answer@1pass 1.00
expected "8460", got "8460"
Trace
Not recorded
Logs
Not recorded
Trial 2 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July (31 days):
- Business plan, 4 seats × 25.00: 100.00
- Priority support: 31.00
- Setup fee: 50.00
The customer cancels at the end of 9 July.
How much is refunded? Reply with the amount in euro cents, digits only.
Output
8460
Scores
answer@1pass 1.00
expected "8460", got "8460"
Trace
Not recorded
Logs
Not recorded
Trial 3 pass
Input
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July …Show all 511 charactersShow less
Refund rule: when a customer cancels, each line of the invoice is refunded for its unused whole days: price × unused days ÷ days of the period. A setup fee is not refunded. The refund is paid in euros at 1 USD = 0.91 EUR, and rounded down to the cent once, on the total.
Invoice in USD, 1 to 31 July (31 days):
- Business plan, 4 seats × 25.00: 100.00
- Priority support: 31.00
- Setup fee: 50.00
The customer cancels at the end of 9 July.
How much is refunded? Reply with the amount in euro cents, digits only.