Research / Open benchmark, version 0.2.0
Pay-gap reasoning
The figures the EU Pay Transparency Directive asks an employer to report, worked from a raw payroll extract. Every reference answer is computed by code, and every line is graded by code.
What it asks
Each task is one invented employer: a payroll extract of 21 to 50 rows and these rules. The model reports eight figures as JSON.
- Scope: include a worker only if they were employed on 31 December 2025: start_date on or before 2025-12-31, and end_date empty or after 2025-12-31.
- Figures compare women (F) and men (M). Workers whose gender is recorded as X are left out of every figure, including the count.
- Hourly pay = (annual_base_eur + annual_variable_eur) / (weekly_hours x 52).
- A gap is (men's figure - women's figure) / men's figure x 100. A positive gap means men are paid more.
- The mean and median hourly gaps use the mean and the median hourly pay of each gender. The median of an even count is the mean of the two middle values.
- Variable pay gap: compare the mean annual_variable_eur of the women and of the men who received variable pay (annual_variable_eur above 0).
- Variable pay recipients: the share of in-scope women, and of in-scope men, who received variable pay, as percentages.
- Quartile pay bands: sort in-scope workers by hourly pay, lowest first, ties broken by employee_id. With n workers, band k (k = 1 to 4) holds positions floor((k - 1) x n / 4) + 1 to floor(k x n / 4). Report the share of women in each band as a percentage, band 1 first.
- Category gaps: the mean hourly gap within each category of workers.
- A joint pay assessment is triggered for every category whose mean hourly gap, rounded to one decimal, is 5.0 or more in either direction.
A task, in part
The first rows of task pg-001, as the model sees them. Rows it must leave out look like every other row.
employee_id,gender,category,weekly_hours,annual_base_eur,annual_variable_eur,start_date,end_date
E434,M,Engineering L1,40,44910,3090,2022-01-23,
E417,M,Marketing L2,40,59660,2240,2023-03-05,
E422,F,Engineering L3,40,70000,0,2022-04-25,
E454,X,Engineering L3,20,41100,3180,2020-08-07,
E449,M,Operations L2,32,37710,0,2021-03-15,
E415,M,Marketing L2,20,27250,1060,2015-09-26,
E424,F,Support L2,40,34440,0,2014-11-27,
E439,F,Engineering L1,24,27960,3070,2021-03-08,
... 34 more rowsHow it is graded
The rubric for the same task. A task’s score is the weighted share of lines the answer gets right.
| Rubric line | Checked as | Weight |
|---|---|---|
| Counts 39 workers in scope | Exact value | 2 |
| Mean hourly gap is 8.6% | Number, ± 0.1 | 3 |
| Median hourly gap is 0.6% | Number, ± 0.1 | 2 |
| Variable pay gap is -16.6% | Number, ± 0.1 | 2 |
| 56.5% of women and 62.5% of men received variable pay | Named figures, ± 0.1 | 1 |
| Women's share by quartile band is 77.8, 40, 70, 50 | Four numbers, ± 0.1 each | 2 |
| Engineering L1 gap is -2% | Named figures, ± 0.1 | 1 |
| Engineering L3 gap is 6.5% | Named figures, ± 0.1 | 1 |
| Marketing L2 gap is 16.2% | Named figures, ± 0.1 | 1 |
| Operations L2 gap is 0.7% | Named figures, ± 0.1 | 1 |
| Support L2 gap is 18.3% | Named figures, ± 0.1 | 1 |
| Joint assessment for exactly: Engineering L3, Marketing L2, Support L2 | Exactly this set | 3 |
Across the set
441 rubric lines over 40 tasks, and the cases that catch a careless answer.
| What the lines check | Lines | Weight |
|---|---|---|
| Who is in scope | 40 | 80 |
| Mean hourly gap | 40 | 120 |
| Median hourly gap | 40 | 80 |
| Variable pay | 80 | 120 |
| Quartile pay bands | 40 | 80 |
| Gaps by category | 161 | 161 |
| Joint assessment trigger | 40 | 120 |
| Rows that test the scope rules | Across the set |
|---|---|
| Left before 31 December 2025 | 66 |
| Joined in 2026 | 15 |
| Leaving in 2026, so still in scope | 17 |
| Gender recorded as X | 34 |
| Part-time, so hourly pay matters | 476 |
What is held back
The task set, so it cannot leak into a model’s training data.
Every report prints the hash of the tasks file it ran, so a score always points at one exact set:
sha256 3e6a7b239ce3259eea162cd4cde8fac2f4e495379bbf5477b28298415a360a59
A second, independent program recomputes every reference answer from the text of each task before the set is used. The rules are in the method.