When an accreditor or a governing body now asks a school to evidence teaching effectiveness, the easy answers arrive first: the satisfaction scores, the pass rates, the peer-observation forms, the awards. Each has its place. None of them answers the one question that matters. How much did the student actually move while in our hands?
That is what a learning-gain metric tries to capture. It shifts the lens from the quality of the teaching to the distance the learner travelled, and it changes the conversation in a department from “was the teacher good” to “did learning happen, and by how much”. This Monday’s reflection is on how to build such a metric well, because a learning-gain number built carelessly does more harm than no number at all. It rewards easy intakes, punishes ambitious teaching, and invites the quiet corruption of standards. Built well, it becomes the most honest instrument an academic leader has.
What learning gain is, and why it is fairer than the alternatives
A mark at the end of a course tells you where a student finished. It does not tell you where they began, and so it cannot tell you what the teaching contributed. A learning-gain metric is built on a simple discipline: measure the same construct at entry and at exit, on the same scale, and report the difference.
The reason this is fairer than raw exit marks is the reason it is harder to do. Two teachers do not inherit the same students. A cohort that walks in already strong will post high exit marks whatever happens in the room. A cohort that walks in weak may learn a great deal and still finish below them. Raw marks reward the lucky intake. Learning gain, done properly, rewards the movement.
The formulae, and what each is for
There are three respectable ways to express a gain, and a serious school will use more than one because each answers a different question.
1. Normalised gain (Hake’s g) answers: of the room the student had to improve, how much did they actually cover?
g = (post − pre) / (max − pre)
where pre and post are the scores on the same instrument and max is the maximum attainable (often 100). Dividing by the room left to grow is what makes the measure fair across different starting points. The convention, from Hake’s large physics study, reads the bands as: g of 0.7 or above is a high gain, 0.3 to 0.7 a medium gain, and below 0.3 a low gain. For a whole class, compute it from the class means, written ⟨g⟩.
2. Effect size (Cohen’s d, or Hedges’ g for small classes) answers: how large was the learning movement, in a language that travels across very different courses and into the research literature?
d = (mean_post − mean_pre) / SD_pooled
where SD_pooled is the pooled standard deviation of the two sets of scores. For the small cohorts typical of our classrooms, apply the Hedges correction, which simply multiplies d by a factor slightly below one to remove the small-sample bias. The familiar bands are 0.2 small, 0.5 medium, 0.8 large, though in pre-and-post designs values well above one are common and should be read with care.
3. Value-added answers, at programme or institutional level: did this student do better, or worse, than someone with their starting profile should have been expected to?
residual = actual_exit − predicted_exit
where predicted_exit comes from a model (in practice a regression) that estimates each student’s expected exit performance from prior attainment and other intake characteristics. The residual, the part of the result the intake does not explain, is treated as the teaching contribution. Value-added is the hardest of the three to game, because it explicitly adjusts for who walked in the door, and it is the right tool when you must compare across sections, departments or years.
A worked example: why normalisation changes the verdict
Take two sections of the same first-year course, taught by two colleagues, assessed on the same blueprinted pre-test and post-test out of 100.
Section A enters at a mean of 42 and exits at 71. The raw gain is 29 marks. Section B enters at a mean of 55 and exits at 78. The raw gain is 23 marks.
On raw gain, Section A looks clearly stronger, by six marks. But Section B’s students had less room to grow. Normalise:
Section A: g = (71 − 42) / (100 − 42) = 29 / 58 = 0.50
Section B: g = (78 − 55) / (100 − 55) = 23 / 45 = 0.51
The two teachers achieved, in truth, almost identical learning gains, both squarely in the medium band. The raw-gain ranking was an artefact of the intake, not a measure of the teaching. This single sum is the reason normalisation is not a technical nicety but a matter of fairness to faculty.
Now express Section A as an effect size. Suppose the pre-test standard deviation is 12 marks and the post-test 13. The pooled standard deviation is about 12.5, so:
d = (71 − 42) / 12.5 = 2.3 (large; Hedges g ≈ 2.3 for a class of about thirty)
A large effect, consistent with the medium normalised gain, and now stated in a form a journal reviewer or an external examiner will recognise.
Finally, value-added for one student in Section A. The model, fitted on prior attainment across the cohort, predicts an exit score of 65 for a student with her entry profile. She scores 74. Her value-added residual is +9: she gained nine marks more than her starting profile predicted. Aggregate those residuals across a teacher’s students and you have a measure of contribution that the intake cannot explain away.
The guards: where most attempts fail
The guards decide whether a learning-gain programme is worth anything, and they are usually the thinnest part of it. Six matter most.
Use the same construct, not the same paper. If you administer the identical test twice, part of the “gain” is simply familiarity with the questions. Use parallel forms of equal difficulty, or equate the two papers statistically, so that the rise reflects learning and not recall of the items.
Keep transfer items. Reserve a portion of the post-test for problems set in an unseen context. If students improve on the rehearsed format but not on the transfer items, you have measured teaching to the test, not understanding. The gap between the two is itself a diagnostic.
Track attrition honestly. A gain computed only on the students who sat both tests flatters any course that quietly shed its weakest learners between entry and exit. Always record who is missing from the post-test, and report the gain alongside the attrition, not on the survivors alone.
Check the instrument’s reliability. The pre and post must measure the same thing dependably. A coefficient such as Cronbach’s alpha on each administration tells you whether the instrument is stable enough to carry the weight you are putting on it.
Mind small numbers. In a class of twenty-five, a single absent or distracted student swings the mean. Aggregate across sections or across terms before drawing any conclusion about a teacher, and treat one course’s gain as a signal, never a sentence.
Decouple gain from grades and from reward. The most important guard of all. The moment a high gain raises a teacher’s appraisal, and a low one threatens it, the incentive to inflate the post-test or depress the pre-test appears. Keep the pre-test ungraded and low-stakes, keep the metric developmental wherever you can, and ensure that lowering standards can never raise the number that touches the teacher’s career.
An implementation rubric
The honest way to introduce learning gain is to treat it, like every other academic process, as something that matures over years rather than arrives in a memo. The rubric below uses the AcademicOS five-level maturity ladder. Locate the school on each row, and the next column is the year’s agenda.
| Component | 1 Absent | 2 Ad hoc | 3 Defined | 4 Consistent | 5 Self-improving |
|---|---|---|---|---|---|
| Baseline (pre-test) | No entry measure; exit marks only | One or two enthusiasts run a pre-test | A pre-test exists for named courses | Every core course runs a low-stakes, ungraded pre-test | Baselines are equated across years and trended |
| Instrument | End-of-term exam reused | Ad hoc quiz, untested | Items mapped to outcomes and blueprinted | Parallel forms with transfer items, reliability checked | Item bank refreshed from data each cycle |
| Metric | Raw marks compared | Raw gain quoted | Normalised gain computed per course | Normalised gain plus effect size, reported by section | Value-added modelling across the programme |
| Guards | None | Aware of the risks | Attrition and test-familiarity noted | Decoupled from grades; biases managed; attrition reported | Guards audited; gaming attempts visible and addressed |
| Cadence | Once, at the end | Pre and post only | Pre, post and one mid-point check | Week-one, mid-point and exit, used to intervene in-term | Trajectory feeds an early-warning loop |
| Use | Filed | Discussed informally | Seen by the teacher | Reviewed with mentor; informs course redesign | Closes the loop: a documented change every cycle |
Two design rules sit above the table. First, build the cadence, not just the bookend. A week-one diagnostic, a mid-point check and an exit measure turn a single end-of-course verdict into a trajectory, so a flat or falling segment surfaces in week six when you can still act, not in the results meeting when you cannot. That is what makes learning gain a lead indicator rather than a post-mortem. Second, decide at the outset whether the metric is developmental or evaluative, and say so plainly, because the instant it becomes high-stakes the incentive to corrupt the baseline appears, and the cleanest measurement in the world will not survive it.
The Monday morning question
Pick one core course this term and ask its teacher a single question: if we had measured these students at entry, would we be able to show what they gained, or only what they scored? If the answer is “only what they scored”, you have found both the gap and the place to begin. Run one honest pre-test, keep it out of the grade, and compute one normalised gain at the end. The number will be imperfect and the cohort will be small, but you will have started measuring what we move, and not merely what walks through the door.