Who Grades the Grader? Why Examiner Consistency Is Assessment’s Hidden Fault Line 

Nirvana Intellect — Blog Series, Academia Track, Part 2 

Part 1 of this series looked at why a score alone isn’t evidence. This one goes a layer deeper: even when an institution has a rubric, a trained examiner, and a proper marking process, there’s still no guarantee that two qualified people looking at the same answer will land on the same mark. If the person doing the grading isn’t consistent — with themselves, and with every other examiner on the panel — the rubric underneath them barely matters. 

Marking is judged, not just applied 

It’s tempting to think of a rubric as a piece of code: input an answer, output a mark. In practice, applying a rubric to a real constructed response is an act of professional judgment, and judgment drifts. Examination regulators have studied this directly. Analysis by England’s exams regulator Ofqual into how markers handle borderline cases found that, in reviews of contested marks, examiners were consistent with the rules a clear majority of the time — but a meaningful share of disputed results still came down to legitimate differences of professional opinion rather than a clean marking error (gov.uk). That distinction matters: an institution can’t fix a difference of opinion the way it fixes an error, which is exactly why the review process needs to be built to catch drift before results are published, not just after they’re appealed. 

Standardisation helps — but it isn’t a fix on its own 

Most exam boards run some form of standardisation: markers review sample scripts together, or online, before live marking begins, to align on how the rubric should be applied. It works, to a point. Ofqual’s own research programme into marking quality found that online standardisation — now the dominant method, having largely replaced in-person meetings — leaves a meaningful share of examiners unconvinced it’s actually calibrating them consistently, and identified specific practices (like personal feedback calls after approval) that improve it but aren’t universally used (gov.uk). In other words: standardisation is necessary, but institutions that treat it as a one-time checkbox before marking starts are missing where consistency actually breaks down — over the course of marking, not just at the start of it. 

Double marking helps, but not the way most people assume 

The intuitive fix for inconsistency is to have two examiners mark everything. Cambridge Assessment’s research division has studied this in detail, comparing outcomes when a second marker sees the first marker’s annotations versus marking “blind.” Their findings complicate the intuitive fix: agreement is higher when the second marker can see the first marker’s annotations, but that’s ambiguous evidence — it could mean the annotations genuinely help calibrate the second judgment, or it could mean the second marker is simply anchoring to the first one’s opinion rather than forming an independent view (Cambridge Assessment). Blind double marking is more rigorous, but far more expensive to run at scale — which is exactly why most institutions don’t do it, and exactly why the results that most need a second independent opinion — the borderline, contested ones — are often the ones getting the least scrutiny. 

What this means for institutional practice 

None of this is an argument against human judgment in grading — constructed-response evaluation genuinely needs it. It’s an argument for treating marker consistency as something that has to be actively managed, not assumed once a rubric exists and training has happened. In practice that means: 

• Calibration that continues through the marking window, not just before it — spot-checking examiner agreement as marking progresses, not only in a pre-marking standardisation session. 
• Routing borderline and hard-to-mark responses for a second, ideally blind, review — rather than applying the same single-marker process to every script regardless of how close it sits to a grade boundary. 
• Recording the rubric criteria applied to each response, not just the final numeric mark — so a moderator or appeals panel can see the reasoning, not just the outcome. 
• Treating every professional testing standard as the floor, not the ceiling — the Standards for Educational and Psychological Testing exist precisely because validity has to be actively demonstrated, not assumed from having a process in place at all. 

Where AI genuinely helps — and where the accountability still sits 

This is also where a governed AI layer earns its place, rather than just adding another automated tool to the pile. An AI system can flag when a particular examiner’s marks are statistically drifting from the panel average, surface borderline responses for mandatory second review, and draft a consistency-checked first-pass score against the rubric for a human moderator to confirm. What it shouldn’t do is quietly replace the second opinion altogether. UNESCO’s own guidance on AI in higher education frames this as fundamentally a governance question, not a technology question — the tool has to sit inside a policy that keeps a qualified person accountable for the outcome (UNESCO IESALC). See how Nirvana Intellect builds this into evaluation and capability intelligence → 

The real question institutions should be asking 

Not “do we have a rubric,” but “can we show that two examiners applying it independently would land in the same place, and if they wouldn’t, do we know before the result is published or only after a parent calls?” That’s the difference between a marking process and a governed one. Explore Nirvana Intellect’s approach to academic assessment governance → 

That’s Part 2. Next in this track: what an institution actually needs on hand the day a result is formally challenged — and why most don’t have it.