How PenLeap Grades Short Answers Without Hallucinating Marks

Ask a raw LLM to "grade this Class 8 answer out of 5" and it will confidently hand back a number it cannot justify — sometimes 4, sometimes 2, for the same answer on two runs. On PenLeap, our in-house exam-prep product for students 11 and up, that variance is unacceptable: a wrong mark erodes a student's trust and a parent's. So we do not ask the model for a score. We ask it to check the answer against an explicit rubric, clause by clause, and we gate every mark behind a confidence threshold. This post is the grading pipeline, the human-in-loop rules, and how we audit it. ## The Answer in 60 Words We never ask the model for a single lump score. We give it a rubric broken into scoring clauses, ask it to mark each clause as met, partially met, or not met, with a quoted span from the student's answer as evidence. Each clause carries a confidence value. Low-confidence or borderline totals route to a human. We audit a 5% sample weekly to catch drift. ## Why This Matters Now (June 2025) Edtech is racing to add AI grading, and most of it hallucinates marks because it treats grading as a generation task instead of a verification task. The stakes are real: a student told they scored 2 out of 5 on a correct answer stops trusting the platform, and a parent paying fees notices fast. As AI grading spreads across Indian coaching and school products in 2025, the differentiator is not "we use AI to grade" — every competitor says that. It is "our marks are defensible, every one traceable to a rubric clause." That is an engineering choice, and it is the one we made. ## Why Do LLMs Hallucinate Marks? An LLM hallucinates a mark when you ask it to produce a score directly, because scoring is an opinion it generates rather than a fact it checks. Reframing grading as clause-by-clause verification removes most of the variance. ## How Does Rubric-Anchored Scoring Work? Rubric-anchored scoring breaks a question's mark scheme into discrete, checkable clauses and grades each one independently. The total is the sum of clause results, not a number the model invents. Take a Class 8 science question worth 3 marks: "Explain why a desert plant has thick stems." The rubric clauses might be: (1) states the stem stores water — 1 mark; (2) links storage to scarce rainfall — 1 mark; (3) uses correct terminology like "adaptation" — 1 mark. The model checks each clause against the student's answer, quotes the supporting span, and returns met / partial / not-met with a confidence value. Our colleague Hrishikesh, who architected the engine, calls this "turning a judgement into a checklist" — and checklists are auditable. ## What Are the Confidence Thresholds? We split every graded answer into three zones. The thresholds are tuned per subject and reviewed monthly against the audit sample.

Zone	Condition	Action
Auto-publish (high confidence)	All clauses clear-cut, confidence above threshold, two passes agree	Mark shown to student immediately
Human review (borderline)	Any clause marked "partial," confidence in the middle band, or the two passes disagree	Queued for a human grader before publish
Auto-flag (low confidence)	Model signals low confidence, off-topic answer, or unparseable response	Held; human grades from scratch

The two-pass rule matters. We grade each answer twice with the same rubric in independent calls. If the two passes agree and both clear the confidence bar, we trust the mark. If they disagree — even by one clause — the answer goes to a human. Disagreement between passes is itself the strongest signal that the answer is ambiguous. ## The Grading Pipeline (How We Built It) Here is the pipeline, stage by stage, each with the check that keeps it honest. ## What You'll Need to Build Something Similar ## Common Mistakes (When Grading Goes Wrong) We learned these the hard way on real student data. Four to avoid. First, asking for one lump score "to save tokens." It is a false economy — you save a few tokens and lose every guarantee. Clause-by-clause grading costs more per answer but is the entire reason the marks are defensible. Second, accepting a "met" clause with no quoted evidence. Without the requirement to quote the student's words, the model credits points generously and invents justification. Reject any clause that claims "met" without a span. Third, computing confidence and ignoring it. If borderline marks auto-publish anyway, the routing is decorative. Act on the signal or do not bother computing it. Fourth, never auditing. Model behaviour and prompts drift. Without a weekly re-grade sample, a regression in grading quality can run for weeks before a parent complaint surfaces it. ## When NOT to Auto-Grade Auto-grading is wrong for some question types, and pretending otherwise is how products embarrass themselves. Genuinely open-ended creative writing, where there is no clause-based mark scheme, should not be auto-scored on correctness — at most, assisted with feedback a human signs off. High-stakes terminal exams that determine admissions need a human grader of record regardless of how good the engine is; the engine assists, it does not decide. And any subject where a single answer can be right in ways the rubric author did not foresee needs a generous human-review path. Auto-grade the high-volume, well-structured, formative practice — leave the edge cases to people. ## Real Example: Formative Practice at Scale On PenLeap, the bulk of short-answer grading is formative practice — students drilling Class 6–10 science and English between classes, where fast, trustworthy feedback keeps them going. This is exactly the high-volume, well-structured case auto-grading suits. The clause-level feedback is the quiet win: instead of "3/5," the student sees which specific points they earned and which they missed, quoted back. That is the same real-time, rubric-based feedback pattern we apply to enterprise AI work — see how we think about confidence and verification in our piece on building a RAG support bot with an "I don't know" guardrail. We have built rubric and assessment engines as client work too, including the test platform behind ExamReady, and the broader pattern feeds directly into our AI and automation practice. For the question-generation side that feeds these rubrics, see how PenLeap generates MCQs with verified answer keys. ## Frequently Asked Questions ### Why not just ask the LLM to grade out of 5 directly? Because single-score grading is unstable — the same answer can get different marks on different runs, with no traceable reason. We break the mark scheme into clauses, grade each as met / partial / not-met with quoted evidence, and sum the results. That makes every mark defensible. ### How do you stop the model inventing marks the student didn't earn? We require a quoted span from the student's answer for every clause marked "met." If the model claims a point was earned but cannot quote the supporting words, the grade is rejected and re-run. Evidence requirements force the model to be honest. ### What is the two-pass grading rule? We grade each answer twice in independent calls using the same rubric. If both passes agree and clear the confidence threshold, we trust the mark. If they disagree on any clause, the answer routes to a human. Disagreement is a strong signal the answer is ambiguous. ### How much human involvement is left after automation? Borderline and low-confidence grades route to humans, plus we re-grade a random 5% sample of auto-published marks weekly. For high-volume formative practice, most marks auto-publish; high-stakes or open-ended work always keeps a human grader of record. ### What confidence threshold do you use? It is tuned per subject and reviewed monthly against the audit sample, not a single fixed number. The principle is consistent: clear-cut, high-confidence, two-passes-agree answers auto-publish; anything borderline or low-confidence routes to a human before a student sees it. ### Can this approach grade essays or only short answers? It suits short, structured answers with a clause-based mark scheme. Open-ended creative writing has no fixed clauses to check, so we use it for feedback that a human signs off rather than for an auto-decided correctness mark. Match the method to the question type. ### How do you catch grading drift over time? A subject expert re-grades a random 5% of auto-published marks every week, and we track the agreement rate on a dashboard. A dip triggers a threshold review and prompt fix before any regression spreads. Without this sampling, drift can run unnoticed for weeks.