Adaptive Difficulty in PenLeap: A Lightweight Bandit for Question Selection

On PenLeap, our in-house exam-prep product, the engine that picks a student's next question is a multi-armed bandit, not the heavyweight item-response-theory (IRT) model textbooks reach for first. We chose a bandit deliberately: it cold-starts in a handful of attempts, runs in a few milliseconds per pick, and is simple enough that one engineer can reason about every decision it makes. The goal isn't to estimate a perfect "ability score" — it's to keep each student in the difficulty band where they learn fastest, and to do it cheaply enough to run for tens of thousands of students. This post is the engineering write-up: how the bandit works, how we calibrate difficulty, how we handle the cold-start problem, and how we measure whether any of it actually helps a kid learn. ## The answer in 60 words We model each difficulty band as an "arm" and pick the band a student plays next using a Thompson-sampling bandit. Each arm tracks a Beta distribution over the student's success probability. We steer toward a roughly 70% success rate — hard enough to learn, easy enough to stay motivated. Cold-start uses a population prior; difficulty is calibrated from historical pass rates; learning gain is measured with pre/post deltas. ## Why a bandit, and why this matters now The standard tool for adaptive testing is IRT — model every item's difficulty and discrimination, estimate each student's latent ability, serve the item with maximum information. It's powerful and it's also a lot: you need a large response matrix to fit item parameters, the math is unfriendly to debug at 2 a.m., and it's built for measuring ability precisely (think standardized exams), not for maximising practice value session to session. PenLeap's job is the second one. A multi-armed bandit reframes "which question next" as "which difficulty band gives this student the best learning return right now" — an explore-vs-exploit problem bandits were made for. With LLMs now generating an effectively unlimited pool of fresh questions per band, the scarce resource isn't questions — it's the student's attention. The bandit spends that attention well. ## Multi-armed bandits in one paragraph A multi-armed bandit is the slot-machine problem: you face several machines (arms), each pays out with an unknown probability, and you want to maximise reward over many pulls. The tension is explore (try arms you're unsure about) vs exploit (pull the arm that's paid best so far). In PenLeap, the arms are difficulty bands for a given skill, a "payout" is the student answering correctly and staying engaged, and the bandit balances trying a harder band against staying where the student is succeeding. For the foundational, product-side view of why this changes a student's experience, see our earlier piece on why a free-tier student gets a different question bank — this post is the engine under that experience. ## The design, component by component ## The reward function (where the real design lives) The naive reward — "+1 for correct, 0 for wrong" — quietly optimises for the wrong thing: it pushes the bandit toward the easiest band, where correctness is guaranteed and learning is near zero. Our reward is shaped around the target band instead: | Outcome | Naive reward | PenLeap reward | |---|---|---| | Correct in an easy band | 1.0 (looks great) | low — the student already knew this | | Correct in a target/hard band | 1.0 | high — genuine stretch with success | | Wrong in a band well above level | 0 | mildly penalised, not zero — some struggle is healthy | | Correct but very slow / hint-heavy | 1.0 | discounted — fluency matters, not just the final tick | We fold response time and hint usage into the signal so a slow, hint-assisted correct answer doesn't get treated like fluent mastery. This is the same instinct behind the rubric-anchored, confidence-aware scoring we use elsewhere on PenLeap — covered in how we grade 9,200 Class 10 essays daily without hallucinated marks. ## Calibrating difficulty (so a "band 4" means the same thing for everyone) ## Solving cold-start (the hardest practical problem) A bandit with no data picks badly. A brand-new student has no history, and gating them behind a long diagnostic quiz kills the first session — the one that decides whether they come back. Our approach: This is the same cold-start discipline we apply when launching any new learning surface, including the conversational practice on our voice-AI app TalkDrill, where a new user's first session has to feel calibrated from turn one. The engine here is the kind of work our AI & automation team ships for clients who need adaptive or recommendation logic without a research-team budget. ## Measuring learning gain (not just engagement) Engagement metrics lie. A student can do 200 easy questions, feel great, and learn nothing. The metric that matters is learning gain, and we measure it deliberately: ## When not to use a bandit here Reach for IRT or a Rasch model instead when (a) the output is a high-stakes score that has to be psychometrically defensible — a bandit optimises learning, not measurement; (b) you need to place a brand-new student precisely on day one for a formal report, where a calibrated diagnostic beats a few exploratory questions; or (c) your question pool is tiny and fixed, where there's little to "explore" and a simple difficulty ladder is clearer. And skip adaptive difficulty entirely if your content library is too small to fill five real bands — adaptivity over four questions is theatre. We've talked teams out of a bandit more than once; the honest answer is that a clean fixed ladder beats a half-fed bandit every time. As Vivek, our co-founder, puts it: "We didn't want the smartest algorithm. We wanted the one we could still explain to a worried parent." That constraint shaped every choice above. You can see the product itself at PenLeap, our in-house exam-prep platform. ## Common mistakes (we hit three of these) Symptom: "Students plateau in easy bands and never improve." Cause: rewarding raw correctness. Fix: shape the reward around a target success band so easy wins are worth less than hard ones. Symptom: "New students get demoralising hard questions in session one." Cause: cold-start with a flat prior. Fix: a population prior from same-class students plus aggressive early updates. Symptom: "A broken question wrecked a student's path." Cause: serving freshly generated items at full rate before calibration. Fix: quarantine new questions at a low serve rate until they have enough attempts. Symptom: "Engagement is up but test scores aren't." Cause: optimising the wrong metric. Fix: measure normalized pre/post gain and time-to-mastery, not minutes-in-app. Symptom: "The bandit keeps a student stuck at one band." Cause: not enough exploration once it's confident. Fix: Thompson sampling provides exploration for free, but cap arm confidence so a long correct streak doesn't fully freeze the distribution. ## FAQ ### Why use a multi-armed bandit instead of IRT for adaptive learning? A bandit optimises learning gain per session and cold-starts in a few attempts, while IRT is built to measure ability precisely and needs a large response matrix to fit. For low-stakes, high-frequency practice where you want maximum learning per minute, a bandit is lighter and easier to reason about. For defensible high-stakes scores, prefer IRT or Rasch. ### What does the bandit actually optimise for? Not raw correctness — that traps students in easy questions. It steers toward roughly a 70% success rate, the band where challenge and motivation balance, using a reward that values hard-won correct answers over easy ones and discounts slow, hint-heavy responses. ### How does PenLeap handle a brand-new student with no history? With a population prior. Each difficulty arm is initialised from the average success curve of students in the same class and skill, so a new student starts at a sensible band rather than the easiest one. The first 3–5 answers are weighted heavily so the student's own signal takes over quickly — no separate diagnostic quiz to fail. ### How is question difficulty calibrated? Each question starts with a difficulty tag from its syllabus level, then gets re-calibrated from its empirical pass rate — controlled for the answering students' bands — once it has enough attempts. New questions serve at a low rate until calibrated, and items with random or suspicious pass rates are auto-flagged for human review. ### How do you know adaptive difficulty actually helps? We measure normalized pre/post learning gain and time-to-mastery, and run an A/B against a holdout cohort that gets questions fixed at their class level. The bandit has to beat fixed difficulty on gain per minute, or it isn't worth its complexity. We also watch session-abandon rate as a frustration guardrail. ### Is Thompson sampling hard to implement? No — that's part of why we chose it. Each arm keeps a Beta distribution updated with successes and failures; to pick, you sample once from each arm and choose by the rule you want. There's no epsilon to hand-tune, and the whole pick runs in a few milliseconds. The hard part is the reward shaping and calibration, not the sampler. ### Can this approach work outside education? Yes. The same explore-vs-exploit framing fits content recommendation, dynamic pricing tiers, and onboarding-flow selection. The arms change, but the pattern — small, explainable, cheap to run, easy to A/B — transfers directly to many product decisions where you'd otherwise reach for a heavier model.