한국어

Mistake Notebook: Why You Get the Same Questions Wrong

At a glance
  • Wrong answers get fixed when you see the right answer. Without it, only 3% fixed themselves.
  • Lucky guesses fade too. Without the answer shown again, 40% of them survived, against 85% with it.
  • After one look at the answer, 86% of errors were corrected on a retest 6 minutes later, but only 56% in a group retested a week later.
  • Errors you were sure of are the easiest to fix, and the most likely to come back after a week.
  • Put lucky guesses in the notebook as well, and re-solve every entry with the answer covered within a week.

A mistake notebook works when it does two things. You check the right answer as soon as you miss a question, and you re-solve it with the answer covered within a few days. In experiments, errors left without the right answer almost never fixed themselves, and after one look at the answer, many errors were wrong again on a retest a week later. Lucky guesses belong in the notebook too.

Does a mistake notebook work?

We could not find a controlled study that compares a mistake notebook itself with other methods. What it relies on, seeing the right answer after an error and recalling it again later, has been tested many times.

Metcalfe (2017) reviewed the research on learning from errors. Making errors and then getting corrective feedback helps learning, and corrective feedback that includes an analysis of the reasoning behind the mistake is crucial. Studying only to avoid errors may work against you.

So whether a mistake notebook helps depends less on keeping one than on what you do inside it.

Is checking the right answer enough?

Checking the right answer is what matters. Seeing it after a miss raised later recall many times over.

Pashler and colleagues (2005) taught 258 people 20 Luganda words. After two study passes, people took tests on the words, and the feedback after each answer differed by group, from none to the correct answer. A final test came a week later.

For words people got wrong, seeing the correct answer raised recall a week later by 494%.

So do not stop at a red cross on the page. Look at what the right answer is.

Should lucky guesses go in the notebook?

Yes. Correct guesses faded by more than half when the answer was not shown again.

In Pashler's study, showing the answer after a correct response made little difference. Butler and colleagues (2008) pointed out that people there could skip answers they were unsure of. So they gave 30 students a multiple-choice general knowledge test, made them answer every question, and had them rate their confidence each time. Half the questions got the correct answer as feedback. After a short filler task, everyone answered the questions again without the choices.

Without the answer, neither errors nor lucky guesses stuck

  • Saw the answer
  • No answer shown
0%25%50%75%100%Saw the answer · First-test errors · 82%82%No answer shown · First-test errors · 3%3%First-testerrorsSaw the answer · Lucky guesses · 85%85%No answer shown · Lucky guesses · 40%40%LuckyguessesSaw the answer · All first-test correct · 93%93%No answer shown · All first-test correct · 79%79%All first-testcorrectNo answer: 3%
Share of first-test answers that were correct on the final test (%). The first test was multiple choice, the final test was answering without choices, after a short filler task. A lucky guess is a correct answer rated "guess".Source: Butler, Karpicke & Roediger (2008), Experiment 1 results
See the numbers
Saw the answerNo answer shown
First-test errors82%3%
Lucky guesses85%40%
All first-test correct93%79%

Only 3% of errors fixed themselves without the answer. Of correct guesses, 85% survived when the answer was shown and 40% when it was not. Between 38% and 40% of correct answers were rated "guess" or "low confidence", so this is not a small slice.

If you only look at the tick marks when you grade, you miss that slice. Treat a correct answer you were unsure of like a wrong one.

Why do I keep making the same mistakes on tests?

Because one look at the right answer does not last. Errors you were sure of are the ones most likely to come back.

Start with the good news. Butterfield and Metcalfe (2001) found that the more confident people were in a wrong answer, the more likely they were to correct it on a retest after seeing the right answer. This is called the hypercorrection effect.

Butler, Fazio and Marsh (2011) asked whether it lasts a week. Fifty students answered 120 science questions, rated their confidence, and saw the correct answer to each for 6 seconds. Half answered the same questions again 6 minutes later, and the other half a week later.

One look at the answer corrected 86% of errors at 6 minutes and 56% at a week

  • First-test errors corrected
  • All answers correct
0%25%50%75%100%First-test errors corrected · After 6 minutes · 86%86%All answers correct · After 6 minutes · 90%90%After 6 minutesFirst-test errors corrected · After a week · 56%56%All answers correct · After a week · 71%71%After a week30 points lower
Group averages (%) for people retested after 6 minutes and different people retested after a week. The groups did not differ significantly on the first test. Everyone saw the correct answer for 6 seconds right after answering each first-test question, and the retest showed no answers.Source: Butler, Fazio & Marsh (2011), results
See the numbers
First-test errors correctedAll answers correct
After 6 minutes86%90%
After a week56%71%

A week later, high-confidence errors were still more likely to be corrected. But the share of errors corrected was 86% in the group retested after 6 minutes and 56% in the group retested after a week, and where people failed, they sometimes gave the very same wrong answer again. The higher their original confidence in an error, the more likely it was to come back unchanged.

The authors suggest that once the correct answer is forgotten, the long-held error takes its place again, and that one round of feedback is not enough to make a lasting fix of errors made with high confidence. They also describe a separate study (Fazio, 2011) in which people retook the test right after seeing the answers and forgot almost nothing a week later.

What should go in a mistake notebook?

Leave material you can re-solve from, rather than neat copies of the questions. Four fields are enough.

FieldWhat to writeWhy
QuestionA photo or page number, with the answer coverableTo re-solve it later without the answer
ConfidenceSure, unsure, or guessSpots lucky guesses and confident errors
WhyOne line: missing concept, misread, slipLooks at the reasoning behind the error
Re-solvedDate and whether you got itShows if you came back within a week

The "why" field is the look at your reasoning that Metcalfe (2017) calls crucial. The confidence field makes the two findings above visible on the page: lucky guesses fade easily, and confident errors come back easily.

How can you start today?

Start with one page of practice questions from today. You do not need a separate notebook.

  1. Mark your confidence as you answer. Write sure, unsure or guess next to each question number.
  2. Look at the right answer as soon as you grade. For each miss, read the answer and the working, and write one line on why you missed it.
  3. Do the same for lucky guesses. A tick mark does not count if you were not sure.
  4. Re-solve once the same day, with the answer covered. In one study described by Butler and colleagues (2011), people who retook the test right after seeing the answers forgot almost nothing a week later.
  5. Re-solve again within a week. Start with the errors you were sure of, since those return most. Do not drop an entry the first time you get it right; come back once more after a longer gap.

How to space the returns is in Spaced repetition schedule: when should you review?, and why you should keep the cards you already got right is in Are Flashcards Effective? Keep the Cards You Already Know.

In Aloom

Recall in Aloom does the "cover the answer and re-solve" step every day. The answers you keep after asking AI come back as questions, 1, 3, or 5 a day as you choose, and you recall before you see the answer. If you miss one, you see the answer right away, and that question comes back on a later day. "I don't know" counts as a miss, so a lucky guess cannot pass for knowing. Questions you get right come back too, after a gap. Each successful recall makes that answer's ink darker. See how Aloom works

FAQ

Do mistake notebooks actually work?

We could not find a controlled study that tests a mistake notebook against other methods. But seeing the correct answer after an error raised recall a week later by a large margin (Pashler et al., 2005), and after one look at the answer, fewer errors were corrected on a retest a week later than on one 6 minutes later (Butler et al., 2011). A notebook that includes checking the answer and re-solving is the kind you can expect to help.

What is the error log study method?

It is the same idea under another name: a running list of the questions you missed, with why you missed them, re-attempted on a schedule. The research above suggests adding two things many templates leave out: your confidence on each answer, and the questions you got right by guessing.

Why do I keep making the same mistakes on tests?

One look at the answer does not last. In Butler and colleagues' (2011) study, 86% of errors were corrected by people retested after 6 minutes but 56% by people retested a week later, and errors made with high confidence were the most likely to return unchanged. Re-solve them with the answer covered within a week.

References