한국어

ChatGPT Practice Tests: Do AI Quiz Questions Help You Learn?

At a glance
  • An AI-made practice test is still a practice test. Answering from memory is what does the work.
  • With the right answer shown after each question, short-answer practice did best on a final test three days later.
  • Wrong options you read on a multiple-choice test can come back later as your answers.
  • In one 2023 study, 25% of 60 questions written by ChatGPT 3.5 had wrong or misleading answers.
  • Ask for short-answer questions, check each answer against the source text, and retake them two or three days later.

An AI-made practice test helps you learn, as long as you answer it from memory. What decides how much it helps is how you take it, not who wrote it. Ask for short-answer questions instead of multiple choice, check each answer against your own notes, and take the same questions again a few days later. Some AI-written questions have wrong answers, so check the answer in the source, not in the chat.

Does studying with AI practice questions work?

Yes. Whoever wrote the question, pulling the answer out of memory is what does the work.

Adesope and colleagues (2017) ran a meta-analysis of practice testing experiments. Practice tests helped learning more than restudying and every other comparison condition, and how much they helped depended on features of the practice test. The rest of this note is about those features.

One study put ChatGPT-made questions into a course. Yusof (2025) had second-year education students at a Malaysian university answer questions without notes for 10 to 15 minutes in class, after the lecture, followed by the answers and a discussion. Most questions were written by the lecturer. In one class session, students wrote questions with ChatGPT, which the lecturer reviewed and revised. In the revision week before the final, students answered the same questions again and used ChatGPT to check their answers.

GroupStudents sampledFinal exam mean
Earlier classes (no ChatGPT practice)64 of 19725.44
Later classes (ChatGPT practice)64 of 18628.81

The difference was statistically significant. But this compares earlier and later classes, and their final exams had the same format but were not identical, so it cannot separate the AI from the weekly practice and the second round. Students wrote ChatGPT questions in only one session, and the lecturer corrected them first.

Multiple choice or short answer: which should you ask for?

Short answer. When students saw the right answer during practice, short-answer practice led to the best performance three days later.

Kang, McDermott and Roediger (2007) had students read four short journal articles. After each article they took a multiple-choice test, took a short-answer test, read statements with the same facts, or did an unrelated task. In the second experiment, students got corrective feedback during these first tests. Three days later everyone took a final test with both multiple-choice and short-answer questions.

With feedback, short-answer practice led to the best performance on the final test. The authors took this to mean that the more demanding the retrieval, the more it helps later.

The first experiment, with no feedback, came out differently. Multiple-choice practice led to better final scores than short-answer practice, and it did not differ significantly from simply reading the statements. The authors added feedback in the second experiment to even out how much of the tested information each group saw.

Short-answer questions and checking the answer go together. Without the check, short answers can do worse.

What happens if you skip checking the answers?

Some of the wrong options you read can come back later as your answers.

Roediger and Marsh (2005) found that taking a multiple-choice test helped people on a later test, and they also found the cost. The more wrong options people had read, the smaller the benefit, and the more often they gave those wrong options as answers later. They wrote that multiple-choice testing may create false knowledge without anyone meaning to.

Taking an AI quiz, glancing at the score and moving on leaves those wrong options unchecked. For what to do with the questions you miss, see Mistake Notebook: Why You Get the Same Questions Wrong.

Can you trust the questions AI writes?

Many are fine, but some are wrong. Check the answers in your source.

Ngo and colleagues (2024) pasted immunology study guides into ChatGPT 3.5 in January and February 2023 and asked it to write four multiple-choice questions with explanations, 60 questions in all. Four reviewers graded them, and the harshest grade counted.

A quarter of 60 ChatGPT 3.5 questions had wrong or misleading answers

0%25%50%75%100%Share of questions · Usable with minor fixes · 32%32%Usable withminor fixesShare of questions · Needed major edits · 43%43%Neededmajor editsShare of questions · Wrong or misleading · 25%25%Wrong ormisleading1 in 4 wrongor misleading
Percent of 60 immunology multiple-choice questions by grade, using the strictest of four reviewers. Most of the "needed major edits" group (19 of 60) gave no explanation for the wrong options.Source: Ngo et al. (2024), Tables 2 and 3
See the numbers
Share of questions
Usable with minor fixes32%
Needed major edits43%
Wrong or misleading25%

Another study came out more hopeful. Cheung and colleagues (2023) mixed 50 medical multiple-choice questions written by ChatGPT from textbooks with 50 written by two professors, and five assessors from the UK, Ireland, Singapore and Hong Kong graded them without knowing who wrote which. The AI questions scored a little lower only on relevance (7.56 vs 7.88 out of 10), and the total scores did not differ significantly. ChatGPT took 20 minutes 25 seconds; the professors took 211 minutes 33 seconds. The AI questions' scores were also spread more widely than the professors' questions.

ChatGPT wrote 50 exam questions in about a tenth of the professors' time

050100150200250Minutes · ChatGPT · 20.420.4ChatGPTMinutes · Two professors · 211.6211.6Two professorsAbout a tenthof the time
Minutes to write 50 medical multiple-choice questions. ChatGPT took 20 minutes 25 seconds and two professors 211 minutes 33 seconds in total. Rounded to one decimal place.Source: Cheung et al. (2023), results
See the numbers
Minutes
ChatGPT20.4
Two professors211.6

Both studies used 2023 models, and today's models may do better. This note did not find a study that graded current models the same way. Either way the advice holds. The person taking the quiz cannot tell which question is wrong, so check each answer against a sentence in the source, not against what the AI says.

How can you try this today?

Take one chunk of what you studied today. Any AI will do.

  1. Ask for short answers. Paste the material and say: "Using only this material, write 8 short-answer questions. For each, note the answer and the exact sentence from the material it comes from, but don't show them until I've answered. Ask one question at a time."
  2. Close the material and write your answer. Write at least a line, not just in your head. If you don't know, write that.
  3. Check the answer against the source sentence. If the quoted sentence isn't in your material, or says something else, drop the question.
  4. If you use multiple choice, check the wrong options too. After you pick, read why each option you didn't pick is wrong, and look hardest at the wrong one you chose.
  5. Retake the same questions two or three days later. Start with the ones you missed. The final test in the study above came three days later.

If you'd rather have the AI guide you with questions than hand you answers, see Is ChatGPT Study Mode Good?.

In Aloom

In Aloom, your recall questions are made from the answers you chose to keep after asking AI. Each day you answer 1, 3 or 5 of them, recalling before you see anything. They are multiple choice and true or false, so after you answer, Aloom shows why the right one is right and why the closest wrong one is not, so that a wrong option doesn't settle in as a memory. Each question comes from one answer you kept, so you always know where to check. The more you recall, the darker the ink. See how Aloom works

FAQ

Can ChatGPT make practice tests?

Yes. Ask for questions drawn only from your material, in short-answer form, with the answer and its source sentence held back until you have answered. The source sentence lets you check whether the AI's answer is right. In a 2023 study, a quarter of ChatGPT 3.5's questions had wrong or misleading answers (Ngo et al., 2024).

Is multiple choice or short answer better for studying?

Short answer, if you check the answers. In Kang and colleagues (2007), short-answer practice with feedback did best on a final test three days later. Without feedback, multiple-choice practice did better than short answer. But wrong options you read on a multiple-choice test can come back later as answers (Roediger & Marsh, 2005).

Are AI generated practice questions as good as a teacher's?

Sometimes. In one study, ChatGPT's medical questions scored about the same as professors' overall, but varied more (Cheung et al., 2023). In another, only 32% were usable with just minor formatting fixes (Ngo et al., 2024). Treat each question as a draft and check its answer in your source.

References