Innovative Multiple-Choice in formative tests

My recent blogpost, a discussion in a linkedin group on spychometrics and the just announced conference on formative assessment make me write this small post.

  • Answer Untill Correct (AUC) – test-taker selects answer, then receives feedback if correct or not, if not –> test-takers selects next answer etc.:
  • Discrete Option Multiple Choice (DOMC)  – test-taker is only shown one answering option of an mc question at a time, student must select correct answer when it is displayed:
  • Certainty Based Marking (CBM) –  test-taker must provide certainty level of being correct for each option in an mc question:
  • NEW! Confidence  Based Marking – seems an interesting variant of Certainty Based Marking (but uses the old term of Gardner-Medwin for it unfortunately) – first, the test-takers only sees the stimulus, then rates confidence of being able to answer correctly, then is presented all options and then selects the correct option:
  • Number right elimination testing (NRET) – a variant of elimination testing (ET) – mark incorrect answers, mark correct answer and provide certainty level for it:
  • Liberal (Free-Choice) Multiple-Choice Tests – test-taker may select any option of an mc question but correct option is awarded 3 points, any incorrect selected option is awarded -1 point. Main idea: System discourages guessing:, try it at

So many ideas!


IF-AT testing – another interesting format for formative uses of multiple-choice questions

In the category ‘exotic uses of multiple-choice questions’ I ran into some information regarding the IF-AT mode of multiple-choice testing. The IF-AT mode is a specific form of testing on the basis of ‘answering untill correct’. What is IF-AT testing? According to Mike and Beth Epstein:

“The Immediate Feedback Assessment Technique, also known as the IF-AT, is an exciting and revolutionary new testing system that transforms traditional multiple-choice testing into an interactive learning opportunity for students and a more informative assessment opportunity for teachers.

Here’s How. Use of the IF-AT testing system enables students to be provided with immediate feedback about the accuracy of their answers to each question in a test/quiz/homework assignment, etc. as the students are completing each item. The IF-AT system provides immediate affirmative feedback (if a student’s answer choice is correct) and/or corrective feedback (if a student’s answer choice is incorrect).”

The interesting idea of IF-AT is that the system explicitely tries to reward partial knowledge. It does that by also rewarding points to test-takers if they do not answer a question correctly in first instance, but also in second and third instance. Of course, the number of points awarded decrease with the number of attempts. So if a test-taker answers a four-option multiple-choice question correctly the first time, 4 points are awarded. The second change is awarded with 2 points, the third with 1 point and finally 0 points are awarded.

The IF-AT testing mode works with scratch-off forms and that seems quite straight-forward. I would imagine that especially computer based testing system would be able to take advantage of this method. Of course, the iCMA system of the Open University does this and Sally Jordan writes a lot about that system:

But also Moodle has nice features to support this (thanks Liz!):

I do think this mode of testing is worthwhile for formative purposes. It can provide very interesting information to discuss subject matter ‘in ones own head’,  with peers and teachers. I think it helps  to correct misconceptions or students to become aware of for example subtleinterpretation differences regarding facts, concepts and procedures etc..

Now, I cannot really work out how such a testing mode would work in practice for summative purposes. The first problem lies in the fact that a test-taker is actually made aware during the test about his score progress (the running score can easily be assessed by a student). This has an effect on the answering strategies a student uses I think. Also, I could not find information on how to set cut-scores or how to deal with correction for guessing. The latter becomes somewhat more complicated of course using such a mode of testing.

Yet, I think this testing mode fits nicely with ideas for formative assessment using multiple-choice questions such as Quizzing, Certainty Based Marking, Concept Testing, Collaborative Testing, Gaming and Classroom Clickers.


The IF-AT website:

An informative Youtube video explaining IF-AT:

A research article by Epstein, Epstein and Brosvic can be found at

EAC Outcomes revisited – data-mining in de toetsmodule van Blackboard wordt mogelijk

In een eerdere post had ik al eens melding gemaakt van het programma EAC Outcomes. Het is een applicatie die bestaat uit een Windows programma met een Blackboard Building block. Het programma geeft je de mogelijkheid om de resultaten van alle toetsen die in Blackboard aanwezig zijn  te kunnen bekijken, downloaden en analyseren. Uitgebreid analyseren. Zowel qua aantallen, maar ook psychometrisch.

Naar aanleiding van de steeds groter wordende aandacht voor toetsing en tussentijds toetsen op de VU is dit programma van groot belang. Blackboard zelf geeft namelijk helemaal geen inzicht in de mate waarin de assessement module wordt ingezet in het onderwijs. Met deze tool wordt dat mogelijk. Mijn nieuwe baas heeft aangegeven dat het belangrijk is om deze ‘management informatie’ boven water te krijgen. Dus dat moesten we maar gaan onderzoeken en doen. Voor de prijs hoeven we het niet te laten: Dollar 5000,- per jaar is te overzien.

Via een Adobe Connect meeting hebben de makers uitgelegd hoe het systeem precies werkt. Ik heb het voor het gemak maar opgenomen. Het is hier te vinden. Het duurt ongeveer 35 minuten.

EAC-outcomes is ook al beschikbaar voor Moodle. De makers van het programma zijn aan het onderzoeken of ze hun koppeling kunnen maken met Sakai en Canvas Instructure. Interessant.

Overigens heeft het systeem ook een mogelijkheid om Surveys via Blackboard veel doelmatiger vorm te geven en de resultaten te verwerken. Er zit al een mobiele interface bij. Ook handig.

ClassMarker – Online Testing Free Quiz Maker Create the Best web-based quizzes

Weer een nieuwe online quizzing tool (althans voor mij). Ziet er heel aardig uit. Een mooie concurrent voor bijvoorbeeld ZOHO Challenge.

Easily Create First-Class Web Based Quizzes & Tests!

The Best Online Testing for Business & Education

ClassMarker’s secure, professional web-based testing service is an easy-to-use, customizable online quiz maker for business training & educational assessment with tests and quizzes graded instantly – saving hours of paperwork!

ClassMarker – Online Testing Free Quiz Maker Create the Best web-based quizzes.

Onderwijs Brabant – Lector Desirée Joosten: Toetsing is een ondergeschoven kindje

Toetsen is een hot item in het onderwijs. De overheid ziet het als een instrument om de kwaliteit van het onderwijs te verbeteren. Critici zien het als een toetsgekte, met kwalijke effecten als teaching to the test. Dr. Desirée Joosten – ten Brinke doet er als lector onderzoek naar.

Onderwijs Brabant – Lector Desirée Joosten: Toetsing is een ondergeschoven kindje.

Increased Structure and Active Learning Reduce the Achievement Gap in Introductory Biology

Getipt door onze eigen Ad Valvas: Massale colleges hoeven niet slecht te zijn voor het onderwijs, zolang docenten hun studenten bij de les houden met korte opdrachten, gerichte vragen en wekelijkse toetsen. Aha, een stokpaardje van me natuurlijk.

Science, technology, engineering, and mathematics instructors have been charged with improving the performance and retention of students from diverse backgrounds. To date, programs that close the achievement gap between students from disadvantaged versus nondisadvantaged educational backgrounds have required extensive extramural funding. We show that a highly structured course design, based on daily and weekly practice with problem-solving, data analysis, and other higher-order cognitive skills, improved the performance of all students in a college-level introductory biology class and reduced the achievement gap between disadvantaged and nondisadvantaged students—without increased expenditures. These results support the Carnegie Hall hypothesis: Intensive practice, via active-learning exercises, has a disproportionate benefit for capable but poorly prepared students.

See full report here:

Retrieving Essential Material at the End of Lectures Improves Performance on Statistics Exams

John Kleeman of Questionmark twittered the following very interesting link to an article in the journal Teaching of Psychology. My University does not have full access to this article, but the abstract is very promising.

“At the end of each lecture in a statistics for psychology course, students answered a small set of questions that required them to retrieve information from the same day’s lecture. These exercises constituted retrieval practice for lecture material subsequently tested on four exams throughout the course. This technique is called the PUREMEM (pronounced “pure mem”) procedure for Pure Memory or Practicing Unassisted Retrieval to Enhance Memory for Essential Material. Exam scores were significantly and substantially higher in a section of the course taught with the PUREMEM procedure than one taught without it. Students liked the procedure and believed it increased learning via several different mechanisms.”

Klik hieronder voor het artikel