Rule of thumb: 40 questions in a 4-choice multiple-choice test. Why?

I often get asked how many questions need to be included in an end-of-course test for a typical higher education course. The answer to that problem is both simple and complex.

The simple answer to the problem is to use the rule-of-thumb: For a test consisting of 4-choice multiple-choice test use about 40 questions.

And why 40?

Well, consider the following:

IF

  • you manage to design fairly well discriminating test items (eventual Rir > 0.25) – implying that there is also a spread in the degree of knowledge or skill among the student population …
  • you want to keep administration time reasonable (say the test time should remain within 1-2 hours)
  • your subject matter is not too limited nor too comprehensive (say between 8-12 different topics)

THEN

  • The test reliability will be probably be acceptable (say alpha > 0.75)
  • And the validity (in terms of coverage of the subject matter) will probably be acceptable

The somewhat more complex answer to the problem is that if the premises are different (you are not able to design well discriminating questions, your subject matter is more comprehensive, the spread in knowledge and skill is small, you can allot more time for the test takers) then you should consider to use more questions in your test. It is not hard science.

For a nice elaboration on this issue if found http://www.indiana.edu/~best/bweb3/test-reliability/ interesting.

The rule-of-thumb is a typical Dutch rule-of-thumb by the way. See also for example http://www.utwente.nl/ces/toetsing/Docenten/Toetsconstructie/Aantal-vragen/

PwC via Questionmark also have some specific considerations for this. See http://blog.questionmark.com/how-many-items-are-needed-for-each-topic-in-an-assessment-how-pwc-decide.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s