Cover

Guidelines

Skill in Techniques of Item Writing

 

Before we start on discussing the challenges in item writing and the ways to overcome the hurdle of writing better, we must know the difference between literal, inference and critical questions. Literal questions ask the students to find specific details stated within the text. Inference and critical questions require the students to stop and think about the text.

 

What is a Multiple Choice Item?

 

The multiple choice item (MCQ) consists of two distinct parts:

 

1. The first part that contains task or problem is called stem of the item. The stem of the item may be presented either as a question or as an incomplete statement. The form makes no difference as long as it presents a clear and a specific problem to the examinee.

 

2. Second part presents a series of options or alternatives. Each option represents possible answer to the question. In a standard form one option is the correct or the best answer called the keyed response and the others are misleads or foils called distracters.

 

The number of options used differs from one test to the other. An item must have at least

three answer choices to be classified as a multiple choice item. The typical pattern is to have

four or five choices to reduce the probability of guessing the answer. A good item should

have all the presented options look like probable answers at least to those examinees who

do not know the answer.

 

Parts of a multiple choice question (Bull & Mckenna, 2002)

A traditional multiple choice question (or item) is one in which a student chooses one answer from a number of choices supplied. A multiple choice question consists of a:

  • stem - the text of the question

  • options - the choices provided after the stem (these include the key and the distracters)

  • the key - the correct answer in the list of options

  • distracters - the incorrect answers in the list of options

Terminology : Multiple Choice Questions

 

1.Stem: presents the problem

2. Keyed Response: correct or best answer

3.Distracters: appear to be reasonable answers to the examinee who does not know the content

4. Options: include the distracters and the keyed response.

 

Common Errors in Writing Multiple-Choice Items:

 

  1. Item stems that are too lengthy or imprecise.

Poor: Which one of the following is something that can be found in a short story that is also found in this personal narrative?

Corrected: Which of these short story elements is present in this personal narrative?

  1. Item stems that are choppy or abstract. Test takers need to have a clear picture of their task before they begin reading the answer choices.

Poor: Reading this chart - A) can tell someone the growth of four stocks over a four-year period

Corrected: This chart shows the growth of – A) four stocks over a four-year period.

  1. Item options that lack balance. This problem can be seen in several ways, including “stacked” options, options that are dramatically different in “flavor,” and one option that is obvious in some way.

Poor: Animal tracks can be best found in

    1. mud or snow (stacked option – it contains two elements)

    2. grass

    3. water

    4. the library (option is dramatically different in “flavor”)

Corrected: Animal tracks can be best found in a) mud b) grass c) water d) sand

  1. Item stems and their options that needlessly repeat words.

Poor: Penguins cannot fly because

    1. their wings are too small

    2. their wings are too flat

    3. their wings are too curved

    4. their wings are too heavy

Corrected: Penguins cannot fly because their wings are too

  1. small

  2. flat

  3. curved

  4. heavy

  1. A stimulus, item stem, or an option containing information that clues the correct answer of the item itself or of another item in the test.

  2. Items that measure more than one standard.

Example: Holly sitted next to the band leader.

 

How should this sentence be written?

    1. Holly sat next to the band leader.

    2. Holly sit next to the band leader.

    3. Holly sitting next to the band leader.

    4. holly sat nexst to the band leader.


In this case, the item was intended to measure the standard “Use correct verb formation.” In this item, options a, b and c provide various forms of the verb. Option D tests capitalization and spelling (which are measured under separate standards). Therefore, this item is assessing a student’s ability to do more than merely choosing the correct verb in the sentence. The option d need to be revised (only the verb contains the error) to match the standard.

 

Guidelines for Response Options:

 

  1. No response option (correct answer or distracter) should be significantly different from the other three options (in length, complexity, structure). However, pairs of options may reflect minor differences (two long and two short options).

  2. The response options should relate to the stem in the same way both grammatically and conceptually.

  3. No response option (correct answer or distracter) should eliminate another response action. Thus two distracters should not be opposites of each other.

  4. No response option (correct answer or distracter) should deny the truth of the stem.

  5. Response options should not give clues to students. Avoid the use of templates (never, always, all). Words used in the stem should not be repeated in the response options (correct answer or distracter).

  6. Avoid repeating words or phrases in the options that could be placed in the stem.

  7. The distracters should be plausible and reasonable in terms of the stimulus, the item stem, and the student’s level of knowledge.

  8. Distracters may include a) common misconceptions or misunderstandings or b) misreading of the stimulus. For some tests, it is helpful for item writers to include the rationales (reasons) for the distracters.

  9. Do not use distracters that have nearly the same meaning even though they are worded differently. (Julian was happy to go. Julian was looking forward to the trip.)

 

Submission Checklist:

 

When you are ready to submit your materials, please be sure you have completed the following:

 

  1. Read and understand the project’s guiding documents (example: test and item specifications)

  2. Follow the guidelines in the Item Writer’s Guide.

  3. Create original, interesting, grade-appropriate items that measure the standards.

  4. Use the review checklists for each item you are submitting to ensure it is of the highest quality.

  5. Complete the template provided for each item and follow submission guidelines.

  6. Verify with source materials for factual information.

 

14 Rules for Writing Multiple-choice questions

 

1. Use Plausible Distracters (wrong-response options)

• Only list plausible distracters, even if the number of options per question changes.

• Write the options so they are homogeneous in content.

• Use answers given in previous open-ended exams to provide realistic distracters.

 

2. Use a Question Format

• Experts encourage multiple-choice items to be prepared as questions (rather than incomplete statements)

 

 

Incomplete Statement Format:

 

The capital of California is in ____________. Less Effective

 

Direct Question Format:

 

In which of the following cities is the capital of California? More Effective

 

3. Emphasize Higher-Level Thinking

 

• Use memory-plus application questions. These questions require students to recall principles,

rules or facts in a real life context.

• The key to preparing memory-plus application questions is to place the concept in a life situation or context that requires the student to first recall the facts and then apply or transfer the application of those facts into a situation.

• Seek support from others who have experience writing higher-level thinking multiple-choice questions.

 

Memory Only Example

Which description best characterizes whole foods?

a. orange juice

b toast                                           Less Effective

c. bran cereal

d. grapefruit

 

Memory-Plus Application Example

 

Sally’s breakfast this morning included one glass of orange juice (from concentrate), one slice of toast, a small bowl of bran cereal and a grapefruit. What “whole food” did Sally eat for breakfast?

a. orange juice

b. toast                                                Effective

c. bran cereal

d. grapefruit

 

Memory-Plus Application Example

                                                                            Effective

Which one of the following best illustrates the law of diminishing returns?
a. The demand for a farm product increased faster than the supply of the product.

b. The population of a country increased faster than the means of subsistence.

c. A machine decreased in utility as its parts became worn.

d. A factory doubled its labor force and increased production by 50 percent

 

Ability to Interpret Cause-and-Effect Relationships Example

 

Why does investing money in common stock protect against loss of assets during inflation?

a. It pays higher rates of interest during inflation.

b. It provides a steady but dependable income despite economic conditions.

c. It is protected by the Federal Reserve System.                                Effective

d. It increases in value as the value of a business increases.

 

Ability to Justify Methods and Procedures Example

 

Why is adequate lighting necessary in a balanced aquarium?

a. Fish need light to see their food.

b. Fish take in oxygen in the dark. Effective

c. Plants expel carbon dioxide in the dark.

d. Plants grow too rapidly in the dark

 

4. Keep Option Lengths Similar

• Avoid making your correct answer the long or short answer

 

5. Balance the Placement of the Correct Answer

• Correct answers are usually the second and third option

 

6. Be Grammatically Correct

• Use simple, precise and unambiguous wording

• Students will be more likely to select the correct answer by finding the grammatically correct option

 

7. Avoid Clues to the Correct Answer

• Avoid answering one question in the test by giving the answer somewhere else in the test

• Have the test reviewed by someone who can find mistakes, clues, grammar and punctuation

problems before you administer the exam to students

• Avoid extremes – never, always, only

• Avoid nonsense words and unreasonable statements


8. Avoid Negative Questions

 

• 31 of 35 testing experts recommend avoiding negative questions

• Students may be able to find an incorrect answer without knowing the correct answer

 

9. Use Only One Correct Option (Or be sure the best option is clearly the best option)

 

• The item should include one and only one correct or clearly best answer

• With one correct answer, alternatives should be mutually exclusive and not overlapping

• Using MC with questions containing more than one right answer lowers discrimination between students

 

  1. Give Clear Instructions

 

  1. Use Only a Single, Clearly-Defined Problem and Include the Main Idea in the Question

    • Students must know what the problem is without having to read the response options

 

12. Avoid the “All the Above” Option

 

Students merely need to recognize two correct options to get the answer correct

 

13. Avoid the “None of the Above” Option

• You will never know if students know the correct answer

 

3.1 Writing Stems

(i) Present a single, definite statement or direct question to be completed or answered by one of the several given choices

 

A. original stem

 

Polysaccharide

  1. are made up of thousands of smaller units called monosaccharides

  2. are NOT found in the aloe vera leaf

  3. are created during photosynthesis

  4. can be described by the chemical formula: CHHOH

B. improved stem

 

Polysaccharides of the plant cell wall are synthesized mainly in the

  1. endoplasmic reticulum

  2. cytosol

  3. plasma membrane

  4. Golgi complex

 

In Example A, there is no sense from the stem what the question is asking. Example B more clearly identifies the question and offers the student a set of homogeneous choices.

(ii) Avoid unnecessary and irrelevant material in the stem. It should be clear and unambiguous

A. original stem:

 

Paul Muldoon, an Irish postmodern poet who uses experimental and playful language, uses which poetic genre in "Why Brownlee Left"?

  1. sonnet

  2. elegy

  3. narrative poem

  4. dramatic monologue

  5. haiku

B. improved stem

 

Paul Muldoon uses which poetic genre in "Why Brownlee Left"?

 

  1. sonnet

  2. elegy

  3. narrative poem

d. dramatic monologue

e. haiku

Example A contains material irrelevant to the question. This sort of material should not be used to make the answer less obvious. This tends to place too much importance on reading comprehension as a determiner of the correct option

(iii) Use clear, straightforward language in the stem of the item.

Questions that are constructed using complex or imprecise wording may become a test of reading comprehension rather than an assessment of whether the student knows the subject matter.

A. original stem

 

As the level of fertility approaches its nadir, what is the most likely ramification for the citizenry of a developing nation?

 

  1. a decrease in the workforce participation rate of women

  2. a dispersing effect on population concentration

  3. a downward trend in the youth dependency ratio

  4. a broader base in the population pyramid

  5. an increased infant mortality rate

B. improved stem

 

A major decline in fertility in a developing nation is likely to produce a

 

  1. decrease in the workforce participation rate of women

  2. dispersing effect on population concentration

  3. downward trend in the youth dependency ratio

  4. broader base in the population pyramid

e.an e. increased infant mortality rate

(iv) Use negatives sparingly in the stem. If negatives must be used, capitalize, underscore, embolden or otherwise highlight them. Negatives include ‘except’, ‘only’

A. original stem

Which one of the following is not a symptom of osteoporosis?

 

  1. decreased bone density

  2. frequent bone fractures

  3. raised body temperature

  4. lower back pain

B. improved stem

Which one of the following is a symptom of osteoporosis?

 

  1. decreased bone density

  2. raised body temperature

  3. hair loss

  4. painful joints

 

Negatives in the stem usually require that the answer be a false statement. Because students are likely in the habit of searching for true statements, this may introduce an unwanted bias.

(v) Put as much of the question in the stem as possible, rather than duplicating material in each of the options.

A. original stem

 

Theorists of pluralism have asserted which of the following?

 

  1. The maintenance of democracy requires a large middle class.

  2. The maintenance of democracy requires autonomous centres of contervailing power.

  3. The maintenance of democracy requires the existence of a multiplicity of religious groups.

  4. The maintenance of democracy requires a predominantly urban population.

  5. The maintenance of democracy requires the separation of governmental powers.

B. improved stem

 

Theorists of pluralism have asserted that the maintenance of democracy requires

 

  1. a large middle class

  2. autonomous centres of contervailing power

  3. existence of a multiplicity of religious groups

  4. a predominantly urban population

  5. separation of governmental powers

 

Another example: If the point of an item is to associate a term with its definition, the preferred format would be to present the definition in the stem and several terms as options, rather than to present the term in the stem and several definitions as options.

 

(vi) Avoid irrelevant clues to the correct option in the stem.

Grammatical construction, for example, may lead students to reject options which are grammatically incorrect as the stem is stated. Perhaps more common and subtle, though, is the problem of common elements in the stem and in the answer.

Consider the following item:

What led to the formation of the States’ Rights Party?

a. The level of federal taxation

b. The demand of states for the right to make their own laws

c. The industrialization of the South

d. The corruption of federal legislators on the issue of state taxation

 

One does not need to know U.S. history in order to be attracted to the answer, b.

 

 Procedural Rules:

 

Use either the best answer or the correct answer format.

Avoid complex multiple-choice & Type K format.

Format the item vertically, not horizontally.

Allow time for editing and other types of item revisions.

Use good grammar, punctuation, and spelling consistently.

Minimize examinee reading time in phrasing each item.

Avoid trick items, those which mislead or deceive examinees into answering incorrectly.

 

Content-related Rules:

 

Base each item on an educational or instructional objective.

Focus on a single problem.

Keep the vocabulary consistent with the examinees' level of understanding.

Avoid cuing one item with another; keep items independent of one another.

Use the author's examples as a basis for developing your items.

Avoid over specific knowledge when developing the item.

Avoid textbook, verbatim phrasing when developing the item.

Avoid items based on opinions.

Use multiple-choice to measure higher level thinking.

Test for important or significant materials; avoid trivial material.

 

Stem Construction Rules:

 

State the stem in either question form or completion form.

When using the completion form, don't leave a blank for completion in the beginning or

middle of the stem.

Ensure that the directions in the stem are clear, and that wording lets the examinee know

exactly what is being asked.

Avoid window dressing (excessive verbiage) in the stem.

Word the stem positively; avoid negative phrasing.

Include the central idea and most of the phrasing in the stem.

 

General Option Development Rules:

 

Use as many options as are feasible; more options are desirable.

Place options in logical or numerical order.

Keep options independent; options should not be overlapping.

Keep all options in an item homogeneous in content.

Keep the length of options fairly consistent.

Avoid, or use sparingly, the phrase all of the above.

Avoid, or use sparingly, the phrase none of the above.

Avoid the use of the phrase I don't know.

Phrase options positively, not negatively.

Avoid distracters that can clue test-wise examinees; for example, avoid clang

associations, absurd options, formal prompts, or semantic (overly specific or overly

general) clues.

Avoid giving clues through the use of faulty grammatical construction.

Avoid specific determiners, such as never and always

 

Key Option Development Rules:

 

Position the correct option so that it appears about the same number of times in each

possible position for a set of items.

Make sure there is one and only one correct option.

 

Distracter Development Rules:

Use plausible distracters; avoid illogical distracters.

Incorporate common errors of students in distracters.

Avoid technically phrased distracters.

Use familiar yet incorrect phrases as distracters.

Use true statements that do not correctly answer the item.

Avoid the use of humor when developing options.

 

Express Items as Precisely, Clearly and Simply as Possible

 

Unnecessary material reduces the effectiveness of an item by forcing examinees to respond to the irrelevant material and perhaps be distracted by it.

 

Emphasize General Tasks Rather than Small Details

 

Avoid Jargon and Textbook Language

 

Locate and Delete Irrelevant Clues

 

Occasionally, verbal associations and grammatical clues render an item ineffective.

 

Eliminate Irrelevant Sources of Difficulty

 

Strengths and limitations of MCQs (Zimmaro, 2004:11)

 

Strengths:

1. Achievement of learning outcomes from simple to complex can be assessed.

2. Highly structured and clear tasks are provided.

3. A broad sample of achievement can be assessed.

4. Incorrect alternatives provide diagnostic information.

5. Scores are less influenced by guessing than true-false items.

6. Scores are more reliable than subjectively scored items (e.g. essays).

7. Scoring is easy, objective, and reliable.

8. Item analysis can reveal how difficult each item was and how well it discriminated between the strong and weaker students in the class

9. Achievement can be compared from class to class and year to year

10. Can cover a lot of material very efficiently (about one item per minute of testing time for straightforward questions).

11. Items can be written so that students must discriminate among options that vary in degree of correctness.

12. Avoids the absolute judgments found in True-False tests.

 

Limitations:

1. Constructing good items is time consuming.

2. It is frequently difficult to find plausible distracters.

3. Can be ineffective for assessing some types of problem solving and the ability to organize and express ideas.

4. Real-world problem solving differs – a different process is involved in proposing a solution versus selecting a solution from a set of alternatives.

5. Scores can be influenced by reading ability.

6. There is a lack of feedback on individual thought processes – it is difficult to determine why individual students selected incorrect responses.

7. Students can sometimes read more into the question than was intended.

8. Often focus on testing factual information and fails to test higher levels of cognitive thinking.

9. Sometimes there is more than one defensible “correct” answer.

10. They place a high degree of dependence the instructor’s writing ability.

11. Does not provide an assessment of writing ability.

12. May encourage guessing.

 

Impressum

Tag der Veröffentlichung: 08.12.2014

Alle Rechte vorbehalten

Widmung:
For assessment writers who struggle to do their best.

Nächste Seite
Seite 1 /