How do I construct a multiple-choice (MC) item?

What is a multiple-choice item and what does it look like?

How do I write a multiple-choice item?

How do I recognize poorly written multiple-choice items?

How do I assign item status, status on compromise, and randomize answers?

How do I re-assign multiple-choice items from deleted objectives?

How do I align K-levels?

What are additional sources of information on multiple-choice item construction?

What is a multiple-choice item and what does it look like?
Multiple-choice (MC) items are assessment questions that have one stem and multiple alternatives. An advantage of MC items is that they are quickly and objectively scored. Another advantage is that well written and structured MC items can assess many different and specific areas of some content in a quick manner.

MC items can be used to assess any knowledge level except K2: Recall and K5: Synthesize. K2: Recall cannot be tested because the answer is listed. The student would be recognizing the answer from the list and not recalling the answer. K5: Synthesize cannot be tested because it requires the student to apply knowledge to create new and/or unique solutions. If the answer is listed, then the answer is not new and/or unique. While a MC item cannot be written for K5: Synthesize, it is possible to write MC items for K5: Analyze and K5: Evaluate.

There are a few limitations of MC items:

• Guessing of answers may result, because the answer is listed.
• Most of the time, MC items indirectly assess the target behavior.
• MC items are more time-consuming to construct than other types of items.

A MC item has two parts: the stem and the alternatives. The stem of the MC item contains the problem statement, and it must include all of the information required to answer the item. The alternatives are a list of possible answers, which either complete the stem or fill-in-the-blank within the stem. Only one alternative can be the correct answer. The other incorrect alternatives are distracters.

The following is an example of a MC item:

Example:

Of the four stages of sea growth, which of the following would be the first stage?

Source: NAVEDTRA 130A, Volume II, A-6-2-5

In the example above, "Of the four stages of sea growth, which of the following would be the first stage?" is the stem. Four possible answers are listed. These are the four alternatives for the MC item. Alternative C (c. slime) is the correct answer and the other three alternatives are plausible distracters.

Back to top

How do I write a multiple-choice item?
A multiple-choice (MC) item is made up of a stem and multiple alternatives. Click on the steps below to guide the development of multiple-choice items.

Click on the steps below to guide the selection of an assessment category.

1. Identify the learning objective and K-level to be assessed.
2. Determine if it is appropriate to write a multiple-choice item for the objective.
3. Locate a reference source for the multiple-choice item.
4. Write the stem of the multiple-choice item.
5. Determine the correct alternative.
6. Write the incorrect alternatives (distracters) of the multiple-choice item.
7. Correctly format the multiple-choice item.
8. Review the multiple-choice item.
   
   
   
1. Identify the learning objective and K-level to be assessed.
   Prior to actually writing a MC item, identify the learning objective and K-level that are to be assessed. The identified learning objective should already have a K-level (knowledge level) associated with it. (See the K-levels Tab of the Objectives Module for more information.) You must determine whether the MC item will be written at or below the objective's K-level.

• It is important that the knowledge level of the MC item matches the knowledge level of the information used on the job. Thus, the K-level of the MC item should match the K-level of the learning objective. For instance, if the K-level of the objective is K4: Apply, a K1: Recognize MC item will not fully assess the objective because the students will be recognizing information and not applying information.

• However it may be necessary to write and use MC items that are lower than the objective's K-level. For objectives with K-3: Comprehend, K-4: Apply, or K-5: Analyze/Synthesize/Evaluate levels, items written below the objective's K-level can be used to pinpoint where students may be having problems with the material. For instance, if the K-level of the objective is K4: Apply, and the student is having problems applying the information, a K1: Recognize MC item may help you determine if the students could not apply the information, because the students could not remember the information.

• Under no circumstances should the K-level of the MC item be higher than the objective's K-level. For instance, if the K-level of the objective is K1: Recognize, and a K4: Apply MC item is used, the students will be assessed on information that is beyond the objective and not presented to them in the course.
  

2. Determine whether it would be appropriate to write a multiple-choice item for the objective.
   For K2: Recall and K5: Synthesize, MC items may not be appropriate since they cannot fully assess these K-levels.
   

3. Locate a reference source for the multiple-choice item.
   If the information required to achieve the learning objective is contained in an identifiable document, refer to the information when writing the item and note the reference source. This will be useful for verification purposes in future reviews.

   While citing reference sources, avoid taking items directly verbatim from the materials. Students may be able to memorize the material without understanding it.
   

4. Write the stem of the multiple-choice item.
   When writing a MC item stem, be sure that it contains only one central thought and that it is written to the specified K-level. Writing an item to a specific K-level can be tricky and may require some trial and error. Use the following chart as a reference when writing the MC item stem.

General rules for multiple-choice item stem creation
The stem of the MC item should:

• Test only one idea or central thought.

• Be clear and unambiguous, but still contain all information, conditions, assumptions, and details to answer the question correctly. The knowledgeable student should be able to answer the question without reading the alternatives.

• Lead to only one alternative.

• Include certain words or phrases if all alternatives require them.

• Be phrased positively instead of negatively. If a negative must be used, it should be highlighted (in caps or underlined) so that the student will notice it and interpret the item correctly.

• Reference an illustration by its figure number if it uses an illustration on a separate sheet of paper.

• End in a question mark if it is in the form of question and is a complete sentence.

• Preferably be in a completed question form rather than in an incomplete statement form except when it would make the MC item grammatically clumsy or difficult to understand. In general, the question form is easier to construct than the incomplete sentence form.

• Have a maximum of one completion position (blank) either near or at the end of the stem (if an incomplete statement is required).

• Have emphasis (underline, bold, and/or all caps) on any words (usually adverbs or adjectives) that completely change the meaning of the question. For example, "Which is the MOST likely to…"

Source NAVEDTRA 130A, Volume I. 8-B-3-1 to 8-B-3-8

5. Determine the correct answer (alternative).
   The alternatives of the MC item are designed to either answer the question or complete the statement in the stem. There should be only one correct alternative. The correct alternative should be clearly correct. Now that the stem has been created, write the answer in a form that matches the item stem. This correct alternative will be used to determine the appropriate format and content of the distracters.
   

6. Write the incorrect alternatives (distracters) for the multiple-choice item.
   Each incorrect alternative should be plausible (reasonable or believable) but clearly incorrect. Use the content contained in the correct alternative to guide the content of the incorrect alternatives. The content of all the alternatives should be similar. Also, all of the alternatives should have the same format, structure and grammar. When writing the distracters, it may be helpful to use the corrective alternative and find ways to make it incorrect.

   Typically there are four alternatives for each MC item. However, due to the content of the item, it may be more practical to have more than or less than four alternatives. Note that decreasing the number of alternatives for an item increases the guessing factor. For example, if the MC item contains four alternatives, then a student has a 25% (1 in 4) chance of getting the item correct by guessing. If this same item only has three alternatives, the student has a 33% (1 in 3) chance, a higher chance, of correctly guessing the correct alternative.

   When selecting the order of the alternatives, position the correct answer among the alternatives by a random selection process to avoid any patterns, which may bias the test. However, when the MC items involve numerical answers, the alternatives must be arranged in ascending or descending order.
   

General rules for creating the alternatives for a multiple-choice item
The alternatives of the MC item should:

• Fit well with the stem. The grammar, structure, and punctuation should make sense and conform to the grammar, structure, and punctuation of the stem.

• Have only one correct answer. Distracters should be plausible but clearly incorrect. A distracter should not be subject to automatic elimination by the students because they are irrelevant or unrelated to the question. A good rule is to develop distracters based upon common misconceptions and errors. Distracters may be prepared based on how students might incorrectly manipulate terms, symbols, theories, and ideas. Look at the correct answer and determine how it may be made incorrect.

• Be closely related to the other alternatives. The difficulty of the item will depend largely upon the alternatives. The more closely related the alternatives are, the more difficult it is to select the correct answer. This reduces the chance of correctly answering the item by guessing.

• Be of approximately the same length and complexity and be expressed in similar form.

• Not use interrelated answers, such as C is true if A and B are false, all of the above, and none of the above.

• Avoid negative wording, which is confusing-however, if used, highlight negative wording by capitalizing, underlining or italicizing.

• Not contain words such as always, never, or not.

• Use vocabulary that is familiar or can be explained within the limits of the MC item.

Source NAVEDTRA 130A, Volume I. 8-B-3-1 to 8-B-3-8

7. Correctly format the MC item.
   There are several ways to format a MC item. These formats are based on the MC item's stem type. The formats are closed stem as a question, closed stem as an incomplete statement, and open stem.

   Select the multiple-choice format by clicking on the link below for more information.

• Closed stem as a question
• Closed stem as an incomplete statement
• Open stem
  

8. Review the item.
   After writing the item review it for content, formatting, grammar, and whether the item is written to the intended K-level. The "Quality Checklist for Multiple-Choice (MC) Item Construction" provides a guideline for this. Again, it is important to ensure that the actual K-level of the item is not above the K-level of the learning objective.

Back to top

How do I recognize poorly written MC items?
Writing test questions that challenge the student to correctly answer questions based on their knowledge rather than their guessing ability is easier when some common errors are avoided. Below are some examples of common errors and how to make them better. The incorrect pieces are underlined.

Common error 1:
Using similar wording in both the stem and the correct alternative suggests the correct answer.

Common error 2:
Stating the correct alternative in greater detail than the other alternatives often cues the correct answer. All alternatives should be about the same length and have the same amount of detail.

Common error 3:
When the grammar and structure of the stem does not match all of the alternatives, the non-matching alternatives may be discarded by the student. In the example above the correct alternative (A) is the only alternative that makes sense when read with the stem.

Common error 4:
Using two or more alternatives with the same meaning eliminates them as useful distracters and simplifies the choices. If two or more alternatives have the same meaning and are both correct, this is also incorrect because there can only be one correct answer.

Common error 5:
Using alternatives that are included in other alternatives causes confusion for the student. In the example below, alternative b includes alternative a. Therefore, if alternative b is correct, alternative a is also correct.

Common error 6:
If all of the alternatives are not plausible, the item becomes easier and a person guessing the answer has a better chance of getting the item correct. In the example below, the student can eliminate the last alternative and now has a 1 in 3 chance of selecting the correcting answer.

Common error 7:
If the item is negatively stated, the negative word or phrase should be highlighted (in caps or underlined) so that the student will notice it and interpret the item correctly. Always use "except" instead of "not." In the example below, "except" should be written as "EXCEPT," "EXCEPT," or "except."

Common error 8:
Do not use repetitive words. This allows for the stem and the alternatives to be more clearly stated.

Common error 9:
Do not use "not" in the stem and do not use double negatives. In the example below the use of "not" twice makes the MC item confusing. In this case the item is more difficult because the item cannot be interpreted correctly.

Back to top

How do I assign Item Status, Status on Compromise, and Randomize Answers?

• Item Status
• Status on Compromise
• Randomized Answers
• Last Used on Test

Item status
Item status is used to identify the status of the item. Only items with a status of active or validation may be used on an exam. The five item statuses are active, inactive, validation, development, and review.

Active item status is used when an item has met the standard criteria of 100 samples and a minimum difficulty index of 0.4. Validation status is used when the item has been approved to be on exams, but has not met the active status criteria.

Source: CNET. Question Status. Retrieved May 10, 2002 from http://wwwnt.cnet.navy.mil/cetars/help/glossary/question_status.htm, http://wwwnt.cnet.navy.mil/cetars/help/glossary/active_test_question.htm, http://wwwnt.cnet.navy.mil/cetars/help/glossary/validation.htm

Status on Compromise
Status on Compromise pertains to test compromise. Test compromise is an unauthorized disclosure of a test and/or answers to a test. Compromised test questions create situations where unqualified students may pass knowledge tests. These students may then take performance tests without the basic knowledge of the equipment and may injure themselves, other personnel, and/or equipment.

Test compromise may be suspected when the difficulty index of an item changes drastically from the previous offerings of the course. For example, if the difficulty index of an item is usually 0.5 (1/2 of the students answer the item correctly) and on one test, all the students answer the item correctly (difficulty index (DIFF) = 1.00), compromise of the item may be suspected. If compromise is suspected, the MC item should be marked as "Suspected" for "Status on Compromise."

Items that are suspected of being compromised should be taken out of the item pool when marked as suspected. These items can be replaced with different items. Retaining the suspected item ensures that a similar item will not be added to the active test bank.

You can reduce the possibility of compromise during test review by not reading missed items and answers verbatim. In addition, have alternate versions of the test. The rule of thumb to determine the number of versions is to have enough test versions so that two classes that are on board at the same time are administered two different versions.

Randomized Answers
Randomized Answers is for changing the listed sequence of alternatives when an MC item list is generated. Randomizing answers allows the same multiple-choice item to vary across different tests, because the order of the alternatives varies. It is a good idea to randomize answers, but items should not be randomized when there is a numerical or sequential order to the alternatives.

Example:

What is the operating time, in seconds, for the pressurization/compensation blow valve to roll from shut to open?

a. 1 to 3
b. 4 to 6
c. 7 to 8
d. 9 to 11

In the example above, the alternatives are in a numerical or sequential order, as they should be.

Example:

What is the operating time, in seconds, for the pressurization/ compensation blow valve to roll from shut to open?

a. 7 to 8
b. 1 to 3
c. 9 to 11
d. 4 to 6

In the example above, the alternatives are randomized and they should not be. They should be listed in sequential order.

Last Used on Test
Last Used on Test is a textbox for noting when the item was last used on a test. This can be used when generating an item list. In some situations, it may not be useful to use the item without a time gap between uses. For instance, when generating a comprehensive test, do not use the same item that was used on a knowledge progress test. Using the same item may increase the likelihood of item compromise. Secondly, students will be tested on their ability to memorize the answer to the items, and not on their learning.

Back to top

How do I re-assign multiple-choice items from deleted objectives?
When objectives are deleted from the course, the multiple-choice (MC) items for those objectives remain in "Charting a Course" until the MC items are deleted. This allows for the archiving of MC items and/or the reuse of MC items with other similar learning objectives in the course. With minor changes and editing, some MC items may be revised, assigned, and used with another learning objective. This decreases development time. For K2: Recall and K5: Synthesize, multiple-choice items may not be appropriate. Multiple-choice items cannot fully assess these K-levels.

When editing and re-assigning MC items it is important that:

• The new objective is similar to the old objective.
• The MC item assesses the information covered in its new learning objective.
• The K-levels of the MC item and the new learning objective are the same.
• The Item Status, Status on Compromise, and Randomize Answers, Last Used on Test are updated for the item.

Once the MC item is reassigned and edited, the item will appear on the View Item List for its new learning objective.

Back to top

How do I align K-levels?
K-levels (knowledge levels) indicate the different ways that knowledge is used on the job. The purpose of assigning K-levels to knowledge objectives is to facilitate the choice of appropriate instructional strategies and in the designing of appropriate MC items to measure the learning and mastery of these objectives.

K-level alignment is an evaluation process in which the K-level of the multiple-choice (MC) item is determined and compared with the K-level of the item's learning objective. Ideally, the K-level of the item should equal the K-level of its objective.

For instance, if the K-level of the objective is K4: Apply, a K1: Recognize MC item will not fully assess the objective because the students will be recognizing information and not applying information.

However, items with lower K-levels may be used to assess the learning objective if a series of items is used. For objectives with K-3, K-4, or K-5 levels, items written below the learning objective's K-level can be used to pinpoint where students may be having problems with the material.

For instance, if the K-level of the objective is K4: Apply, and the student is having problems applying the information, a K1: Recognize MC item may help you determine if the students could not apply the information, because the students could not remember the information.

However, under no circumstances should the K-level of the MC item be higher than the K-level of its objective.

For instance, if the K-level of the objective is K1: Recognize, and a K4: Apply MC item is used; the students will be assessed on information that is beyond the objective and not presented to them in the course.

If the K-level of the MC item is higher than the learning objective's K-level, the MC item may be testing material that was not covered and is not expected from the students. In order to prevent this from occurring, the MC items should be continuously checked for K-level alignment, especially if:

• The objective has been changed or edited
• The information presented for the objective has been changed
• The K-level of the objective has been changed
• New MC items have been added to the item bank

• What the item is asking the students to do in order to answer the item correctly.
• How the information will be presented to the students.

Use the following table to assist you when assigning K-levels.

Here are some examples of MC items and their K-levels.

A K-level Alignment Overview results when all of the items for an objective have been assigned a K-level. The table lists the amount (quantity) of MC items that were rated at each K-level. If any item has been assigned a K-level above the K-level of the objective, a red exclamation point appears in the table and the list.

Ideally, the K-level of the item should equal the K-level of its objective, but items with lower K-levels may be used to assess the objective if a series of items are used. For objectives with K3: Comprehend, K4: Apply, or K5: Analyze/Synthesize/Evaluate levels, some items for an objective may be written below the objective's K-Level to discover where students may be having problems with the objective. However, under no circumstances should the K-level of the MC item be higher than the K-level of its objective.

Here is an example of a K-level Alignment Overview Table.

Example:

In the example table above, there are a total of 11 (2+3+4+2=11) MC items for the learning objective. Two MC items are rated as K1: Recognize, three MC items for K3: Comprehend, four MC items for K4: Apply and two MC items for K5: Analyze/Synthesize/Evaluate. If the learning objective's K-level is K4: Apply, then the two items that were rated as a K5: Analyze/Synthesize/Evaluate should be reevaluated and/or edited to fit the K-level of the learning objective.

What are additional sources of information on multiple-choice item construction?
The following is an additional non-Navy source of information on multiple-choice item construction:

Oosterhof, A. (2001). Classroom applications of educational measurements, 3rd Ed. Upper Saddle River, NJ: Merrill Prentice Hall, 87-251.

Back to top