How do I generate a
multiple-choice (MC) item list?
What
is a multiple-choice item list?
What
does a sample multiple-choice item list look like?
What
is an answer key and what does a sample answer key look like?
What
are points to consider when generating a multiple-choice item
list?
What
is a multiple-choice item list?
The multiple-choice (MC) item list includes all selected MC
items from a course. In other words, only MC items selected
by the user will be printed or exported. List generation is
useful when selecting MC items for export to other compatible
Navy tools, selecting MC items for a paper-based review and/or
selecting MC items for a test.
When printing a MC item that was selected
for randomization, the alternatives will print in a
different sequence than what is displayed on the screen. Randomized
Answers is for changing the listed sequence of alternatives
when a MC item list is printed. Randomizing answers allows
the same multiple-choice item to vary across different tests,
because the order of the alternatives varies. Selecting an
item to have randomized alternatives is done in the MC Item
Construction Tab. It is a good idea to randomize answers,
but items should not be randomized when there is a numerical
or sequential order to the alternatives.
Example:
The alternatives of the following item
are in sequential order.
What is the operating time, in seconds,
for the pressurization/ compensation blow valve to roll from
shut to open?
a. 1 to 3
b. 4 to 6
c. 7 to 8
d. 9 to 11
If this item was selected for randomization,
the item might print like the example below, which appears
awkward. Therefore, since the alternatives form a numerical
sequence, they should NOT be randomized.
Example:
What is the operating time, in seconds,
for the pressurization/compensation blow valve to roll from
shut to open?
a. 7 to 8
b. 1 to 3
c. 9 to 11
d. 4 to 6
What does a sample multiple-choice item list look like?
The following is a sample multiple-choice item list. This
is displayed when you click the preview/print selected questions
button. Note that when choosing this list, the questions are
displayed without the answers.
Back
to top
What
is an answer key and what does a sample answer key look like?
The following is a sample answer key
for selected items. An answer key is a list of answers for
the selected multiple-choice items. The answer key may be
helpful, especially when the alternatives are randomized.
Back
to top
What are points to
consider when generating a multiple-choice item list?
When generating a multiple-choice item list, it is important
to note the purpose of the item list. If the items will be
exported to another test item bank, the item status and status
on compromise should be taken into consideration. Similarly,
if the items will be printed for a paper-based review, the
item status should be taken into consideration.
When generating a multiple-choice item
list for use on a test, it is important to:
1. Determine
the mastery level.
2. Determine
the ideal and practical amount of items needed to assess the
learning objective or content area.
3. Select
the items with the appropriate item status.
4. Check
the selected items to ensure that the information contained
in the MC items do not answer any other items in the list.
5. Check
the selected items to determine if there are very similar
items.
6. Estimate
test time.
1. Determine the mastery level.
The mastery level identifies the minimum level of understanding,
knowledge, or skill necessary in order to achieve a learning
objective or content area. Mastery learning measures how well
a student achieves the learning objective (LO), because instruction
and testing is linked to a specific LO in mastery learning.
In order to determine if a student has mastered the learning
objective, the mastery level of each LO must be determined.
Ideally, the achievement level of a LO should be 80%, but
may vary. For an 80% mastery level, if 10 items are used to
assess the LO, the student must correctly answer 8 items to
pass the LO.
Mastery level of a learning objective
may depend on:
- The overall criticality of the LO.
In general, learning objectives with higher criticality
scores have higher mastery levels.
- The safety criticality score of
the LO. In most cases, learning objectives with high safety
scores should have higher mastery levels. For instance,
if safety is a must to prevent injury or death, a 100% mastery
level should be set.
- The on-the-job performance criticality
score. If an LO is important to on-the-job performance,
especially on day one, it may be important that the student
really knows the information. As a result, the LOs with
a high on-the-job performance score may have higher mastery
levels.
- If the information covered by the
LO is essential to the learning of other learning objectives.
If new learning is built on the material covered in the
LO, it may be important that the LO is mastered
before the student begins learning the new information.
Thus, the mastery level may be higher. This varies depending
on the criticality of the current and new information.
2. Determine the ideal and practical
amount of items needed to assess the learning objective or
content area.
In general, the more items used to assess a learning objective,
the more accurate the results in evaluating the student's
learning. When assessing a LO, use as many questions as necessary
in order to determine achievement of the LO. However, due
to constraints it may be impossible to use a lot of items.
In some cases, a simple LO may only need one item in order
to determine achievement, but in other cases, asking more
items may be appropriate. For example, if testing the capitals
of the 50 states and the students are expected to know all
of the capitals, it may be difficult to have 50 items on a
test (the ideal amount). Instead, take a sample of the states,
perhaps a random 20% and/or the most difficult. This is the
practical amount. Assume that if the student correctly answers
the sample items, they know all of the capitals. As a result,
there are less than 50 items on the test. In general,
- LOs that are highly critical should
have more test items than LOs that are not as critical.
- LOs with a higher K-level may have
more items, because lower K-level items may be necessary
in order to determine where the learners are having problems.
- The amount of items should be proportional
to the amount of instructional time. For instance, If 50%
of the teaching time was spent teaching one LO or content
area, then 50% of the items on the test should be geared
toward the one LO or the content area.
- If two similar LOs supports and/or
covers the same material, fewer items may be used to assess
each LO.
3. Select the items with the appropriate
item status.
Only items designated as active or validation may be used
on a test. The amount of items selected for a learning objective
should depend on the practical amount calculated in step 2.
4. Check the selected items to ensure
that the information contained in the MC items do not answer
any other items in the list.
Item selection should not be random. The information in a
stem and/or the alternatives of one test item may contain
the answer to another test item.
5. Check the selected items to determine
if there are very similar items.
A very similar item is an item that says and asks the same
question, except it is reworded. In general, there should
not be very similar items in the list, unless the very similar
item tests new information. Item selection should not be random.
6. Estimate test time.
A majority of students should be able to complete all items
within the test time period. Estimate the time to complete
each item and use the estimation in the selection of items.
Reselect items as necessary and repeat steps 3 to 6.
Back
to top
Where
can I learn more about selecting items for a test?
The following is an
outside source that you can refer to in order to learn more
about selecting items to use on a MC test.
Block, J., Efthim, H., & Burns,
R. (1989). Building effective mastery learning schools.
White Plains, NY: Longman Inc.,135-217.
Levine, D., & Associates. (1985).
Improving student achievement through mastery learning
programs. San Francisco: CA: Jossey-Bass Publishers, 241-255.
Oosterhof, A. (2001). Classroom
applications of educational measurements, 3rd ed. Upper
Saddle River, NJ: Merrill Prentice Hall, 87-251.
Back
to top
|
