How do I conduct an Item Analysis on a Test?

What is an item analysis and what can I learn from it?

How do I interpret a Standard Training Activity Support System (STASS) Test Analysis Report?

How do I conduct an item analysis on a test?

What is an item analysis and what can I learn from it?
Through an item analysis, each item on a test and the entire test as a whole unit are evaluated based on the student responses to the items. As a result, it is possible to determine which items are poorly written and which items do not completely assess their learning objective. If an item is poorly written, a student may be able to select the correct response without knowing the material.

Back to top

How do I interpret a Standard Training Activity Support System (STASS) Test Analysis Report?
A Standard Training Activity Support System (STASS) Test Analysis Report (CSM0510R) is generated by some commands for each test that is given. The information on this report is useful in conducting an item analysis. For each item, the report details:

• The amount of students that selected each alternative.
• The alternatives selected by the students in the upper 27%.
• The alternatives selected by the students in the lower 27%.
• The correct alternative.
• The difficulty index.

The following is an abbreviated example of a STASS Test Analysis Report.

Note: Only items 1 -3 are shown. The rest of the items for the test are not displayed.

The POS or position number refers to the sequential number (or position) of the item on the test.

The RANGE identifies the part of the sample for which the information is displayed. The three ranges reported are total students (TOTAL), the upper 27% (UPPER), and the lower 27% (LOWER). The TOTAL includes the UPPER, the LOWER, and the middle 46% of the students taking the test.

• For item #1 in the example above, 14 (0+14+0+0) students answered the item. Of the 14, three students were in the upper range, and three students were in the lower range.
  

• For item #2 in the example above, 13 (0+0+0+13) students answered the item. Of the 13, three students were in the upper range, and two students were in the lower range.

• For item #3 in the example above, 14 (0+2+9+3) students answered the item. Of the 14, three students were in the upper range and three students were in the lower range.

A, B, C, and D represent the four alternatives for a multiple-choice item. Each column lists the number of students who selected that alternative for the item. An asterisk (*) indicates the correct alternative.

For item #3 in the above example, nine students selected alternative C, the correct alternative. Two students incorrectly selected alternative B and three students incorrectly selected alternative D. All of the students in the upper 27% of the test, and one student in the lower 27% selected alternative C. However, one student in the lower 27% of the test incorrectly selected alternative B and another lower 27% student incorrectly selected alternative D.

OMIT is the number of students who did not respond to (omitted) the item.

For items #1 and #3 in the above example, all of the students answered the item. However, the TOTAL row for item #2 shows that one student did not answer this item. This same student is also in the lower 27%, because there is a 1 in the LOWER row.

DIFF is the difficulty index of the item. The difficulty index refers to the proportion of students who answered the item correctly. The difficulty index is calculated by dividing the number of students who answered the item correctly by the total number of students taking the test. The difficulty index ranges from 0.00 (no one answered item correctly) to 1.00 (everyone answered item correctly). Thus, the larger the difficulty index, the easier the item. The acceptable range of difficulty for technical training is .50 to .90. Sometimes a difficulty of 1.00 may be desirable, such as in the area of safety, where it is crucial that everyone knows the information.

• For item #1 in the example above, all 14 students selected the correct alternative. Thus 14 divided by 14 is 1.00, the difficulty index for item #1.

• For item #3 in the example above, nine students answered the item correctly. Thus nine divided by 14 is .64, the difficulty index for item #3.

QUESTION is the item number of the item from the test bank.

Source: CNET/USN. CSM 0510R TEST ANALYSIS REPORT. Retrieved August, 17, 2001 from http://www.cnet.navy.mil/netpdtc/intrpd/stasshelp/csm_rpt/csm0510r.htm.

Back to top

How do I conduct an item analysis on a test?
There are four major steps in conducting an item analysis.

1. Identify the test conditions.
Test conditions may include situations/conditions, which may have influenced the results of the test.

2. Conduct an item analysis by interpreting the results from a Standard Training Activity Support System (STASS) Test Analysis Report.
You will answer nine questions that will assist you in finding patterns in the data. These patterns will help you evaluate each item on the test.

3. Interpret the results from the item analysis.
From the nine potential patterns identified in step 1, you will identify three major patterns

4. Identify areas for course review and student performance from the interpretation of the item analysis.
From the three major patterns, you will identify opportunities for course review and student performance improvements.

1. Identify the test conditions.
Test conditions may include situations and conditions that may have influenced the results of the test. In this section, provide a summary of the administration of the specified test. Please indicate location of the test administration, the number of people tested and any unusual situations and conditions that occurred that may have interfered with the administration of the test. These unusual situations and conditions may or may not have caused interruptions or cancellations to the test. Such occurrences may include:

• Emergency situations requiring the evacuation of the testing room.
• Outside disruptions.
• Unauthorized personnel entering the testing room during the test.
• A test taker being disruptive during the test.
• A test taker becoming ill or otherwise having to leave the room during the test.
• A test taker deviating from acceptable conduct during the test.

Test conditions are important because they may influence the results of the test. As a result, the item analysis may be slightly affected but results in an "inaccurate" test. It is important to constantly review the test conditions as you are interpreting the item analysis.

2. Conduct the item analysis.
When conducting the item analysis in "Charting a Course", answer the nine questions to guide the evaluation of the STASS Test Analysis Report. These questions are related to the test and the test items. The answers to these questions serve as the starting point for the analysis and revision of tests and their items.

Question 1. Is there a pattern of correct answers throughout the test?

Question 2. Which items, if any, were missing by all or almost all of the students?

Question 3. Which items, if any, were omitted by a high percentage of students?

Question 4. Which items, if any, have an incorrect alternative chosen by a high percentage of high-achieving students?

Question 5. Which items, if any, have a difficulty index smaller (closer to 0.00) than the other items for this objective?

Question 6. Which items, if any, were answered correctly by more low-achieving students than high-achieving students?

Question 7. Which items, if any, were missed by none or almost none of the students?

Question 8. Which items, if any, have incorrect alternatives that were NOT selected?

Question 9. Which items, if any, have a difficulty index larger (closer to 1.00) than the other items for this objective?

Question 1. Is there a pattern of correct answers throughout the test?
Look at the correct answers to make sure that they do not form a predictable pattern. For instance, when using an answer sheet that will be optically marked or graded, do the answers form diagonal lines, straight lines, and/or curves?

Question 2. Which items, if any, were missed by all or almost all of the students?
Look at the STASS Test Analysis Report and look for items in which the difficulty index is very small (closer to 0.00) and/or the correct alternative for the item was rarely selected.

Example:

In the example above, the DIFF index is 0.07 and only 1 student selected alternative D, the correct alternative.

Question 3: Which items, if any, were omitted by a high percentage of students?
Look at the STASS Test Analysis Report and look for items in which the OMIT column has more than one or two in the TOTAL row. Additionally, if a large percentage of OMITs fall in the UPPER range, this may further illustrate that a problem exists.

Example:

In the example above, the OMIT column has 3 in the total row indicating that three of the 14 persons taking the test omitted this question. Out of the three persons, two persons were in the UPPER range.

Question 4: Which items, if any, have an incorrect alternative chosen by a high percentage of high-achieving students?
Look at the STASS Test Analysis Report and look for items in which an incorrect alternative has been overly selected by the students in the upper 27%. Approximately the same number of students should have selected each incorrect alternative. If this is not the case, then there may be a problem.

Example:

In the example above, four of those in the UPPER range responded to option C an incorrect response.

Question 5: Which items, if any, have a difficulty index smaller (closer to 0.00) than the other items for this objective?
Look at the STASS Test Analysis Report and compare the difficulty indexes of items that measure the same objective. The difficulty indexes should be approximately equal. Look for items in which the difficulty index is closer to 0.00 than any other item for the objective.

Example:

The above example shows four items on a test that come from the same objective. Item 4 has a difficulty index (DIFF = 0.20) that is smaller than the other items for the objective.

Question 6: Which items, if any, were answered correctly by more low-achieving students than high-achieving students?
Look at the STASS Test Analysis Report and look for items in which the correct alternative has been selected by more students in the lower 27% than by students in the upper 27%. If this is not the case, then there may be a problem.

Example:

In the above example, 3 persons in the LOWER range answered option C which is the correct answer. No one in the UPPER range answered this option.

Question 7: Which items, if any, were missed by none or almost none of the students?
Look at the STASS Test Analysis Report and look for items in which the difficulty index is very large (closer to 1.00) and/or the correct alternative for the item was almost always selected. Also, look at the students' selection of alternatives and note any patterns of incorrect alternatives. There may or may not be a problem here.

Example:

In the above example, the difficulty index (DIFF) of the item is 1.00 indicating no one missed this item.

Question 8: Which items, if any, have incorrect alternatives that were NOT selected?
Look at the STASS Test Analysis Report and look for items in which all of the incorrect alternatives have not been selected. Approximately the same number of students should have selected each incorrect alternative. If this is not the case, then there may be a problem, especially if the difficulty index is nearer to 0.00 than 1.00.

Example:
 

In the example above, no one chose option C.

Question 9: Which items, if any, have a difficulty index larger (closer to 1.00) than the other items for this objective?
Look at the STASS Test Analysis Report and compare the difficulty indexes of items that measure the same objective. The difficulty indexes should be approximately equal. Look for items in which the difficulty index is closer to 1.00 than any other item for the objective.

Example:

The above example shows four items on a test that come from the same objective. Item 4 has a difficulty index (DIFF = 0.95) that is larger than the other items for the objective.

3. Interpret the item analysis.
Once the item analysis has been conducted, the results from the nine questions must be reorganized and interpreted. The results are organized into four sections: 1)Test Conditions, 2) Step 1: Length of time allotted to take test, 3) Step 2: Items missed by most or almost all the students, 4) Step 3: Items missed by none or almost none of the students

Test Conditions
The reported test conditions are displayed to provide a context for interpreting the results of the item analysis. The test conditions may affect the results of the item analysis and it is important to consider them.

Step 1: Length of time allotted to take test
The results from questions 2 and 3 from the item analysis are used to help assess whether the students had an enough time to complete the test. By looking at the results from these questions, determine if there are questions at the end of the test or scattered throughout the test that the students are skipping.

Step 2: Items missed by most or almost all the students
The results from questions 2, 4, 5, and 6 from the item analysis are used to help assess the reasons why most or almost all the students are missing certain items. As indicated in Conducting Item Analysis, there can be multiple reasons why a particular item is flagged based on these questions. In this section, items are reviewed by content experts to pinpoint the probably reasons and determine whether the item needs to be revised.

For the items in this section, you will need to 1) check the items for any miskeys, 2) review the items with a panel of content experts, and indicate whether any items need revisions.

First, check the items for any miskeys. Prior to reviewing the items, you will want to check for any potential miskey. Make sure that the correct alternative is correctly marked on the answer key. If an item has been miskeyed, you will need to: 1) correct the miskey and 2) review the revised item statistics to see if it is flagged based on any of the other item analysis questions. If after fixing the miskey the item is no longer flagged, then the item does not have to be reviewed.

Second, review the items with a panel of content experts. Conduct an item review meeting for the flagged items; evaluate each item based on the Quality Checklist for Multiple-Choice (MC) Item Construction. Answering the following questions will help guide you through the review.

When working with tests that have higher stakes (high consequences for performing poorly on the test), try to have a panel of at least five experts reviewing the item. It is recommended to have the experts review the item as a group so that the item can be discussed from different perspectives and all can agree on any revisions that may be needed.

Finally, indicate whether any items need revisions. When completing the item review meeting, indicate the items that need revision in the appropriate space in step two of the Interpreting Item Analysis screen.

Step 3: Items missed by no or almost none of the students
The results from questions 7, 8 and 9 from the item analysis are used to help assess the reasons why no or almost none of the students missed certain items. Ideally, this may have occurred because each student clearly understood the objective; however prior to making that interpretation, verify that either the test has not been compromised or there is nothing within the item or the test that is giving away the answer to the question.

Like in Step 2, this section is used to determine which items, if any, need to be reviewed and/or revised by content experts. For the items in this section, you will need to 1) determine if any items have been compromised, 2) determine if items on the test give away the answer to any of the flagged question, 3) review the items with a panel of content experts, 4) indicate whether any items need revisions.

First, determine if any items have been compromised.
Test compromise may be suspected when the difficulty index of an item changes drastically from the previous offerings of the course. For example, if the difficulty index of an item is usually 0.5 (1/2 of the trainees answer the item correctly) and on one test, all the trainees answer the item correctly (difficulty index (DIFF) = 1.00), compromise of the item may be suspected.

If compromise is suspected, the MC item should be marked as "Suspected" for "Status on Compromise" in the MC Item Construction tab. Items that are suspected of being compromised should be taken out of the item pool when marked as suspected. These items can be replaced with different items. Retaining the suspected item ensures that a similar item will not be added to the active test bank.

Second, determine if items on the test give away the answer to the flagged question.
Review all the items on the test and see if any of them provide a clue that gives away the answer to the item under review. If so, you should 1) remove item in question, 2) replace the item with another item in the item bank covering the same learning objective, 3) review the new item to verify that it does not provide clues to other items on the test and vice versa.

Third, review the items with a panel of content experts.
Conduct an item review meeting for the flagged items; evaluate each item based on the Quality Checklist for Multiple-choice (MC) Item Construction. Answering the following questions will help guide you through the review.

When working with tests that have higher stakes (high consequences for performing poorly on the test), try to have a panel of at least five experts reviewing the item. It is recommended to have the experts review the item as a group so that the item can be discussed from different perspectives and all can agree on any revisions that may be needed.

Finally, indicate whether any items need revisions.
When completing the item review meeting, indicate the items that need revision in the appropriate space in step three of the Interpreting Item Analysis screen.

4. Identify areas for course review and student performance.
Now that the item analysis has been conducted and interpreted, enter notes that may be helpful in evaluating student performance or to use when the course is under review. The focus of this section is looking at how the students performed on the objectives as a whole. When reviewing this section, look at the objectives that contain many items that were missed by a high percentage of students. Use the following questions to guide you as you are reviewing the items.

• How did the students perform on this item compared to the learning objective as a whole?
• Were most or all the students missing all or most of the items in the learning objective?
• Is the learning objective particularly difficult to learn?
• Were the students able to grasp the objective when it was presented in the course?
• Were students lacking in necessary prerequisites to accomplish this objective?
• Was there ample time in class to present the learning objective?
• Was there ample time for the students to practice?
• What was the time elapse from the time of teaching the learning objective and taking the test?
• Are the students able to make linkages between lessons or topics?

Back to top