How do I conduct an assessment analysis?

What is an assessment analysis and why should I conduct one?

What is an assessment category and how do I select one?

What are assessment types and how do I select them?

What are examples of assigned assessment categories and assessment types?

What are additional sources of information on assessment analysis?

What is an assessment analysis and why should I conduct one?
An assessment analysis determines how student achievement of the learning objectives (LO) will be evaluated. In an assessment analysis, each learning objective is assigned an assessment category and at least one assessment type. This ensures that all LOs are assessed in the appropriate manner for determining if the learner has attained the skills and knowledge specified by the learning objectives.

Back to top

What is an assessment category and how do I select one?
There are two types of assessment categories: performance and knowledge. These categories are determined by the expected result of the learning objective. In general, if the learning objective is a skill then it should be assessed on a performance test. If the learning objective (LO) is knowledge, then it should be assessed on a knowledge test. This is the general rule; however, due to constraints, there are exceptions.

Click on the steps below to guide the selection of an assessment category.

1. Determine if the learning objective is ideally assessed on a performance test or a knowledge test.

2. Determine if the ideal assessment category is impractical due to constraints.

3. Determine if the selected assessment category is appropriate in regards to the other related learning objectives.

1. Determine if the learning objective is ideally assessed on a performance test or a knowledge test.
   In a perfect world without any constraints, the assessment category should be based on the expected student outcome of the learning objective. Identify whether the learning objective is a skill or knowledge. The learning objective should have been previously assigned in the Objectives module and can be identified in the S/K column of the Assessment Analysis Tab. If the LO is a skill then ideally it should be assigned to a performance test. If the LO is knowledge then it ideally should be assigned to a knowledge test.

   Performance tests measure a student's ability to perform a specific skill or behavior by using actual equipment or training devices. Performance tests can be a process test, a product test, or a combination of the two.

   Source: NAVEDTRA 130A, Volume I, 8-1-1
   
   
   • Process tests assess whether the student can correctly perform the steps of the procedure or process.
     
     
   • Product tests assess whether the student can produce or construct a product that meets required specifications.
     
     
   • Combination tests assess whether the student can both perform the required procedural steps and a product that meets required specifications.
     
     Source: NAVEDTRA 130A, Volume I, 8-A-3-1
     

A performance checklist based upon the learning objectives is usually used to evaluate the performance test. It is used to determine if the required steps were or were not performed, or if the characteristics of the final product were present or absent. A performance checklist may contain:

• The sequence of steps of the process being evaluated.

• The required specifications or characteristics of the final product being evaluated.

• Both the sequence of steps of the process being evaluated and the required specifications of the final product.

The following is an example of a learning objective that would be ideally assessed on a performance test.

Example:

Troubleshoot a piece of radar equipment.

In the example above the outcome of the objective requires the students to perform a series of steps in order to troubleshoot the equipment correctly therefore would be best measured on a performance test.

The following table shows what a performance checklist for this example may look like. The table contains three critical steps in the sequence of troubleshooting a piece of radar equipment. The students are evaluated on whether they were able to perform each of the steps.

Example:

Knowledge tests are used to measure the student's achievement of theory and/or background knowledge in support of performance of a skill. Specifically, they test whether the student:

• Has acquired the necessary knowledge to do the process and/or make the product.
  
• Understands the associated safety/hazard precautions.
  
• Can use the technical documentation.

Knowledge assessments include knowledge progress tests and comprehensive tests. Knowledge tests can be comprised of multiple-choice items, matching, fill-in-the-blank, short answer items, and essay questions.

Source: NAVEDTRA 130A, Volume I, 8-A-4-3, 8-A-5-1, 8-1-1

• Knowledge progress tests are tests given to the students during the course to determine how well they are progressing toward the achievement of the objectives. There may be multiple progress tests during the course.
  
  Source: NAVEDTRA 135B, J-8
  
  
• Comprehensive tests are used to measure the mastery of critical objectives and/or the retention of previously tested objectives. Comprehensive tests can occur within-course and/or as a final comprehensive test at the end of the course. They may be a performance test or a knowledge test.
  
  Source: NAVEDTRA 135B, J-2

The following is an example of a learning objective that would be ideally assessed on a knowledge test.

Example:

Describe the safety precautions that need to be observed while operating and troubleshooting an Integrated Weapons System.

In the example above, the outcome of this objective requires students to explain the safety precautions used when working with an Integrated Weapons System. Being able to explain the safety precautions of an Integrated Weapons System, prior to actually working on them is critical for the protection of the students, others, and the equipment itself. Therefore, this objective would ideally be measured by a knowledge test.

2. Determine if the ideal assessment category is impractical due to constraints.
   A constraint is any situation that prevents the testing of the LO as stated. Testing constraints may be limited manpower, limited equipment, limited space, limited time, and so on. When performance testing a group of individuals, it is important to evaluate the individual's performance and/or the individual's participation within the group. In other words, along with evaluating the process performed or product produced by the team as a whole, it is important to evaluate each member of the team individually. Each member should have positively contributed to the team and performed their role. As a result, depending on the circumstances, group testing may reduce or increase the amount of constraints. For instance, group testing may take less time than individual testing, so any time constraint may be eliminated. On the other hand, if the group testing is complicated, additional instructors may be necessary to fully evaluate the performance. Thus, the group testing increases personnel constraints.

   An example of a performance test constraint is when the performance test calls for the student to troubleshoot a piece of radar equipment, but the radar equipment is not available. In this scenario, a knowledge test should be used instead of a performance test, because the constraint prevents a performance test.

3. Determine if the selected assessment category is appropriate in regards to the other related learning objectives.
   Lower level objectives may be assessed during the testing of higher-level objectives. In other words, by testing one objective, it is possible to simultaneously test another objective. This is important because lower level knowledge learning objectives may be assessed on performance tests.

   Example:

   If the student is expected to recognize the major parts of radar equipment (knowledge item), and the student must also troubleshoot a piece of radar equipment (performance item), it may be possible to assess both LOs at once by conducting a performance test. Before, during, or after the troubleshooting, the student can state or show the major areas of the radar equipment.

   However, due to safety or cost, certain knowledge objectives should measured prior to assessing the performance objectives. For example, an instructor would want to assess whether a student is able to describe what are the safety precautions required when inspecting a service pistol (knowledge item) prior to being assessed on the process of inspecting a service pistol (performance item).

Back to top

What are assessment types and how do I select them?
Assessment type is the specific method used to test the skill or knowledge specified by a learning objective (LO). There are two major types of assessment types: formal and informal. For many courses it is impossible to formally test every single learning objective. However, it is important to try to assess each LO by some manner.

Formal assessments are used to determine student achievement of the course objectives. There are three main types of formal assessments:

• Knowledge progress tests
  
• Performance progress tests
  
• Comprehensive tests

These types affect a student's standing in a course and must be grade weighted.

Pretests are considered a special type of a formal assessment. They are used to determine student achievement of course objectives at the beginning of a course, unit, or lesson. These are used to assess incoming knowledge and skills. While they are considered a type of formal assessment, they do not affect a student's standing in a course and are not grade weighted.

Decisions regarding the type of formal assessment are made in the Scheduling and Grading Tab.

Informal assessments should be used to assess all low critical objectives that will not be formally assessed. Informal assessments may also be used in addition to formal assessments to assess highly critical LOs Informal assessments may or may not be graded. If informal assessments are used in calculating the course grade, they must be standardized. Informal assessments can also be used to provide extra practice and instructor feedback for the learning objective. There are four types of informal assessments:

• Homework is any activity or assignment that is done outside of the class time.

• Quizzes are short tests used by the instructor to measure achievement of material recently taught.

  Source: NAVEDTRA 135B, Appendix J-9
  

• Practical work is a lab assignment and/or an in-class assignment.

  Source: NAVEDTRA 130, Volume I, 8-3

• Question and answers are questions asked to the students during the course of the lesson and/or activity.

Use the steps below to guide the selection of the appropriate assessment type(s).

1. Review the criticality scores of the learning objectives.
   
2. Determine if the learning objective should be formally assessed.
   
3. Determine if informal assessments should be used.
   
4. Determine the type of informal assessment that should be used.
   
   

1. Review the criticality scores of the learning objectives.
   Comparing criticality is one method of determining which learning objective should be formally assessed. Criticality is calculated by rating high, medium, or low in the following areas: Safety, performance frequency, importance to on-the-job performance, and importance to overall course mission.
   

• High criticality indicates that the skill or knowledge is critical to safety, frequently applied in a performance setting, of high importance to job performance, and/or an integral part of the course mission.
  
  Source: NAVEDTRA 130A, Volume I, 8-3-4

• Medium criticality indicates that the skill or knowledge is of secondary importance to safety, performed on an occasional basis, influences job performance, and/or influences ability to achieve the overall course mission.
  
  Source: NAVEDTRA 130A, Volume I, 8-3-4

• Low criticality indicates that the skill or knowledge is only remotely significant in ensuring safety of personnel and equipment, is rarely applied in a performance context, has little influence on job performance, and/or is of minimal importance to the course mission.
  
  Source: NAVEDTRA 130A, Volume I, 8-3-4
  
  See Objectives Criticality How To… for more information on calculating criticality scores.

2. Determine if the learning objective should be formally assessed. Use the criticality scores to determine which learning objectives should be formally assessed. In "Charting a Course," the administrator has set a minimum cutoff criticality score. Any LO that scores higher than the minimum cutoff score is automatically designated for formal assessment.
   
   If a learning objective has not been automatically selected for formal assessment you may still choose to formally assess it. For instance, you may choose to formally assess an LO that has a low overall criticality score because it has a high on-the-job performance score, because it is i mportant to the course mission, or because you cannot assess it though the testing of other LOs for the course.

3. Determine if informal assessments should be used.
   Informal assessments may be used for many reasons. However, due to constraints, such as time and resources, it is inappropriate to use every type of informal assessment for every objective. Informal assessments are useful when:

• The LO has not been scheduled for formal testing. All Los should be assessed in some manner. Any LO that is not formally tested should be informally assessed.

• The LO is for a skill (S) that has a knowledge component, which will not be formally assessed. For instance, if an LO has a knowledge component that will be assessed in a performance test, it is useful to informally assess the knowledge component and provide feedback to the student before the performance test is given.

• The LO has a history of being difficult. If a learning objective has been difficult for past students, an informal assessment will provide both students and the instructor with information about the student learning, deficiencies, and misconceptions. The feedback should be used to correct the student's errors before the formal assessment occurs.

• The LO has a high criticality score. In general, the importance of learning the LO increases with the criticality score. Extra practice and feedback through informal assessments will increase student achievement.
  

4. Determine the type of informal assessment that should be used.
   The type(s) of informal assessments should depend on whether the informal assessment would contribute to the course grade. Additionally, constraints and resources (course time, activity length, budget, and personnel) will influence the selected informal type. For instance, if a piece of radar equipment is needed to do the activity, homework may not be ideal, because the students cannot take the equipment home with them. In this case, practical work and/or question and answer may be appropriate.
   

Back to top

What are examples of assigned assessment categories and assessment types?
For the examples given below, the students are novices to SCUBA diving. They do not have any experience with the equipment or the process. The administrator has set the criticality cutoff score to 5 for the course.

Select the objective below by clicking on it for a more detailed explanation.

This learning objective is not automatically selected for formal assessment, because its criticality score of 4 is less than the administrator's cutoff score of 5. Although the criticality score for this learning objective is 4, the on-the-job performance score is medium. Thus, this LO should be formally assessed, because the students are inexperienced and the on-the-job performance is important. This learning objective should be knowledge tested, because the novice student is expected to recall general information about dive planning. An informal assessment can be used to provide practice and feedback to the inexperienced student. If an informal assessment is used, question and answer may be used during the instruction.

This learning objective should be performance tested, because the student will be producing an actual product for evaluation. The criticality level is 3. Due to the low criticality score, the LO should be informally assessed. Homework and practical work will provide practice in the classroom (practical work) and out of the classroom (homework) in preparing the report.

This learning objective should be knowledge tested, because the novice student is expected to recall general information about the various scales. The criticality score for this learning objective is 2. Due to the low criticality score, the LO should be informally assessed. Since the LO asks for stating the general contents, question and answer may be an appropriate informal assessment.

Back to top

What are additional sources of information assessment analysis?
The following is an additional non-Navy source of information on assessment analysis:

Oosterhof, A. (2001). Classroom applications of educational measurements, 3rd Ed. Upper Saddle River, NJ: Merrill Prentice Hall, pp 87-251.

Back to top