How do I collect and analyze Student Critique information?

What is the Student Critique of Training and why is it important?

How do I decide when to administer the Student Critique of Training?

How do I select a critique for my classroom type?

When should I use open-ended questions and when should I use Likert scale questions?

What are the steps in analyzing responses to the Student Critique of Training?

What is the Student Critique of Training and why is it important?
The Student Critique of Training is a questionnaire that students complete at the end of a course or at intervals during a course. It is the means by which students provide feedback to instructors, course developers, and course managers on areas such as the effectiveness of the training and the course materials, the effectiveness of the instructor, the safety of the training environment, and the condition of the training facilities. This feedback can then be used to identify areas where the course is effective and areas where the course can be improved. (Source: NAVEDTRA 135B, 5-3-1, 5-3-2, and 5-3-4)

There are seven categories of questions on the Student Critique of Training:

• Course Materials
  
• Lesson Topics
  
• Objectives and Assessments
  
• Safety and Training Facilities
  
• Instructor
  
• Safety
  
• Student Experience

This module supports a certain set of questions. You may select from six critiques based on the classroom type and type of feedback desired. The questions below will guide you in deciding which critique to administer. To view the critiques, you must have Adobe Acrobat.

Back to top

How do I decide when to administer the Student Critique of Training?
All students who complete the training should complete the Student Critique of Training. Students who do not complete the training should also be encouraged to provide feedback.

Critiques must be administered on a scheduled basis. If a course is one week or longer, you must administer a student critique at the end of the course. However, if your course is longer than a week, you may administer the critiques at interim points. Such points may be at the end of each unit, so that students can may record scores and comments during the course rather than having to recall events and perceptions at the end. You may also want to administer an interim critique after units and/or lessons that have been recently revised or updated. This will allow you to collect specific information on new portions of the course.

Source: NAVEDTRA 135B, 5-3-2

Back to top

How do I select a critique for my classroom type?
Each classroom falls into one of three types depending on the degree to which technology is used in the classroom. Choosing your classroom type will direct you to the appropriate Student Critique of Training form. These three classroom types are:

• Traditional Classroom is a classroom that does not meet the specifications listed for an Introductory Automated Electronic Classroom (I-AEC) or for an Advanced Automated Electronic Classroom (A-AEC).
  
  
• Introductory Automated Electronic Classroom (I-AEC) is a classroom where the instructor has a computer subsystem, a presentation system, and a local area network (LAN).
  

  Source: Chief of Naval Education and Training Integration Training Working Group; December Conference Materials; NMCI/AEC's PowerPoint slide 13; available online at http://www.cnet.navy.mil/cnet/rtwg/dec.htm
  

• Advanced Automated Electronic Classroom (A-AEC) is a classroom where the instructor uses a variety of technologies, such as an electronic podium, interactive dry write board, digital projection system, document camera, and individual computer work stations.

  Source: Chief of Naval Education and Training Integrated Navy Training Requirements and Planning Databases (INTRPD) Summit; Education and Training Strategies (ETS) Division PowerPoint slide 6; available online at http://cnet.navy.mil/netpdtc/intrpd/summit_briefs.htm
  

  
  

Back to top

When should I use open-ended questions and when should I use Likert scale questions?
There are two main types of critique questions: open-ended and closed-ended. Open-ended questions allow for a respondent to answer the question in his/her own words while closed-ended questions provide the respondent with a list of choices from which he/she can answer. A Likert scale is a type of closed-ended question. The following are examples of open-ended and closed-ended questions.

Examples:

• Open-ended question

              What did you like best in the course?

• Closed-ended question

              The instructor provided you with adequate feedback on your performance in the course.

 a. Strongly Disagree
 b. Disagree
 c. Neither Agree nor Disagree
 d. Agree
 e. Strongly Agree

Critiques can consist of either open-ended or closed-ended questions, or a combination of the two. Each type of critique has advantages and disadvantages. When determining which type of questions to use, consider the following:

• What information do you want to get from the critique?
  
• How many students will be completing the critique?
  
• How much time do you have to analyze the data?
  
• How articulate are the students?

• What information do you want to get from the critique?
  If you want to explore an issue in great detail and know students' unique perspectives on a particular topic then using open-ended questions may be beneficial.

  If you want to compare students' answers, then closed-ended questions (Likert scale) may be beneficial to use.
  

• How many students will be completing the critique?
  If the number of students who will complete the critique is large, closed-ended questions (Likert scale) may be beneficial to use. They are easier to score and provide for statistical analysis of responses. Statistical analysis is not as effective with smaller numbers.

  What is considered a large number of students?

  There is no magical number. The smaller the group of students, the greater one individual's response will have on any summary statistics you choose to calculate. This is particularly true with percentages.

  For example, you have a group of five students, and three of them answer "agree" with a statement on the critique and two answer "disagree." Your percentages in this scenario will be 60% and 40%, respectively. However, if the scenario was only a little different, and two people reported to "agree" and three said they "disagree," the percentages would have been 40% agree and 60% disagree. As you can see, the percentages can fluctuate a great deal by only one response.

  Now consider this example. If you have a class with 40 students, and 24 students "agree" with a critique statement and 16 "disagree," your percentages would be 60% agree and 40% disagree. However, if only one student answered differently, and the results were 23 students "agree" and 17 "disagree," the statistics would be 57.5% agree and 42.5% disagree. Although percentages fluctuate with the larger classes, they do not fluctuate as dramatically.

• How much time do you have to analyze the data?
  Normally, close-ended (Likert scale) questions do not take much time for students to complete or for you (or someone else) to score and analyze. Open-ended question require more time for all three tasks - completing, scoring, analyzing. Therefore, if you have a limited amount of time to administer and analyze the data, Likert scales may be more effective, particularly if you are dealing with large number of respondents..
  

• How articulate are the students?
  Since students must answer open-ended questions in their own words, open-ended questions are most effective when students are able and willing to express themselves in writing. In other words, the more articulate your students, the more effective open-ended questions will be. If the students have trouble expressing themselves, then closed-ended questions may be better to use.

   

Advantages and disadvantages of open-ended and closed-ended questions
The following table displays the advantages and disadvantages of using each kind of critique.

Back to top

What are the steps in analyzing responses to the Student Critique of Training?
There are five steps in the analysis of the responses to the Student Critique of Training. For the Likert and Open-ended Responses Critique, you will complete all five steps. For the Open-ended Responses Only Critiques, skip Steps 1 and 2, and complete the remaining steps. Click on the steps below to guide you in analyzing student responses to the Student Critique of Training:

1. a. Obtain student averages.
   b. If student averages have not been calculated for you, calculate student averages.
   
2. Enter student averages.
   
3. Review, classify, and tally student comments.
   
4. Identity possible reasons for opportunities.
   
5. Develop plans for revisions.

1a. Obtain student averages.
If you have received student average ratings for the items on the Student Critique of Training be sure that the scale corresponds to the following:

For items with a 1-5 scale:
1 = Strongly Disagree
2 = Disagree
3 = Neither Agree nor Disagree
4 = Agree
5 = Strongly Agree

For Yes/No items:
Yes = 1
No = 0

For all items:
N/A = Not Applicable

If your data corresponds to the above scales, you may begin entering the average ratings into "Charting a Course."

1b. Calculate student averages.
If you do not have a spreadsheet program or some other existing system for calculating the average rating for each critique item, instructions for how to calculate student average ratings are provided below. Use the following steps to calculate student average ratings:

• Step 1. Obtain student responses to the critique items.
• Step 2. Determine the scale used (5-point Likert scale vs. Yes/No).
• Step 3. Tally all of the ratings for each item. Use the provided tally sheet (PDF).
• Step 4. Multiply the number of times that a rating appeared by the rating amount to obtain the subtotals of each rating.
• Step 5. Add all of the subtotals of each rating to obtain the grand total.
• Step 6. Determine the number of students who answered the item on the critique. This is the total number of tally marks. Do not include "N/A" responses.
• Step 7. Divide the grand total by number of students who responded to the item.

Example:

Critique Item 1A Trainee Guide was necessary for me to understand the material.

Scale: 1 - Strongly Disagree, 2 - Disagree, 3 - Neither Agree nor Disagree, 4 - Agree,
5 - Strongly Agree

Example:

Lesson topics provided me with the knowledge needed to perform in the labs.

Scale: 1 - Yes; 0 - No

Add up all the "Yes" responses and then divide by the number of total responses (only those who responded "Yes" or "No"). Consider the following situation. Ten students responded to the above item. Of those 10 students, five responded "Yes" and five responded "No." To calculate the average rating for the item, divide the number of students who responded "Yes" by the total number of students who responded. The average rating is .50 (5/10 = .5).

2. Enter student averages.
After the averages of the student ratings for each item has been entered into the module, the item will be designated a "strength" or an "opportunity." A "strength" is an area of the course that should be continued or used as a model. An "opportunity" is an area of the course that may be considered for revisions. A "strength" is an item that is equal to or above the Strength Indicator Score (SIS) previously determined by an administrator. In contrast, an "opportunity" is an item that is below the SIS.

3. Review, classify, and tally student comments.
Students are encouraged to make comments about the course. Complete the following steps to organize the comments:

• Step 1. Read through student comments to get an overview of the types of feedback they provided regarding the course.
• Step 2. Read through individual student comments and classify each as describing a "strength" of the course or an "opportunity." A "strength" is an area of the course that should be continued or used as a model. An "opportunity" is an area of the course that may be considered for revisions.
• Step 3. Next, classify each comment according to the appropriate category on the Student Critique of Training. There are seven categories of questions on the critique. They are described below:

  Course Materials includes information about the materials that facilitate learning in a course, such as the Trainee Guide, technical manuals, training aids, and training equipment.

  Lesson Topics includes information about the organization of lesson topics and whether or not those lesson topics provided students with an adequate understanding of the material.

  Objectives and Assessments includes information about the amount of time that learners had to complete practice skills and tests. It also includes information about the linkage of lesson objectives to test questions and the explanation of grading criteria.

  Safety and Training Facilities includes information about how lessons on safety were conducted. It also includes data about the physical conditions of the learning facility, such as classroom equipment, laboratory equipment, training equipment, and training aids.

  Instructor includes information about the instructor preparation for class, professionalism, and willingness to help students outside of class. It specifically deals with whether the instructor taught at a level that students could understand, encouraged students to ask questions, answered student questions effectively, motivated students to learn the material, and was enthusiastic about the subject.

  Safety includes information about how safety was discussed prior to performance laboratories and if it was made a priority during those activities.

  Student Experience includes information about prior and current student experience with class activities and technology.

Note: For the Open-ended Responses Only Critiques, students are asked to place comments in designated sections titled, "Strengths" or "Areas for Improvement" for each category. Confirm that the student comments were placed in the appropriate category.

• Step 4. Now that you have classified the comments as a "strength" or "opportunity," and have categorized the comments by category, tally the number of comments. The following table identifies the possible category combinations.
  

• Step 5. Record the total comments that were made for each strength/category and opportunity/category combination in the "Total Comments" text box on the "Enter Student Critique Comments" page.
• Step 6. Enter all student comments or illustrative examples in the text boxes for each strength/category and opportunity/category combination. Enter student comments so that similar comments are grouped together. For example, if several students said that a strength, or positive feature, of the Trainee Guide was that it was easy to understand, group those comments together.

4. Identify possible reasons for opportunities.
Some factors in a course may influence the reasons that several of the opportunities were identified. By identifying the reasons for the opportunities, you will be able to analyze how to improve the course. Use the steps below to help you identify possible reasons for opportunities:

• Step 1. Review the scores of various items on the Student Critique of Training to determine some of the possible reasons.
• Step 2. Consider how the strengths may be used to develop plans for revision. Some plans for revision may build on a course's current strengths as part of the process of alleviating opportunities.
• Step 3. Review the list of possible reasons and select the ones that are most appropriate for the course.
• Step 4. Enter any possible reasons not listed into the text box.
  
  

5. Develop plans for revisions.
After you have identified the possible reasons for opportunities in the course, you will be able to analyze how to improve the course by developing a plan for revision for each of the reasons. Use the steps below to analyze the reasons and develop appropriate plans for revision.

• Step 1. Review the previously identified list of reasons.
• Step 2. Select the plans for revision that are most appropriate.
• Step 3. Enter any plans for revision not listed in the text box.
  

Back to top