 |
 |
How do I conduct a longitudinal
analysis?
What
is a longitudinal analysis?
How
do I select classes to include in my analysis?
How
do I analyze longitudinal information?
How
do I conduct reviews within each information source (student
enrollment, student achievement, student critique, and instructor
feedback)?
Where
can I go for more information?
What
is a longitudinal analysis?
Longitudinal analysis involves
examining various aspects of the course over time. It allows
course developers to identify trends that may be addressed
when the course is revised. The revision process is, "a
modification to the course mission statement, an increase
or decrease in course length, or training material modification
that requires additional resources."
Though the revision process is on a large-scale, a course
developer may also use the longitudinal analysis for interim
changes.
Source: NAVEDTRA 135B,
4-1-9
Back
to top
How
do I select classes to include in my analysis?
"Charting a Course"
allows you to select the classes that you want to include
in your analysis. The classes are organized according to the
following categories: Class number, Class Convene Date, and
Classroom Type. Use this information to select the classes
you want to compare.
For example, if you are interested
in comparing how students performed in two types of classrooms
(such as traditional versus Introductory Automated Electronic
Classroom or I-AEC), you will want to determine a time period
in which courses of both classroom types were offered. Then,
select all of the courses that are listed for both types of
classrooms within that timeframe.
If you are interested in how students
performed in one particular classroom type (such as Advanced
Automated Electronic Classrooms or A-AEC) over time, use the
information in the Classroom Type column to make your selections.
For the course revision process, you
may be most interested in all of the courses that have been
offered since the last course revision.
Back
to top
How
do I analyze longitudinal information?
There are five steps in conducting a longitudinal analysis.
They are:
- Review student
enrollment information.
- Review student
achievement information.
- Review Student
Critique of Training.
- Review instructor
feedback.
- Develop a revision
strategy.
Because the purpose of a course is
to improve student learning upon course completion, only end
of course information is retrieved from the Student Critique,
Instructor Feedback, and Student Achievement tabs. For the
student achievement information, the number of students passing
each objective will appear for the last test for which the
objective was assessed.
Note: While completing Steps 1 - 4
above, refer to the "How do I conduct reviews within
each information source (student enrollment, student achievement,
student critique, and instructor feedback)?" link at
the beginning of this How To. This link will provide you with
the procedures and information you need to analyze the data
you are reviewing in Steps 1 - 4. Step
5 occurs after you have completed the first four steps. Developing
a revision strategy will involve using all of the information
you collect in the previous four steps.
- Review student enrollment information.
Student enrollment provides some insight into who is or
is not passing a course and why. This information is helpful
in determining if there is a problem area in a course, but
it does not indicate specifically what the problem is. For
instance, a high number of students not passing a course
may indicate an opportunity for improvement. Also, a high
number of Academic Review Boards may indicate that students
are having difficulties with the course.
There are seven classifications
of student enrollment:
- Starting enrollment is the
number of students enrolled in a course on the first day
of class.
- Number of Non-grads, Academic
is the number of students who fail to complete a course
due to their inability to meet the curriculum's educational
requirements.
Source: CeTARS Help Homepage, available
at http://wwwnt.cnet.navy.mil/cetars/help_home.htm
- Number of Non-grads, Non-academic
is the number of students who fail to complete a course
due to factors not related to their academic performance.
Source: CeTARS Help Homepage, available
at http://wwwnt.cnet.navy.mil/cetars/help_home.htm
- Number of Non-grads, Disenrolled
is the number of students administratively removed from
training for a variety of reasons, such as cancellation
of a class or course, a student's inability to meet prerequisites,
or withdrawal by parent command.
Source: CeTARS Help Homepage,
available at http://wwwnt.cnet.navy.mil/cetars/help_home.htm.
- Graduates are the number
of students who have successfully completed a course of
instruction.
- Review student achievement information.
Review the percentage of students passing each terminal
and enabling objective. For each objective, the identification
number of the last test in which it was assessed and percentage
of passing students are listed. Reviewing this data can
help you pinpoint particular parts of the course that were
strong and parts of the course that may need to be revised.
If a low percentage of students are passing a terminal objective,
then each of the corresponding enabling objectives should
be considered for revision, even if they have a high percentage
of passing students.
Also, review reasons for students
passing the objectives. Reflect on instructional strategies,
class activities, and events that helped students pass
these objectives.
Finally, review reasons for students
not passing objectives. Reasons for students not passing
objectives may be found in the Student Critique of Training
and the instructor feedback.
Click on the following link
for information on how to conduct this review:
How
do I conduct reviews within each information source (student
enrollment, student achievement,
student critique, and instructor feedback)?
- Review Student Critique of Training.
The Student Critique of Training displays the average student
rating for each item on the critique form and designates
each item as a strength or an opportunity. Review the possible
reasons for the opportunities and any plans for revision
that were identified.
Click on the following link
for information on how to conduct this review:
How
do I conduct reviews within each information source (student
enrollment, student achievement,
student critique, and instructor feedback)?
- Review Instructor Feedback.
Instructor feedback includes responses to items on the instructor feedback form as well as any problem areas and plans for
revision that instructors entered into the Instructor Feedback
Tab. Use the following information to help you review items
that instructors answered:
Some items are rated on a 1-5 scale:
1 = Strongly Disagree
2 = Disagree
3 = Neutral
4 = Agree
5 = Strongly Agree
Other items are rated with a Yes,
designated as a "1", or a No, designated as
a "0".
Items rated N/A mean that that
item did not apply to the course.
Click on the following link
for information on how to conduct this review:
How
do I conduct reviews within each information source (student
enrollment, student achievement,
student critique, and instructor feedback)?
- Develop a revision strategy.
After examining all the
available information (student enrollment, student achievement,
student critique, and instructor feedback) you should be
able to pinpoint areas that are strengths or opportunities
for improvement.
Previously entered suggestions
for revision are a good starting point in making your
plans for revision. Determine the frequency and emphasis
placed on the suggested plans. More frequent and greater
emphasized suggestions should take priority over other
comments.
You may want to compare information
within each information source by looking for frequent
positive comments, high ratings, or high student achievement
as areas of the course that worked well and should be
repeated. Specify plans for revision in areas of the course
with frequent negative comments, low ratings, or low student
achievement.
Click on the following link
for information on how to conduct this review:
How
do I conduct reviews within each information source (student
enrollment, student achievement,
student critique, and instructor feedback)?
Back
to top
How
do I conduct reviews within each information source?
To conduct reviews for each type of information, follow
the five steps below:
- Identify trends.
- Determine how trends
are related to other information sources.
- Determine why these
trends occurred.
- Determine what
to do as a result of the analysis.
- Identify tends.
A trend chart is a graph that illustrates patterns of occurrences
over time. For this module, those occurrences displayed
are the ratings of questions on the Student Critique of
Training and instructor feedback form, and the number of
students who graduate from a course or pass course learning
objectives.
In this module, bar graphs are
used to illustrate trends of end of course information
for various class offerings. Bar graphs are available
to display end of course enrollment information, the percentage
of students who passed individual course learning objectives,
and average ratings of items on the Student Critique of
Training. For bar graphs related to the Student Critique
of Training, the line that runs across the bars represents
the Strength Indicator Score (SIS). By comparing the bars
for each class against the SIS, you will be able to prioritize
which classes to examine in more depth.
Remember that one of the main purposes
of a trend chart is to determine patterns that have occurred
over time. Patterns to look for are increases, decreases,
or the maintenance of scores (that is, enrollment numbers,
percentages of students passing, and average ratings).
You will also want to make note of spikes and cycles in
the data.
Here are some general rules to
follow as you look for patterns:
- A pattern of increase or decrease
exists when six or more consecutive points are moving in
the same general direction. This direction is either upward
(an increasing pattern) or downward (a decreasing pattern).
- A pattern of maintenance exists
when nine or more points remain generally steady or close
to a similar point.
The following is an example of an increasing
pattern in a trend chart:
|
Type of Pattern: Increase
Cause of Pattern:
After reviewing the training aids the
instructor used during class presentations, it was found that
the instructor added and modified them between April (20020104)
and September (20020109). The instructor added some updated
videos and modified some of the diagrams to show details more
clearly when being projected for the entire class.
|
The following is an example of a maintenance
pattern in a trend chart:
|
Type of Pattern: Maintenance
Cause of Pattern:
The Trainee Guide was used in a similar
manner in all of the classes.
|
Sometimes, trends do not follow a consistent
general direction. Another arrangement that may occur is called
a spike. A spike is a single score that is located far above
or below the general direction of the other scores.
The following is an example of a spike
in a trend chart:
|
Type of Pattern: Spike
Cause of Pattern:
During the month of August (20020108),
students attended a lecture given by an admiral who visited
the schoolhouse. During his lecture, he emphasized the importance
of safety in the job setting. For the remainder of the course,
the instructor drew upon the personal experiences and those
described by the admiral to relate safety lessons to the job
setting.
|
Another type of pattern to look for
is called a cyclical tendency. A cyclical tendency is a pattern
where the trend repeats itself.
The following is an example of a cyclical
tendency in a trend chart:
|
Type of Pattern: Cyclical
Cause of Pattern:
This chart illustrates
that every three months a pattern developed where the
students rated this item high and then lower. Many students
in the classes that met during the second and third
months of each cycle mentioned that there were a lot
of distractions, particularly during tests, in the halls
from students attending lectures. When examining the
schedules of classes that met in close proximity to
this one, it was found that a 3-month course held large
group lectures in the auditorium during the second half
of their course. Since these lectures were held during
the last half of the second month and all of the third
month, the students who were in the course being reviewed
during the first month did not mention the distraction.
However, the students who were in the course during
the second month had to deal with the distractions for
part of their time, while the students in the third
class were greatly impacted by the movement of the other
to and from the auditorium.
|
A key part of examining these trends
is to determine the causes for the patterns.
For trends that follow a general direction
or that are cyclical, it is important to identify the causes
for the pattern. Since these patterns occur repeatedly, they
may be considered "common causes".
For trends with spikes, the causes
may be considered, "special causes". These causes
should be described in the plans for revision so they may
be replicated, maintained, or decreased depending on whether
or not the pattern illustrated positive results.
- Determine how trends are related
to other information sources.
One way to understand a particular trend is to compare trend
charts and comments for similar items or objectives. Similar
trends in similar items and objectives may indicate a common
cause. For student achievement information, you will want
to look at objectives within the same unit. For the Student
Critique of Training or the instructor feedback form, you
may want to compare items that are within the same category,
such as "Course Materials" or "Lesson Topics".
Sometimes, it is helpful to compare
items from other information sources to confirm conclusions
developed from one information source. For example, you
may
want to compare an item on the Student Critique of Training
with similar items on the instructor feedback form. To
do this, go to the Single Offering Tab, select the end
of course information for the appropriate class offering(s),
and go to the "Compare Student and Instructor"
screen.
- Determine why these trends occurred.
To determine the causes of the trends, it is helpful to
examine comments made during previous reviews of the Student
Critique, Instructor Feedback, and Student Achievement Tabs.
You will want to look for comments that were made frequently
or those comments that made reference to a special cause.
Also,
be sure to pay special attention to those comments that
the previous reviewer(s) emphasized as being key reasons
for opportunities and plans for revision.
- Determine what to do as a result
of the analysis.
After you determine the causes of the trends, you will want
to enter your observations and conclusions about the strengths,
opportunities, and plans for revision. These observations
should be made from your analysis of the trend charts and
the comments made by previous reviewer(s). They should summarize
and reiterate the emphasis placed on key factors and events
described by previous reviewers. Now
that you have determined why these trends occurred, you
should be able to address how you will maintain favorable
results and make improvements in the areas that were not
as favorable.
Back
to top
Where
can I go for more information?
Clemson University. (1995). Run charts/Time plot/Trend
chart. Retrieved August 15, 2002, from http://deming.eng.clemson.edu/pub/tutorials/qctools/runm.htm
SkyMark Corporation. Control charts.
Retrieved August 15, 2002, from
http://www.isixsigma.com/offsite.asp?A=Fr&Url=http://www.skymark.com/resources/tools/control_charts.htm
Ishikawa, K. (1985). Guide to quality
control (2nd Ed.). New York: Unipub.
Kume, H. (1985). Statistical methods
for quality improvement. Tokyo, Japan: The Association
for Overseas Technical Scholarship.
Back
to top
|
