University Governance
Faculty Governance Task Force on Evaluation of Teaching
Recommendations for gathering information from students
The Task Force reaffirmed the importance of including student surveys in the process of evaluating teaching. Five sources guided the development of the new Student Survey of Teaching form. First, the Kansas Board of Regents specifies a range of topics to be included in any survey of students. Second, KU student government has created its own online survey for providing students with peer feedback on their experiences in courses. Third, a faculty committee had proposed a set of items that it felt captured the features of teaching that students are ideally in place to describe. Fourth, the Task Force consulted the extensive published work on student ratings of teaching, looking for guidelines and suggestions for appropriate items or topics. Fifth, a faculty research group made up of experts in testing and statistics conducted a trial run of different formats created by the Task Force. The Task Force combined the elements it gathered from these five sources.
Recommendations
The Task Force recommends the following questions for the KU Student Survey of Teaching:
The first eight would be answered with a five point scale labeled strongly disagree, disagree, neither agree nor disagree, agree, or strongly agree. The ninth question asks for a comparison of amount learned with other similar classes, and the five responses would be labeled much less, less, the same, more, much more.
Questions:
1. This instructor’s teaching was clear, understandable and engaging.
2. This instructor was encouraging, supportive and involved in my learning
the course material.
3. This instructor was available, responsive and helpful.
4. This instructor provided content and materials that were clear and organized.
5. This instructor set and met clear goals and objectives for the course.
6. What this instructor expected of me was clear and fair.
7. What this instructor expected of me was appropriately challenging.
8. The instructor demonstrated respect for students and their points of view.
9. Compared with courses at a similar level, I would rate how much I learned
as:
[much less, less, the same, more, much more]
The results from these nine questions will be reported as the percent of responses in each category of the five-point scale, along with the percent of responses to that question for other courses at approximately the same level of the curriculum from the same department. The two distributions will be shown together in one graph, with indications below the graph showing an average score with graphic indicators of the spread of the scores. All nine questions will be reported on a single side of paper, allowing for a concise view of the student survey results and normative comparison.
There will also be a set of questions asking for background information on the students’ reasons for enrollment, year in school, level of participation, and expectations.
Questions:
How important were the following reasons for taking this course?
(not a reason, somewhat important, important, very important)
1. Course fulfills a major or minor requirement
2. Course fulfills an elective requirement
3. Course fulfills a school requirement
4. Course was not full (open)
5. Course was at a convenient time
6. Course topic interests me
7. Course instructor has a good reputation
My student status is:
Undergraduate, Graduate, Other (non-degree, faculty or staff)
What year
of study are you in?
1st,
2nd, 3rd, 4th, 5th, 6th, 7th or more
Did you
complete readings/coursework?
Never,
rarely, sometimes, most of the time, always
How many
times per week did this class meet?
One,
two, three, four, five
Over the
course of the semester, how many class meetings did you miss?
(enter
number)
What grade
to you expect in the class?
A,
A-, B+, B, B-, C+, C, C-, D+, D, D-, F
The Task Force recommends that faculty also ask (on a separate sheet of paper) the following three generic open-ended questions:
- What did you like about this course?
- What changes do you suggest?
- If you’d like to make comments about any of the items on the numerical survey, please write them below.
Faculty members are encouraged to ask students for other information they would find useful in development of their courses. The following are some examples of the kind of OPTIONAL course specific questions that would be customized to the particular features of an individual course:
- Please comment on the topics in the course; which did you feel were most important to your learning and which least?
- Please comment on the readings for the course; which did you find particularly helpful to your learning or not helpful to your learning?
- Please comment on how class [lab, studio, practicum, discussion section, field work, community engagement] time was spent. In what ways did it help you learn? How could this time be changed to help you learn more?
- Please comment on how useful the course assignments were to your learning. Which assignments helped you learn the most? Which ones helped learning the least? Are there any forms of assignment you would recommend adding?
- Please comment on how well major course projects (exams, term papers, collaborative reports, presentations, other products) gave you an opportunity to show what you have learned. Were there additional ways you could have been prepared to enhance your performance on these projects?
Rationale
The Task Force generated several foundational assumptions based on extended discussion and sampling of faculty and student opinion. The overall assumption was that teaching should be evaluated primarily by peer review of a broad range of teaching materials and practices, focused ultimately on how much and how well students learn in their courses. Students play an important role as sensitive field observers of key features of teaching, and student surveys indicate when a class is noteworthy or distinctive in some way, either desirable or undesirable. Those surveys should not, however, be taken as a substitute for professional judgment by faculty peers. The survey results, the peer reviews of course effectiveness and practice, and the faculty member’s own narrative of teaching development should all be considered by any committee making an evaluation of teaching.
On a practical level the Task Force followed several key guidelines:
- The student survey form should be as short as possible, while still providing useful information covering an optimal range of characteristics. Student members noted that it is difficult to sustain careful judgment about teaching when four, five, or even six lengthy forms are filled out within a week or so.
- The form should not include any overall or summary item; research in the field does not support the use of those items and they are more likely to be used in place of more considered and nuanced judgment by evaluation committees.
- Faculty are urged to collect and read focused, open ended comments from students, but those comments are to be collected separately from the surveys and used in evaluation only if initiated by the faculty member. Faculty members are encouraged to describe in their teaching narratives how they have developed their teaching by learning from the student surveys and comments, and those who provide comments need to follow a set of guidelines in their analysis and presentation.
The Task Force worked with two small teams of faculty members who study ratings and do formal analysis of open-ended comments. The experts on measurement and statistics did a pilot test of different forms with more than 1200 students this past spring and summer. The groups assessed a range of teaching characteristics with a longer and shorter form of question, and they used a five-point and a seven-point response scale. The short form with five response points is a demonstrably reliable indicator of student perception about teaching. The performance of the longer instrument with a seven-point scale was marginally superior to the shorter, five-point version, but the gain in psychometric performance was not large enough to overcome the advantages of the shorter version. The experts on analyzing comments outlined the basic properties of a reporting system for commentary, including identifying questions to be answered, specifying categories of responses, and reporting the frequency of those categories, including those who did not respond.
The reporting of the results is under development and will be ready for this semester’s data. The Regents’ requirement for comparisons (normative data) is met in a nuanced way, as the comparison distribution includes only courses of a comparable number level. The current version looks at courses numbered below 300, 300 through 699, and 700 and above. The potential exists for faculty members to obtain individualized comparisons and to use the background data to divide up the survey results. Those advantages await the resolution of issues around access to the database.
A great deal can be learned from the observations offered by students in response to questions about features of a course, and faculty members are encouraged to seek open-ended commentary. It is especially valuable for faculty members to respond to the comments with adjustments in the course, and faculty members can also include those changes as part of their teaching narratives. KU has a policy jointly articulated by the Provost and by Faculty Governance that makes the use of such comments in personnel decisions optional, at the discretion of the faculty member or unit. Accordingly, any open-ended questions used by a faculty member should be on a separate sheet from the numerical rating items, so that they may be collected and distributed independently of each other.