Tuesday, May 10, 2016

AV#147 - A plug for student surveys in teacher evaluations – even when they hurt

                            May 10, 2016


 “Mr. Huidekoper, I think you should relax. Try Quaaludes before class.” (From a student survey)

It is great to hear of more efforts to welcome the students’ voice in evaluating a teacher’s effectiveness.  I celebrate such efforts here (after I first reveal what students said of me: oh how those words can sting!). Maybe we will soon find ways to incorporate feedback from student surveys into the School Performance Framework.  Yes—a portrait of a school based in part on how students view the quality of their classes and teachers.  What a concept!
You know the refrain: we should not judge students, or schools, on one test. We must have multiple measures.  Of course.  Equally true in assessing a teacher’s performance.  In AV#145 I celebrated the evaluation and support that comes from colleagues, as seems possible in the Teacher Leadership Collaborative Model in Denver Public Schools.  If peer review is the best way to improve classroom instruction, student surveys are often the second best tool—well ahead of the principal’s observation.
Most of what follows comes from national studies (including references to work in Colorado, especially in Denver Public Schools and the Thompson School District)—key findings on, and examples of, student surveys playing a greater role in accountability and growth.  I hope it can be useful resource for such efforts in our state.  I make no proposals on how a school, district, or the Colorado Department of Education might incorporate surveys into “educator effectiveness”; in my view, the less prescriptive the better.  Let the school own it.  I simply argue: handled well, such surveys can make us better teachers. 
But first I begin with a confession: what my students told me.
In 18 years of teaching – asking students from grades 6 – 12 to let you know….[1]
The year comes to a close, and the teacher passes out a survey to his students—with some trepidation. By early May you have spent over 160 days together this year, you have a sense of what has and has not gone well, you know you never made a great connection with __, __, and __ —and you recall past surveys where the students’ comments hurt, so you brace for a few stingers.
Still, you know you must get their feedback.  Hey, you’ve been grading them all year—isn’t it their turn to give you that C-?  More importantly, they know how well you teach better than the principal does. Yes, I mean that—at least for grades 7-12.  While you take some comments with a grain of salt—perhaps that student is ticked off by a recent low score on a paper, or feels you snapped too harshly at his or her off-task behavior yesterday—your class knows if your instruction has been clear, energetic, challenging….   Students know your character pretty well too: every day they assess if you are kind, patient, fair—and if—no, when—you fall short. They know—as you do, as should your principal—that you can perform well enough for the administrator’s “walkthrough,” but what has taken place day in day out, beginning back in mid-August … that was no act.  What “the kids” will tell you is likely to be a good reality check.
What do I mean by hurt? No one will ever again hire this 66-year-old, so I might as well confess:
Response to: “When speaking about my work, the teacher did not embarrass me.”
One year the response from my two 8th classes was positive, but almost half of the 7th graders scored me a 1 (strongly disagree) or 2 (disagree). In sharing my results with the school administration, I wrote: “I believe strongly in using student work for comment – for pointing our problems, and of course for giving praise, which I try to do as much as criticize – but these comments from 7th graders are troubling. I obviously need to address this issue when I meet next year’s 8th graders in August.”
Response to: “The teacher did not lecture, or give uninterrupted talks, too often.”
One year, 20 out of 38 7th graders disagreed, as did 16 out of 35 8th graders.  Extremely worrisome, especially as I say I emphasize class discussion and participation, and like to think I invite student voice—and listen well.  Tough to swallow, but I needed to hear it.
Response to: “There was sufficient variety in the kind of classroom activities that took place.”
In one 7th grade class, 8 of 43 scored me a 2; with 8th graders, one scored me a 1 and 12 a 2.  Another area that needed work.
After spending a year together, I would love to get all 3’s (agree) and 4’s (strongly agree) in response to: “The teacher made an effort to know me personally and to teach me as an individual.”
But the response one year was discouraging. Most scored me a 3 or 4, but in 7th grade 9.5 disagreed, and in 8th grade, 14 disagreed.  Not good. In a small K-8 school, where classes were never greater than 23, and where I never had more than 90 students in my 4 classes, I could only blame myself for those low marks from so many.
Then, of course, when I invited other comments, some were heartfelt: “Don’t call on us if we don’t want to talk right then,” or, “Don’t favor students (no offence) but you kind of favored __ and __  and you always talked about what they did great …. Do that with all the kids. I had some hurt feelings.” And some (I think!) just humorous, such as admonishing this Language Arts teacher: “NEVER start the morning (in homeroom) off with Language Arts. It put students in a foul mood and then they don’t listen as much.” (FYI, that was merely #1 on this student’s -thanks Brittney!- top 10 list of suggestions for me!)  
Even more dangerous: inviting comments from high school students.  I always promised not to read their suggestions until the school year was over.  Thirty years later I am still miffed by the comment from a senior in a Shakespeare class who suggested I take drugs to make me less uptight. I was glad to see and talk with her (back in the 1980’s surveys were done by hand, and by year’s end you knew everyone’s handwriting)  at her 10th high school reunion, but even then it was clear—I had failed in some way with several students…. Her words echoed what I often heard: Lighten up. A sense of humor never hurts.
**
I.                    From “Lessons from the Field: THE ROLE OF STUDENT SURVEYS IN TEACHER EVALUATION AND DEVELOPMENT” -http://bellwethereducation.org/sites/default/files/Bellwether_StudentSurvey.pdf, May 2014.
States and districts are taking various approaches to measuring teacher effectiveness…. While student surveys are not new, they typically have been used to take the temperature of the whole school (for example, school climate surveys) rather than focus on specific teachers….
But, because students are the direct consumers of instruction, with a unique perspective on teacher impact, there is a growing belief that students can provide valuable feedback on a teacher’s performance in the classroom. Student perception surveys are increasingly seen as a low-cost and reliable tool for gathering data and feedback on the quality of teaching in individual classrooms. Well-designed student surveys ask students about instructional practices that correlate with improved student learning, such as student-teacher relationships, teacher management of the classroom, rigor of lessons and student engagement, and teacher responsiveness to student struggle. The feedback provides actionable information about students’ classroom experiences….
… it was the Measures of Effective Teaching (MET) study, supported by the Bill & Melinda Gates Foundation, which had the most direct impact on the recent upswing in the adoption of student surveys. The three-year MET study evaluated the use of student surveys in seven different school districts [including Denver Public Schools] … and involved 3,000 teachers. The study concluded that …  “students know an effective classroom when they experience one.” In fact, surveys were found to be broadly predictive of a teacher’s ability to increase student achievement and were a more reliable measure than classroom observations. Importantly, student surveys were also more stable year-to-year than student test scores, meaning student survey results showed promise as an additional measure in teacher evaluations to increase reliability and validity…. (Bold mine)
NCTQ’s report included 15 “Lessons in Teacher Evaluation Policy.” #7 stated: “Surveys have emerged as an important source of data and feedback on teacher performance. It is important for states and districts not to underestimate what it takes to design a high-quality instrument, and adopt validated instruments or get expert help writing, testing and implementing surveys. … “
A recent report from the National Council on Teacher Quality (NCTQ), a research and policy group, notes that many states are incorporating surveys as an important source of data and feedback on teacher performance. Specifically, 17 states now require or allow for use of student, parent, or peer surveys in teacher evaluations. Twelve states require or allow for student surveys in particular. (http://www.nctq.org/dmsView/State_of_the_States_2013_Using_Teacher_Evaluations_NCTQ_Report, p. 31.  See box.  Required in 8 states: Alaska, Connecticut, Georgia, Hawaii, Iowa, Massachusetts, New York, and Utah.) 
 
[Four] other states [Colorado, Missouri, Mississippi, and New Mexico] have stopped short of mandating student surveys but allow them to be used as one measure of teacher effectiveness.  For example, in Colorado, the use of student perception data is strongly encouraged in the state rules as a means to provide teachers with feedback on their performance. [See box—happily, not required, but we can….]
COLORADO: “student survey results can be used….”
Britt Wilkenfeld, Assistant Director of Research for Educator Effectiveness at the Colorado Department of Education, wrote me: "Under current Colorado statute and State Board of Education rules, student survey results can be used for an educator’s evaluation as long as student academic growth still accounts for at least 50% of the evaluation and all of the Teacher Quality Standards are assessed."
Several large school districts have also either begun using—or plan to use—student surveys about their experiences in individual classrooms as a means to inform teacher evaluations and professional development.…  Since the original study results were published, four of the MET districts—Pittsburgh Public Schools, Denver Public Schools, Shelby County Schools, and New York City Schools—have begun administering surveys district-wide.  (Bold mine) MORE ON DPS – Pages 6-7.
From the report’s Conclusion – the “third common measure”
The jury is still out, however, on whether student surveys will join classroom observations and student achievement data as a third common measure in newly redesigned teacher evaluation systems, or if adoption will remain limited to a small number of progressive districts and CMOs.  Based on our research and interviews, we believe that student surveys can be that third measure, because they add value that classroom observations and student learning data cannot. … student survey data provide actionable feedback that teachers and their supervisors can use to target specific areas for improvement.
[This 2014 report pointed to the Colorado Legacy Foundation’s piloting of a student perception survey “to determine its validity and reliability.”[2] The Foundation, now named the Colorado Education Initiative, has produced a survey kit. Below is CEI’s material for grades 6-12. The version for use with grades 3 – 5, can be found at http://www.coloradoedinitiative.org/toolkit/research/.]
**
II.               From “Colorado’s Student Perception Survey - Grades 6-12,” Colorado Education Initiative  http://www.coloradoedinitiative.org/wp-content/uploads/2014/09/SPS_Administration_survey-instrument-6-12-CEI.pdf, 2014.
Colorado’s Student Perception Survey[3]  is a 34 question instrument that measures elements of student experience that have been demonstrated to correlate most closely to a teacher’s ability to positively impact student growth.[4]  Students are asked to indicate how frequently they experience each item with a response scale of always, most of the time, some of the time, and never. … The survey items for grades 6 - 12 are listed below. (CEI’s website also includes “What the Research says about Student Feedback for Teachers” - http://www.coloradoedinitiative.org/wp-content/uploads/2014/09/Planning_comms_Research-Overview-CEI.pdf.)
Thompson Valley is one of the Colorado school districts using CEI’s student perceptions surveys.  (See district web site- http://thompson.k12.co.us/Page/5721)
There we also see two examples of what teachers have said in comparing fall and then spring results.  Cheerful notes on the good progress I made this year, on my success .… OK, but to make this work truly reflective—as in self-critical—let’s be sure we don’t simply pat ourselves on the back.  If the surveys are going to help us improve, we should trust the process, and our students, in a way that allows us to see our shortcomings. Nobody’s perfect!  Reflecting on the not-so-good news – that’s where can benefit so much.
Student Learning: How teachers use content and pedagogical knowledge to help students learn, understand, and improve.                                                          
My teacher makes learning enjoyable.
What I learn in this class is useful to me in my real life.
My teacher teaches things that are important to me.
My teacher knows the things that make me excited about learning.
In this class, we learn a lot every day.
In this class, it is more important to understand the lesson than to memorize the answers.
When the work is too hard, my teacher helps me keep trying.
My teacher accepts nothing less than my best effort.
My teacher knows when we understand the lesson and when we do not.
If I don't understand something, my teacher explains it a different way.
My teacher explains difficult things clearly.
In this class, we have a say in what we learn and do.
My teacher talks to me about my work to help me understand my mistakes.
My teacher writes notes on my work that help me improve.
When we study a topic, my teacher makes connections to other subjects or classes.

Student-Centered Environment: How teachers create an environment that responds to individual
students’ backgrounds, strengths, and interests.

My classroom is organized and I know where to find what I need.
Students feel comfortable sharing their ideas in this class.
My teacher respects my opinions and suggestions.
My teacher cares about me.                                         
My teacher pays attention to what all students are thinking and feeling.
My teacher respects my cultural background.
My teacher respects me as an individual.

Classroom Community: How teachers cultivate a classroom learning community where student
differences are valued.

From a principal
“As a Head of School (I retired last year), I introduced a formative teacher assessment process which included … administering and reporting on a student questionnaire. Each question had a Likert scale plus room for comment. The teacher was asked to analyse and report on the data. While numbers of teachers expressed concern beforehand, believing the responses would be immature, retributive or lacking understanding, the reality was otherwise and teachers acknowledged that their students filled out the surveys seriously and that they found the feedback valuable. However, the process was formative.” - From a comment on the blog, “Next Up in Teacher Evaluations: Student Surveys,” Education Week, 7/11/12.
My teacher would notice if something was bothering me.   
Our classroom materials (books, articles, videos, art,
       music, posters, etc.) reflect my cultural background.
In this class, I feel like I fit in.
I feel like an important part of this classroom community.
My teacher knows what my life is like outside of school.
My teacher knows what is important to me.
I ask for help when I need it.
I feel like I do a good job in this class.

Classroom Management: How teachers foster a respectful and predictable learning environment.
Our class stays busy and does not waste time.
Students in this class treat the teacher with respect.
The students behave the way my teacher wants them to.
Students in this class respect each other’s differences.

III.                From “Student Perception Surveys and Teacher Assessments,” Hanover Research.  https://dese.mo.gov/sites/default/files/Hanover-Research-Student-Surveys.pdf, Prepared for XYZ, February 2013.
MY STUDENT SURVEY
My Student Survey was developed by Vanderbilt University researcher Ryan Balch, and is based on research-based teaching practices. The survey was validated in a large-scale pilot as part of Georgia’s Race to the Top initiative in spring 2011. The pilot study administered the survey to over 12,000 students in grades 6-12 at seven Georgia school districts.  The results of the pilot study suggest that My Student Survey is a reliable evaluation of teacher effectiveness. The research revealed a relationship between teacher ratings and academic student engagement and self-efficacy.  (Bold mine).

The survey questions are organized into categories loosely based on two classroom observation frameworks….  In order to provide the most meaningful feedback to teachers, Balch then grouped questions into six constructs that represent the role of a teacher:
Presenter          Manager        Counselor        Coach          Motivator        Content Expert
[Here are sample questions for each of the constructs.]  The survey includes 55 questions that students rate using a five point frequency-based Likert scale, ranging from “Never” to “Every time.”
My Student Survey Sample Questions
CONSTRUCT
SAMPLE QUESTION

PRESENTER
When explaining new skills or ideas in class, my teacher tells us about common mistakes that students might make
At the end of each lesson, the teacher reviews what we have just learned
MANAGER
My teacher corrects students when they do not follow the rules of the class
We are learning or working during the entire class period
COUNSELOR
My teacher shows respect for all students
My teacher notices when I am not participating in class

COACH
My teacher gives us guidelines for assignments so we know how we will be graded (grading rules, charts, rubrics, etc.)
I have to work hard to do well in this class

MOTIVATOR
My teacher has us apply what we are learning to real-life situations
My teacher encourages me to share my ideas or opinions about what we are learning in class

CONTENT EXPERT
My teacher is able to answer students’ questions about the subject
After asking us questions, my teacher lets us think for a few seconds before we have to answer

TEACHER EVALUATION
… Researchers agree that student surveys should not be the only factor in teachers’ evaluations; however, their reliability suggests their use is appropriate when combined with other measures such as classroom observations and value-added student achievement gains. (Bold mine)
Although the use of surveys is limited to a small number of districts throughout the country, the trend is moving in favor of more comprehensive evaluations and an increasing number of states and districts are using student surveys. (p. 10)
DENVER PUBLIC SCHOOLS
Denver Public Schools took part in the Bill & Melinda Gates Foundation’s MET Project pilot study beginning in 2009. Although Denver administers the survey district-wide to over 60,000 students annually, the surveys are still in pilot form and are used for professional development only, not summative teacher evaluations. The pilot was implemented within Denver’s Framework for Effective Teaching….   A major component of this initiative is the Leading Effective Academic Practice (LEAP) pilot,[5] which incorporates five elements into its performance assessment system: student outcomes, principal observation, peer observation, professionalism, and student perception surveys. Student outcomes remain the primary source of evaluation, but student perception surveys appear to count for 5 percent of the overall performance assessment….

… The pilot program website explains that “LEAP provides teachers with additional feedback and support so they can continue to learn and grow professionally.” Its system of multiple measures is
designed to provide teachers with a “comprehensive body of evidence” so they can identify
strengths, weaknesses, and progress. The student perception surveys play a key role in this system, and the district claims that its teachers have found great value in the survey data and modified their practice as a result.  (Pages 19-20.  Bold mine)


IV.                From “Gathering Feedback from Students” at Vanderbilt University’s Center for Teaching, - https://cft.vanderbilt.edu/guides-sub-pages/student-feedback/, downloaded May 2016.

This web site includes many examples of both In-class Feedback Forms and Online Surveys. Here’s one:

Course: ____________________________  Instructor Name: ____________________________
1 = Never; 7 = Frequently
1
Indicates where the class is going
1
2
3
4
5
6
7
2
Explains material clearly
1
2
3
4
5
6
7
3
Indicates important points to remember
1
2
3
4
5
6
7
4
Shows genuine interest in students
1
2
3
4
5
6
7
5
Effectively directs and stimulates discussion
1
2
3
4
5
6
7
6
Provides helpful comments on papers and exams
1
2
3
4
5
6
7
7
Is tolerant of different opinions expressed in class
1
2
3
4
5
6
7
8
Is available outside of class
1
2
3
4
5
6
7
9
Explains thinking behind statements
1
2
3
4
5
6
7
10
Effectively encourages students to ask questions and give answers
1
2
3
4
5
6
7
11
Adjusts pace of class to the students' level of understanding
1
2
3
4
5
6
7
12
Seems well-prepared
1
2
3
4
5
6
7
13
Stimulates interest in material
1
2
3
4
5
6
7
14
Treats students with respect
1
2
3
4
5
6
7
15
Is effective, overall, in helping me learn
1
2
3
4
5
6
7

What do you think is this instructor's greatest strength?


What suggestions would you give to improve this instructor's teaching?


V.       From “Companies Honing Tools to Survey Students About Teachers.” Education Week,
[The article provides a helpful overview of what is taking place on student surveys. It lists five different survey providers. Two quotes from the article:]

Steve Cantrell, a lead senior program officer for the Gates foundation, said he recognizes teachers' wariness of using surveys for evaluations. But he noted that student surveys were the "most reliable measure" of teachers' effectiveness included in the Gates study. Those results were more consistent, for instance, than classroom observations of teachers or their performance as judged by student test scores, he said. "We're not asking students to judge the quality of the teaching.  We're asking them to judge the instructional environment. ...This is information that teachers can't get from anyone else but the students."

Nonprofit organizations, like Battelle for Kids, have also become interested in the information that can be gleaned from student surveys. In partnership with the Gallup polling organization, the Columbus, Ohio-based Battelle offers an online survey, which can be taken by students on computers or mobile devices, designed to measure students' sense of hope, engagement, and belonging—as well as teachers' classroom management. Battelle and Gallup are convinced that students' belief in their learning environment is crucial to their academic and long-term success. (See box for Battelle’s web site; also see http://www.thestudentexperience.org/)
Battelle for Kids
We developed The Student Experience® Survey in collaboration with Gallup® to capture classroom-level feedback from students to help teachers create a learning experience that supports student success now and in the future. The survey captures feedback around four key themes: 
Designed for students in grades 4–12, the research-based 24-item survey can be completed online in less than 15 minutes, protecting instructional time while providing timely feedback to support improvement. Educators can customize the survey to get a snapshot of the student learning experience by classroom, course, class period, or grade. 

"Students won't learn as much if they don't feel they belong in the classroom," said Tim Hodges, the director of research for Gallup Education. "It's a false choice [when] people say it's [only] the soft stuff, or the academic measures, that matter" to students' success.














Addendum A

Teacher evaluation (a form I created and used – borrowing from many others I had seen)

Please circle one number that represents your view.

           1 - strongly disagree            2 – disagree             3 – agree                  4 – strongly agree

1.    Overall, the teacher presented material in a clear manner.    1              2           3              4

2.   The teacher presented and discussed the literature in a clear manner.     1          2         3             4

3.   The teacher presented and discussed writing skills in a clear manner.       1          2         3             4

4.   The teacher was supportive of my efforts to improve as a reader and writer.   1            2              3              4

5.   The teacher helped motivate me to do better.     1         2           3          4

6.   The classroom environment was friendly and positive.   1          2          3            4

7.   There was sufficient variety in the kind of classroom activities that took place.    1            2         3           4

8.   The teacher made an effort to know me personally and to teach me as an individual.  1        2           3           4

9.   The teacher did not lecture, or give uninterrupted talks, too often.   1         2          3          4

10.   The teacher demonstrated effective classroom management skills to help keep us on task.
1              2              3              4

11.   The teacher’s approach to discipline was fair and effective.   1           2          3           4

12.   The teacher listened to my questions and those of my classmates with respect, and the teacher did a good job of making us feel we could participate.   1                2         3              4

13.   Overall, I felt that I was treated with respect by the teacher.   1            2         3          4

14.   In discussions, the teacher treated my views and the views of my classmates with respect, and we felt we could speak honestly.     1         2                 3           4

15.   I have felt comfortable contributing to discussion in this class.     1          2           3         4

16.   When the teacher spoke about my work in class, the teacher did not embarrass me.   1       2       3         4

17.  The teacher gave out grades that seemed fair.    1          2         3          4

18.  The teacher’s grading system was consistent with most other teachers in the middle school.  1         2       3         4

If you indicate 1 or 2, circle one of the two below, indicating that this teacher’s grading system was                                                harder    or                                 easier.






[1] See Addendum A for the two-page survey I used 10 years ago, for middle school students.
[2]  See CEI’s 141-page STUDENT PERCEPTION SURVEY TECHNICAL REPORT, http://www.coloradoedinitiative.org/wp-content/uploads/2014/09/technical-report-CEI.pdfColorado Education Initiative. (2013). Denver, CO: Author.
[3] Copyright 2014 by The Colorado Education Initiative. All rights reserved.
[4]   A number of items on the Colorado SPS were adapted from items made available for non-commercial use through the Measures of Effective Teaching (MET) Project, funded by the Bill & Melinda Gates Foundation.
[5] This was written in 2013.  Today, LEAP is well past the “pilot” stage.