SDP Fellow Cara Jackson and Urban Teachers set their sights on observations as an evaluation tactic.
It’s scarcely news that brand new teachers aren’t as effective as experienced teachers, leaving many to grapple with how to best prepare new teachers before they step into the classroom. So what tactics are most effective in evaluating whether a teacher is ready to hit the ground running and using that evidence to provide them with targeted, meaningful feedback to improve rapidly?
The lack of practical guidance on how to effectively assess the strengths, limitations, and accuracy of new teacher evaluation led SDP Fellow Cara Jackson and Urban Teachers to determine how to gain a reliable judgement on teacher quality as they entered the workforce. Urban Teachers, a residency-based teacher preparation program leading to dual general and special education licensure for all participants who pass through their performance standards, features continual observation of performance as part of their teacher training approach. This regular evaluation set the foundation for a meaningful and critical look into which evaluation practices would be most effective for teacher credentialing.
Following the research emerging at the time, Jackson decided to focus on observations, a tactic employed by nearly every school district in the country as part of their teacher evaluation systems.
“One of the things we found in the research was that averaging scores over more observations is more reliable than using a single observation,” reflected Jackson. “But because observations are both time- and resource-intensive, we wanted to get a sense of how many we would need to gain a reliable judgement on teacher quality. That was really the basis for the project.”
To answer this research question, Jackson examined data from 40 teachers observed by 9 coaches throughout the 2013-14 school year. Secondary research had suggested that more observations were likely better in determining teacher quality, and this suggestion was corroborated by this work with Urban Teachers. This study revealed a relatively large increase in reliability when moving from one to two observations, which was the most marked jump in reliability. While reliability improved from .70 to .81 when increasing from 1 to 2 observations, adding a 3rd observation increased reliability to .85, and a 4th resulted in .87.
“We had enough evidence to discourage making any impactful decisions about teachers based only data from a single observation,” reflects Jackson, “and ideally each teacher candidate would be rated by more than one observer over the course of the semester or year.”
While these impactful recommendations for observation emerged from this project, the long-term impact of this work also included a general shift in how the agency used data to inform their decision making and practices. “When working in a space like this where we’re trying to bridge the gap between research and practice, it’s important to use data as a basis of a conversation,” Jackson offered. “We put into place some quarterly data checks to look at average scores by observer, for example, then we would have a conversation about what it could mean. If a particular observer was consistently rating teachers higher or lower, for example, we would talk through the anecdotal and contextual information that could inform a finding like that.”
In a reality where teacher effectiveness dramatically increases in the first five years of teaching, effective teacher preparation becomes of paramount importance. By looking into the effectiveness of a simple and widely-used evaluation practice, observations, Jackson and Urban Teachers helped fine tune the measures used to determine when a teacher is ready to enter the classroom. And as teachers enter their careers more prepared to succeed, students will benefit most.
Cara Jackson is an Associate Partner in Policy and Evaluation at Bellwether Education Partners.