In common circumstances, success is a moving target. For more unique groups of students, even more so.
The call for data-driven problem solving has increased in the wake of the pandemic as schools now wrestle with unprecedented conditions. COVID illuminated and exacerbated pre-existing inequities and shifted how (and if) students are engaging and performing in the classroom. And while legislation like the Every Student Succeeds Act (ESSA) in 2015 brought with it a call for more evidence of school accountability, a fact remains as constant now as it was pre-pandemic: Accountability systems are not one-size-fits-all.
Traditional measurement methods only capture a portion of the data needed to understand success for schools that serve special student populations. Schools with large numbers of students returning to school after dropping out or those with disabilities, illnesses, and behavioral challenges often receive additional resources and oversight from regulators to ensure growth and improvement. Yet their special circumstances come with untold stories of growth hidden by traditional accountability measures.
This posed a unique challenge for SDP Fellows Jessica Shopoff, Shibu Joseph, and Tobie Irvine, all charged with compiling data to tell a different story about the progress and status of schools operating off the beaten path.
Creating a student-centered data system for opportunity youth
The term “opportunity youth” identifies a group of young people ages 16-24 who are neither enrolled in school nor participating in the labor market. This is a critical time for young people and can set the occupational trajectory for young people’s lives, yet this group has only grown in the face of the pandemic—around one in three young adults were estimated to fall into this group in June 2020.
Learn4Life; a network of 20 independently-governed, alternative model, public charter schools in California, Michigan, and Ohio that serves opportunity youth; has had much success with students who are unsuccessful in other educational models. Providing the evidence for this success, however, has been tricky, as traditional measures like state tests and graduation rates are unhelpful for students who have already dropped out of high school before reaching their schools.
“We knew we would have to get more nuanced when developing our success metrics,” said Jessica Shopoff.
Yet in order to get more nuanced, Shopoff first had to make the metrics visible. While statistics like graduation rates for their students were important, Learn4Life needed to capture their students’ experiences more granularly. Like many educational organizations, Learn4Life first needed advanced data culture and infrastructure to answer this more complicated question of success.
Thus, Shopoff drew on four distinct principles to set the stage for data: capturing the voices of their students, compiling academic and compliance-related data as well as metrics the students deem important, automating the data, and facilitating the sharing of data across the organization. By incorporating the student voice into their approach, the organization was able to surface data points that may have been hidden by a top-down approach.
“This work really helped us focus on the point of using data for accountability and improvement,” Shopoff reflected. “We want to know what’s working and what’s not, with the ultimate goal being to better serve opportunity youth. And our job is to build both the systems and the culture to help make that happen.”
A different take on school accountability in New York
Meanwhile, in the state of New York, fellows Shibu Joseph and Tobie Irvine sought to surface an alternative way for a coalition of Special Act Public School Districts—districts that serve a higher proportion of court-placed students and students with disabilities—to provide ESSA-compliant accountability metrics to the state.
Under ESSA, these schools are often flagged as underperforming schools by a Comprehensive Support and Improvement (CSI) or Targeted Support and Improvement (TSI) designation. Yet the assessments used to make these designations often don’t capture the whole picture for these students and thus don’t actually answer the underlying question of whether these students are performing and improving at school.
“The identifications made of these schools were not truly reflective of the actual work the school was doing to serve their students,” Irvine noted. “This is a very unique population that requires alternative methods of assessment, which forces Special Act districts to appeal a CSI or TSI designation over and over again.”
To solve this issue, Joseph and Irvine developed an alternative accountability system by reworking the data used to determine performance and facilitate accountability.
For example, traditional measures of chronic absenteeism include excused absences, yet many of these students had higher rates of excused absences for things like doctor visits and court appointments. They also included metrics like course passing and computer-adaptive assessments vs paper-and-pencil assessments—interestingly, this student population tended to score higher on computer adaptive tests.
“By including a number of more custom metrics in our system, we were surprised to see that actually, students in these schools were generally showing good growth,” added Joseph. “In many cases we saw growth rates in the 70th and 80th percentile under our new system.”
The importance of truly understanding what’s really going on with a student’s education will only continue to increase as the U.S. education system recovers from and incorporates the lessons of the pandemic. The ultimate goal is to understand how a student is doing in school and what those schools are doing to facilitate performance and growth. With the amount of data now available, schools serving special populations are poised to invite nuance in their assessments through custom metrics that truly reflect the student experience.
Lessons learned
For those also working on projects of accountability, consider the following:
- How does the one-size-fits-all accountability systems play out for unique populations that learn differently?
- When identifying metrics for success, gather voices from the special education community. What outcomes matter most? What else should you be considering?
- Traditional accountability systems may not always highlight the achievement and growth made by special education students. How will you balance rigorous academic standards with appropriate and fair metrics?"