In the School District of Palm Beach County, ROI helped SDP Fellow Abigail Todhunter-Reid better serve students in need.
Like many districts across the country, the School District of Palm Beach County faced a puzzle. Their strategic plan focused on increasing high school graduation rates and college readiness while also seeking to correct racial disparities in these areas. Like many school districts, they adopted a supplemental program, in this case the Advancement Via Individual Determination (AVID) program, to help solve this puzzle. In 2010, the district introduced AVID, a year-long elective course to engage low-income minority students in advanced coursework and college preparation, in five Title 1 high schools then expanded the program to thirteen schools over the next several years. And results were promising.
By 2018, graduation rates were up (87% of students were graduating as opposed to 2014’s 78%) and gaps were closing—kind of. That year, only 83% of black students and 85% of Latinx students graduated from high school compared to 93% of white students, and college enrollment saw similar gaps with 61% of black students, 58% of Latinx students, and 72% of white students enrolling in college the fall after high school graduation. While support for the AVID program was robust throughout the district, the gap persisted, and a question lingered for district administrators.
Could they be getting more from AVID?
The Many Facets of Program Efficacy
SDP Fellow Abigail Todhunter-Reid knew she had to approach the question from a number of angles. Brought on to conduct a rigorous evaluation of the AVID program, Todhunter-Reid would incorporate information about many aspects of the program to help know how effective the program truly is.
“Leadership was really excited about this program,” said Todhunter-Reid, “and some descriptive data had been showing positive upticks in graduation and college enrollment rates. Yet they wanted a more detailed look into the program, particularly the cost component, to see what kind of return-on-investment (ROI) the program was providing.”
Through an advanced matching approach called Coarsened Exact Matching, Todhunter-Reid began contrasting students enrolled in AVID to similar unenrolled students. “We didn’t compare AVID students to all other students,” explained Todhunter-Reid. “Instead we matched them with students similar in characteristics like gender, race, baseline test scores, attendance, and discipline.” Using a sample of 29,485 students total (1,329 AVID students), students were tracked over a six-year period from 8th-grade to one year post high-school graduation.
Through this work, Todhunter-Reid was able to support some of the positive assumptions stakeholders held of the program. AVID was indeed producing positive results and increasing graduation and college enrollment rates in the schools where it was offered. But in an interesting twist of fate, the cost component revealed something unexpected.
ROI’s Surprising Reveal
To determine ROI, Todhunter-Reid developed a cost-effectiveness ratio that determined the cost per additional positive outcome of the program—essentially the cost for each student that graduated or enrolled in college who was not likely to do so without the program’s intervention. And of the 13 schools utilizing the program in the district, the program was “producing” only about 40 additional graduates. With the cost of implementation at $1200 per student, the cost per additional graduate tallied in at a steep $17,400. And since the program was so widely embraced throughout the district, Todhunter-Reid put this information to interesting use.
“The program was never really at risk of getting cut because people really loved it,” she reflected. “But adding the cost component really started to shift our thinking, and we started looking more into implementation. How could we improve the way we implement the program to ensure we were getting a greater return?”
To answer this new question, Todhunter-Reid started building predictive models to look at the characteristics of the program’s students in 8th grade, before they had entered AVID. When looking at the students’ 8th-grade characteristics, she found that many of the students who were selected into the program already had high probabilities of graduating and enrolling in college. “That indicated that we were selecting a lot of students who maybe didn’t need the full offerings of the program and maybe would have benefitted just as well from a different, less costly program.”
Thus, incorporating ROI information ultimately led district leaders to revise the selection process for AVID.
“Originally, leaders wanted me to envision and identify the students most likely to succeed in this program. Yet the results of this project indicated that we need to focus more on the students most in need of the program. By focusing on the students likely to succeed, we were artificially inflating the effects yet ultimately getting a lower return on investment. To get the greatest ROI, you want to really select the students who are going to benefit from the program.”
Lessons Learned
With manifold choices and limited budgets, school districts face a daunting task when selecting programs and vendors. Oftentimes, decision makers must base their decisions on the word of vendors who have no obligation to be honest—they’re salespeople. Todhunter-Reid’s project reveals the importance of including ROI as a focus for program evaluation, yet this work isn’t without hurdles. Luckily, she shared some advice for how to incorporate this kind of work.
1. Don’t be intimidated: This work is complex, and many often feel intimidated approaching the ROI metric. But Todhunter-Reid assures that once you find the information you need and include the cost, your task will be tedious, but not necessarily difficult.
2. Find opportunity costs: Go beyond the use of expenditure data found in budget documents to include at least some opportunity costs. The most important opportunity costs to include are human capital costs like teacher and administrator time. For example, with AVID, teachers are needed to teach the elective course, and district administrators are employed to oversee program implementation and provide support to schools. Teacher and administrator salaries are often the most expensive components of programs and, if omitted, cost estimates can be wildly inaccurate. The good news is that these costs are often simple to estimate.
3. Standardize the approach: Once a framework for ROI is in place, standardize the approach and evaluate as many programs as possible. The next step in Todhunter-Reid’s project, for example, is to develop a bank of cost effectiveness ratios to use in comparing programs with similar outcomes.
Abigail Todhunter-Reid is an American Educational Research Association Fellow through the Study of Deeper Learning.