Dr. Adriana Villavicencio discusses lessons learned from the impact (and lack thereof) of a NYC program serving Black and Latino boys.
New policies in education often evoke excitement and buzz upon implementation—but lose attention if they later fall short of expectations. However, evaluations of older policies can contain valuable takeaways even in the midst of underwhelming results.
This is precisely why Dr. Adriana Villavicencio, Assistant Professor at the University of California, Irvine, and former Deputy Director of the Research Alliance for New York City Schools, took time this fall to reflect on her massive longitudinal evaluation effort of the now-inactive Expanded Success Initiative (ESI) with the SDP network (and in her book Am I My Brother’s Keeper?).
In 2011 New York City (NYC), graduation rates for all student demographics were rising, but college readiness rates stagnated for young men of color compared to their white and female counterparts. This was true even for students in this group who you might think had more of an advantage—those scoring at the highest academic proficiency levels as incoming 9th graders, for example, still saw a 20 percentage point gap in terms of college readiness.
“Though ESI ultimately fell short of some of its original aspirations, we should still reflect on the program’s lessons learned,” noted Villavicencio. “Especially now, in the wake of the long-term losses of the pandemic that have disproportionately impacted communities of color as well as indigenous and immigrant communities.”
ESI takes on college readiness in Black and Latino boys
The original motivation for ESI was to address the college readiness gap for Black and Latino boys. Formed under the leadership of Mayor Michael Bloomberg in 2012, ESI was the educational branch of the four-part Young Men’s Initiative that sought to redress the inequities between young men of color and other young men in NYC. This was the largest investment in young men of color undertaken before President Obama’s My Brother’s Keeper.
“While it was ultimately designed to improve college- and career-readiness, ESI was really an initiative with multiple goals,” Villavicencio explained. “Being led by Black and Latino males at the time, ESI also sought to change the conversation around boys of color by shifting away from deficit framing and focusing on their assets and how the system might be failing them.”
Other goals of the initiative included culturally-relevant classrooms, changing teacher beliefs about the populations served, and improving relationships between the boys and adults in their schools.
ESI extended $250k over two-and-a-half years to 40 public high schools in NYC. To qualify for these funds, schools had to create or expand programming for young men of color within three domains: academics, youth development, and school culture. “Culturally-relevant education” (CRE) undergirding these domains helped create guidance for how the funds would be used over the life of the program.
Moreover, a team of program leaders provided schools and teachers with various sources of support and professional development over the lifetime of the initiative. “They weren’t just given the funds and told to do whatever they wanted,” said Villavicencio. “The program had a robust infrastructure.”
Did it work?
Villavicencio’s study sought to understand how ESI was implemented in NYC schools and whether it ultimately impacted students’ academic (GPA, credit accumulation, graduation and college readiness rates) and non-academic (sense of belonging and fair treatment in school, college knowledge) outcomes. This four-year study followed the first cohort of incoming 9th graders through graduation and measured the program’s impact by matching these students to students in a set of comparator schools.
Using data from over 500 semi-structured interviews with over 800 school leaders, teachers, and students in the 40 ESI schools; additional interviews, observations, and document reviews from a sub-sample of seven exemplary ESI implementers; and survey data and administrative data from both ESI and comparator schools, Villavicencio dug for the answers to her questions.
“It was a massive undertaking with large amounts of data, and I definitely didn’t do this alone,” Villavicencio reflected. “A team of 20-30 research analysts, including the research alliance’s executive director, and countless students and teachers also offered a lot of their time to help make this evaluation possible.”
What did she find? While implementation was moderately strong for most ESI schools, ESI students did look different than comparator schools in terms of early college preparation. Yet there was no statistically-significant impacts on academic outcomes.
But that isn’t the whole story.
“While the impact was nil in terms of academic outcomes, we did see the impact that ESI had on school culture and student experience.”
These impacts included the adoption of culturally-responsive education, restorative approaches to discipline, and strong student relationships and teacher-student relationships through mentorship. Additionally, Villavicencio documented a positive impact on self-reported measures of students’ sense of fair treatment and sense of belonging in school, as well as a positive impact on students’ interaction with adults to discuss their plans for the future.
“These are notable findings, given what we know about what Black and Latino boys often experience in their classrooms from a very young age.”
Some of what ESI did was change the conversation to think about celebrating the successes of these boys and their ancestors while providing students the space to process traumatic events together, like the killings of Tamir Rice, Michael Brown, and Eric Garner. As one 12th-grade ESI student offered:
“I think this program creates something you can’t get anywhere else...to have these guys every day that I came to school just made me want to come to school that much more because I got to be with my brothers. I got to be with my family."
Lessons learned
After reflecting on what worked and what didn’t in ESI, Villavicencio offered these lessons learned for those adopting similar programs in their schools and districts.
Aspects of ESI worth replicating:
- Focusing explicitly on racism while challenging deficit narratives
- Forming a coalition of stakeholders
- Building capacity in schools with ongoing support
- Shifting teachers’ mindsets, beliefs, and practices
- Improving school culture and relationships
- Investing in rigorous evaluation of programs
Aspects of ESI that weakened the potential for more robust results:
- A diffuse intervention
- Less attention to the academic domain
- Widespread participation over deep engagement
- Limited timeframe
For the full video of Dr. Villavicencio’s presentation, click here.
Dr. Villavicencio is an Assistant Professor at the University of California, Irvine and wrote the book Am I My Brother’s Keeper?.