SDP Fellow Pierre Lucien provides an encouraging report of success in fall of 2020.
In August 2020, while the national mood was leaden with pandemic fatigue, some good news emerged regarding a college enrollment program in Massachusetts. The state’s Early College Initiative, which helped high-school students complete college-level coursework for credit, was proving effective in increasing the number of students completing financial aid applications and enrolling for college—even amidst the upheaval of COVID-19.
Unearthing this good news was SDP Fellow Pierre Lucien.
Since the inception of the program in 2018, leaders had yet to evaluate the efficacy of the Early College Initiative. And unlike academic policy research, which takes a long time to conduct, the evaluation had to happen quickly to inform decisions about the future of the program. As states go, Massachusetts is advanced in its infrastructure and appetite for education data. And as a former CEPR research analyst and newly-appointed SDP Fellow at the Massachusetts Department of Elementary and Secondary Education, Lucien had what he needed to hit the ground running with the state’s robust data system.
“Early college is highly visible and a topic of conversation at the highest level of government in the state,” said Lucien. “Upon arrival at this project, it was clear that this work was going to have a massive impact on policy decisions, and thus on outcomes for students.”
A familiar framework
Using propensity score matching, Lucien compared the outcomes of participants and non-participants and determined that participants were more likely to enroll in a postsecondary institution within six months of graduating from high school. To make this determination, participants were paired with a non-participating student who resembled them in a number of ways—race, gender, socioeconomic status, English language status, and others—with the only difference being their participation in the program.
This work differed from similar efforts like that of Julie Edmunds in North Carolina in that it wasn’t designed as a randomized controlled trial (RCT). “RCTs are considered the gold standard, yet this program wasn’t set up for this kind of evaluation,” Lucien reflected. “Since we were charged with quickly pulling data for decision makers, the quasi-experimental method of propensity score matching offered a nice stop-gap measure to generate meaningful preliminary results.”
The results Lucien excavated with this method were impressive. As stated in this Inside Higher Ed article, “About 85 percent of early-college students earned three college credits with grades of C-plus or above, and nearly half of them earned 12 college credits. Collectively, students in the program earned 5,088 credits in 2019, giving them a collective potential savings of $1 million.”
From the outset, Lucien knew there was something different about this project. The Early College Initiative was a program well known to many leaders across the state, from the governor and education secretary to various representatives, commissioners, and board chairs. Upon arrival to the project and the team at the Massachusetts Department of Elementary and Secondary Education (DESE), Lucien became clear that this was a highly-visible project.
“Since the completion of this work, I’ve basically gone on tour, ” Lucien continued. “I’ve presented to audiences such as district and business leaders, advocacy groups, and board members. When they listed this fellowship, they didn’t advertise just how high-profile this work would be. And it’s really assuring and affirming to see the appetite decision makers have for this kind of data.”
Since this work was completed and distributed, the Early College Initiative program has received more funding and expanded into more schools. The data work will continue as well as the sample size for program participants increases and additional analytical models are introduced. Lucien himself will conduct much of this analysis with the support of DESE and SDP.
“SDP was critical to the realization of this work,” reflected Lucien. “The term ‘support’ can mean many things, but for SDP it runs the gamut from providing resources and feedback to autonomy and advocacy. They’re really putting the right things in place to realize their vision of increasing the analytic capacity of education agencies to affect real change.” Like the work of all SDP Fellows, Lucien’s work shows that good intentions paired with data can drive real change that makes a real difference for states and students.
As this work continues, Lucien offered a couple of takeaways that may be helpful for those undertaking similar projects:
Anticipate all possible interpretations and uses of the data and notate your presentation accordingly. Data used in a public presentation will be repurposed by attendees and stakeholders in ways you don’t intend them to.
Prepare for a varied reception and understanding of preliminary results. Proponents of a policy sometimes won’t distinguish between preliminary results and final results when the numbers show what they want, but they are acutely aware of the distinction when they don’t like the numbers.
Additional Coverage of Lucien’s work:
Clearing the Path to Higher Education | Harvard GSE
Promising Results for an Early College Program | Inside Higher Ed
Early college catching on, but not immune from pandemic impacts | 22 News WWLP
Massachusetts’ Early College Initiative | Education Data Chat Podcast