Beyond the Numbers: Using Qualitative Data to Understand CTE Student and Employer Needs
“There was no system in place for employer needs and wants to shape which learning outcomes should be in our courses and programs. We need a way to link those and to give students good feedback on where their competencies are.” –Dr. Wyl McCully
“We want to institutionalize career planning for our students, to ensure alignment and that our students are prepared.” - Dr. Karen Hicks
When a community-college student has “completed” a program, what does that mean? Most schools look at whether students have satisfied every requirement for a degree or credential and judge success based on that standard.
But the real-world meaning of “completion” for career and technical programs, which stand at the intersection of academic requirements, employer needs, and individual student goals, is not so clear.
“We all have a different definition of what completion means,” said Dr. Wyl McCully, market research analyst at Lansing Community College and an SDP-CTE ECMCF fellow. “Is it the definition from the student? The faculty? The employer?”
At Lansing, data-collection efforts show that many career and technical education (CTE) students don’t finish the programs they start. But the data doesn’t explain why students leave before earning a certificate or credential, including whether such credentials actually confer the knowledge, skills, and abilities that students and employers need.
To bridge that information gap, two SDP-CTE Fellows decided to go right to the source: students and employers themselves. McCully and Dr. Karen Hicks, director of assessment at Lansing, created new surveys and detailed interview protocols to surface and explore the greater context surrounding completion data. That includes students’ individual career goals, employer perceptions of in-demand qualifications and program alignment, and faculty visions for career preparation and progression over the longer term.
“The more we dug into quantitative data, the blurrier and hazier the picture got,” said McCully. “The only way we could answer these questions was qualitatively.”
“The data were not telling the full story,” said Hicks. “We knew we needed to go under the hood and tell a richer story.”
New Methods to Define and Assess Alignment
In theory, career and technical programs should prepare students for employment after completion. But the impact of these programs can be hampered by misalignment with student and employer needs.
First, curriculum and assessments instructors use to measure student success may not line up with the knowledge, skills, and abilities that employers are looking for in the workplace. In addition, programs may include academic requirements that can set students up for career success over the longer term—but the value of that coursework may not be clear to busy working students, who may choose to stop their training once their immediate needs are met. As a result, students may leave programs without a buildable credential, shortchanging their longer-term growth beyond. Or they may successfully complete CTE programs without mastering target competencies, because training is not fully aligned with ever-changing workplace needs.
The SDP-CTE ECMCF fellows set out to create a structured approach to ensure program design and market needs are aligned. First, they developed a detailed interview protocol for faculty to define program outcomes and expectations, or the knowledge and skills that students are expected to master through coursework and assessments. Those questions included prompts to define and describe the program and reflect on completers’ possible career progression, including entry-level and more senior jobs and industries where students could transfer their expertise.
The fellows developed a similarly detailed interview protocol for employers. Questions explored opportunities and requirements of particular jobs as well as which roles Lansing students could potentially fill. They also asked what type of training is needed for jobs at various levels of seniority and whether graduates will likely need different or additional training in the short- and longer-term.
In addition, the fellows created surveys to capture students’ voices and perspectives at various points during their time at Lansing, using multiple-choice and Likert scale designs. In enrollment applications, three survey questions will collect baseline data about students’ reasons for enrolling, whether they initially plan to earn a certificate or degree, and how long they expect to attend. As students move through orientation and prerequisite coursework, brief surveys ask whether training to date is adequate and if they have achieved their goals. Upon completing core courses that typically indicate whether students will go on to complete the credential, a survey asks whether they plan to continue their studies. Finally, for students who complete capstone work at a program’s end, all earlier survey questions will reappear, along with prompts asking students to reflect on their perceived career readiness.
These questions can help determine whether students are “completers” on their own terms, even if Lansing’s current data collection would not count them as such. They can point to explanations for why CTE students continue or cut short their studies. And they can inform more in-depth advising, so faculty can map longer-term career goals onto coursework that may not immediately seem necessary to fast-moving, busy working students.
“I want a student to accomplish the goals they have when they come to us, but I also want the student to be placed in a position that enables them to improve their life,” said McCully. So far, programs in Unmanned Aerial Systems, Cloud Computing, and Electrical Technology have participated in some of these processes. Pilot efforts in five more programs are slated for the coming year.
Implications for CTE Institutions
The SDP-CTE fellows’ project highlights the critical importance of both qualitative and quantitative data to address this common challenge. This type of work runs at a different pace than typical data analysis: quantitative work includes in-depth design thinking at the outset, but then analyses run quickly once data collection is complete. Qualitative work may move somewhat more quickly at the outset, but performing interviews and reviewing the insights they uncover is a slower process.
They point to three major lessons learned from their work:
First, to build or bolster connections between institutional curriculum mapping and market needs, CTE programs should set out slowly and intentionally to engage stakeholders, develop relationships, and build new processes to better align training and the workplace. The fellows prepared detailed interview protocols to guide their work, but caution that institutions may need to prepare unique processes that address their particular needs.
Second, student voice should be at the center of these efforts. Ask students how they define success. What are their goals, triumphs, and challenges? This context can help programs and institutions understand why students persist or stop out and provide advisors with context that can help whether their training adequately prepares them for success in the short and longer term, and how program design can be better aligned with what students need to compete in their target industries.
Finally, focus on narrative over numbers. Quantitative analysis can show how many CTE students are or are not earning credentials, but it can’t explain why. Conversations with faculty can identify what value they intend students to gain from a CTE program, and conversations with employers can provide important anecdotal evidence showing whether the intended value is realized in the labor market. Together with student accounts of their experiences, these stories can identify misalignment and help CTE institutions ensure their programs more clearly meet both student and employer needs.
***
SDP Impact in Focus:
Who: SDP-CTE ECMCF Fellows Dr. Wyl McCully, Market Research Analyst, and Dr. Karen Hicks, Director of Assessment
Where: Lansing Community College in Lansing, Michigan
What: Created and implemented new processes to assess alignment between CTE programs and local employers, including by gathering survey data from students to identify their goals and perceptions of program success.
Why: Like many institutions offering career and technical programs, Lansing Community College did not have a clear method to assess whether current curricula were meeting the needs of the local labor market. Quantitative data showed CTE students left the school without completing a target program more often than non-CTE students, but numbers alone could not explain why that was happening. A new process to assess the alignment and effectiveness of CTE programs introduced detailed interview protocol for faculty and employers and surveyed students on their goals and perceived value of CTE programs at various touchpoints, to help faculty ensure that curricula prepare student for professional success.
“The metrics we use are high level, pretty stringent, and not really suited well to a CTE program. We recognized that and knew we needed to tease these metrics out in a different way.” – Dr. Karen Hicks
SDP thanks the ECMC Foundation for its support launching SDP’s first postsecondary CTE fellowship program.