When institutions look for ways to boost persistence, they often default to adding new programs or resources. These initiatives can continue for years without scrutiny, or they may be evaluated during leadership changes or budget reviews to assess their effectiveness in helping students succeed.
However, this evaluation shouldn’t be an occasional practice. Regularly measuring the efficacy of student success initiatives is crucial for strategic decision-making and planning. This ongoing analysis reveals hidden opportunities to surface the most impactful supports. It allows for precise adjustment, enhances policies and initiatives, and empowers institutions to invest in the right mix of resources.
Consistent initiative analysis doesn’t require reinventing the wheel each time. Institutions can make targeted adjustments to better serve those who benefit most. Partners across the Civitas Learning community who regularly engage in efficacy analysis are seeing significant success. In addition to positively impacting student persistence rates, they are experiencing higher usage of resources, greater ROI on the programs, and better alignment across campus.
Creating a Routine Initiative Assessment Strategy
Many institutions overlook a powerful strategy for understanding student success: consistently evaluating student success initiatives. Regular and repeated analysis across various initiatives and policies helps pinpoint opportunities for improvement and investment. These adjustments improve student persistence, establish better alignment across departments, elevate student engagement levels, and enhance ROI. Understanding what exactly improves outcomes and which students benefit most can be transformative.
Here’s how successful institutions incorporate initiative analysis into their strategic decision-making & planning:
1 | Review
What initiatives, policies, & services are already in place?
Administrators and department leads should conduct a thorough review across all departments to understand what’s available to support students so they can decide what initiatives and programs need a closer look.
Administrators and departmental leads should start by identifying initiatives, events, services, and programs already available across the institution. Creating a comprehensive inventory goes beyond traditional academic support like tutoring and advising and requires engaging with all campus departments to uncover less obvious resources and events that also have a strong influence on student persistence. These include guest speaker sessions, faculty development programs, and student job fairs that are often overlooked in traditional initiative audits.
For each initiative, gather the following details:
- Purpose: What is the initiative designed to achieve?
- Target Audience: Who is the initiative aimed at?
- Oversight: Which department, person, or team is responsible for it?
- Eligibility: Who is eligible to participate?
- Overlap: Does the initiative overlap with any other programs?
Every initiative is created with the intent of helping students succeed. To understand their effectiveness, we need to consider them in the context of the challenges students face today — not when they were designed.
2 | Define
Leaders should determine clear objectives & pose a specific research question to guide the analysis.
They should also define the activity, participants, and time frame for measurement. For example, they might ask, “Does being taught by a faculty member who completed 1+ training series in the prior term impact persistence?” Conducting both long-term and short-term evaluations is crucial for shaping strategies and allocating resources effectively.
Once leaders identify which initiatives they’d like to assess, the next step is to define a research question. This question should consider clear factors such as participants and timing. For example, does attending an additional math tutoring session in the term impact persistence?
Participants may range from narrowly targeted groups like returning students to broad cohorts such as all incoming first-year students. Conducting both long-term and short-term evaluations is crucial for shaping student success strategies, allocating resources effectively, and supporting the institution’s financial stability.
Many institutions recognize the importance of creating inclusive student experiences and understand the impact of initiatives on underserved populations. Initiative analysis has proven to be instrumental in helping to better address inequality on campus. For example, Tacoma Community College (TCC) uses initiative assessment to identify policies that may disadvantage specific student groups. This analysis has helped leaders understand how registration timeframes affect working students, the impact of sending past-due accounts to collections, and the effect of drop policies on students by age or ethnic group.
3 | Analyze
Establish a system for regularly assessing initiatives, allowing for timely adjustments to impact student outcomes.
Administrators and leaders should collaborate with Institutional Research to schedule regular reviews. By integrating initiative efficacy assessment into strategic planning and decision-making, leaders can uncover hidden opportunities to target specific groups and allocate resources to the most impactful interventions, helping to achieve institutional goals.
With objectives clearly defined, it’s important to ensure that data is collected and updated in real-time to effectively analyze results. The data should include student IDs, term information, and other relevant participant details. Real-time data on student activity allows Institutional Research teams (or the individual conducting the analysis) to gather the necessary information and perform the analysis as outlined in the research question. This proactive approach enables institutions to address potential issues promptly, using insights as soon as they are available instead of waiting for scheduled reports.
Establishing a system for regular initiative assessments is crucial for achieving successful student outcomes. The University of Texas at San Antonio experienced this firsthand when they analyzed the effectiveness of their Student Success Coaching program. By using the Civitas Learning Student Impact Platform, they quickly analyzed key programs and discovered that coaching had a significant impact on the persistence of at-risk students who might not typically engage with the program. This insight, in combination with other strategies and efforts, helped UTSA achieve a 16% increase in retention and a 14% increase in completion from 2012 to 2022.
4 | Socialize
Administrators & departmental heads should create a culture of transparency by sharing initiative efficacy with stake-holders across the institution.
This approach encourages dialogue on how to best apply findings so there is alignment from the start. With a shared understanding of each initiative’s impact on student persistence, leaders can make more informed and strategic investment decisions.
With efficacy results in hand, Institutional Research teams should determine the best way to socialize this information so departments can take appropriate action. This should occur on both a macro level, by sharing all results institution-wide, and on a micro level, by providing subgroup results for data-activated decision making. This approach fosters an environment open to feedback and questions. As a result, this fosters an environment open to feedback and questions. Effective communication and tailored messaging are crucial for recommendations to add resources or adjust an initiative. Reports that are too technical or unclear to those outside the department can lead to misinformation, rumors, and resistance among staff and administrators.
In the fall of 2023, Northwest Missouri State University (Northwest) led an institution-wide effort to get everyone on board with a data-driven mindset to achieve student success outcomes. They regularly share findings from the Civitas Learning Student Impact Platform with departmental leaders, allowing them to validate results, interpret their significance, and determine how best to enhance effective elements and rethink those that are less successful.
Work like this has led to noteworthy outcomes for Northwest, such as an 8% improvement in student retention between 2016 and 2021 and an 80% overall retention rate from fall 2021 to fall 2022
5 | Act
Analysis that sits on a shelf might be interesting, but it doesn’t drive change.
Leaders need to take action to see the desired outcomes. Whether changing a policy or connecting students with specific resources, focusing efforts equips institutions to iterate and improve student success outcomes over time.
The final step in integrating regular initiative analysis into decision-making is acting on the results. Administrators and department leads can use these insights to take strategic action, such as implementing policy changes, securing funding, or launching targeted campaigns for specific student groups. These findings guide schools in planning future student success initiatives and refining existing programs to improve their impact.
The Civitas Learning partner community has seen measurable outcomes from these actions. For example, when Snow College assessed its advising program to determine if all students benefit equally from a one-on-one advising experience, they found that some benefited more than others. Specifically, lower-performing students saw a 20% boost in persistence from an individual advising meeting compared to a 3% lift among high-performing students.
Snow College optimized resource allocation by shifting high-performing students to group advising, so they could expand 1:1 holistic advising for students who would benefit most. This adjustment, in conjunction with several other data-activated strategies, led to a 12% increase in retention within just one year and enhanced the program’s impact on a broader population.
Many institutions miss the surprisingly powerful strategy of consistent initiative analysis for improving student success. By regularly evaluating initiatives and services, they can pinpoint specific opportunities for adjustments that enhance student persistence and ensure strategic investment in the right resources. This proactive approach is crucial for transforming student outcomes and elevating the institution’s overall success.
Student Success Impact Report 2024
Explore key trends and insights about which interventions are improving student persistence from across the Civitas Learning community.
