Why Data Matters for Student Success in a Post-Pandemic World

This article is adapted and expanded from an earlier article: Colleen Carmean, David Kil, and Linda Baer, “Why Data Matters for Student Success in a Post-Pandemic World,” EDUCAUSE Review, August 10, 2021.

Lessons learned from student impact data collected by Civitas Learning partners during the COVID-19 pandemic.

Higher education has long studied and shared what works to improve retention. Academic support, student engagement, and innovations in the classroom have again and again been found to keep college students on track to graduate. 

But what happens when a global pandemic hits, campuses shut down, and on-campus social and academic integration is interrupted? Can we still look to past research to determine student success in an increasingly digital future? 

We attempt to answer these questions in our study of 44 programs conducted by 17 institutions during the pandemic. Each campus examined in this study used the Civitas Learning Student Impact Platform initiative analysis capabilities to measure the effects on persistence of various student success programs. The ability for each institution to integrate standard and custom student data points within the Civitas Learning platform resulted in a more nuanced detailed analysis around campus response initiatives and student subpopulations. 

We found that much of what we know about persistence remains true, but there is also much to reconsider as the students entering higher education continue to diversify, bringing different experiences and needs with them to college.

How COVID-19 Disrupted Student Success Best Practices

Prior to COVID, and despite completion rates that are still unacceptably low for a population going to college, higher education was making progress in improving student success.  Best practices research and predictive analytics provided insight to retention efforts that supported students, and especially for students who were at risk for not succeeding in their higher education journey. Predictive analytics techniques assisted campuses in identifying students with high-risk factors for retention. 

With COVID, traditional place-based education, services, and interventions were no longer available. The impact on students was immeasurable. Lives were rapidly thrown into chaos with economic trauma, health concerns, isolation from family and friends, and the dramatic shift from predominantly onsite education moving to remote, emergency instruction models.  Large numbers of students ultimately dropped or stopped out and decided not to enroll in 2020³.

Traditional face-to-face student services were not available during the pandemic, and campuses struggled to determine effective practice on physically closed campuses. Schools suddenly and immediately halted face-to-face practices shown to improve retention of at-risk students in the past – like intensive early warning systems, holistic advising and mentoring, and strong individualized education planning. Equally unclear is what will happen when students return to campus. Disadvantaged populations have suffered unequal loss and as campuses work to recoup financially, they will also struggle with equity, student expectations, and how to evaluate and embrace long-overdue changes that were rapidly implemented in online teaching and learning. Post-COVID, campuses will be challenged to continue pandemic innovations and adaptations that are being demanded in the “next normal”^2.

Lessons Learned from Pandemic Response Data

Campuses that collected response data during the pandemic have learned lessons that will inform lasting change. For student success, pandemic lessons learned include reminders that it is crucial to know your own students: what they need, who is struggling and why, who is leaving when, and which interventions are most beneficial. 

This data is knowable and access to new, analytic data models and tools, designed to examine current practices,  allowed campuses to meet students where they were – struggling with contacts and connections when suddenly online, unsure how to get needed services, or where to turn in a crisis. Timely data matters more than ever as our student bodies become more diverse and our teaching modalities more varied. As the higher education landscape continues to evolve, institutions need the ability to adapt to highly dynamic and also unexpected circumstances more frequently. 

In the pandemic, each campus found itself in extraordinary circumstances, and there was no one size fit all strategy they could apply. As we move forward, it is crucial to use resources to measure what’s needed and what’s working well at specific institutions. Across varied institutions, we found that students responded when campuses stepped up with support, resources, and modalities. 

For many schools, results showed better student usage of pandemic response services than face-to-face support services of the past.  These findings are consistent with a student engagement theory that intentional interventions produce far greater impact since students in vulnerable situations are more receptive and responsive to the right interventions. 

Data analytics can reveal just-in-time student needs, struggles, and connections and then quickly evaluate solutions from an impact perspective. The landscape of higher education showed a remarkable shift online in 2020, and can now respond with evidence-influenced teaching, services, and support for an increasingly diverse nation going to college.

Which Pandemic Responses Most Benefited Students

What have we learned from campuses able to analyze the effect of rapid response pandemic initiatives on persistence using student impact data?

Each student and campus are unique, but several trends emerged in our research. Providing direct solutions for financial need – child care, food, federal CARES Act emergency funding, technology loans and gifts (take-home laptops and WiFi hotspots) – all showed significant effect on persistence. For many institutions, the pandemic revealed that students now struggle financially in ways not seen in previous decades. The highest lifts were often seen in emergency initiatives in direct financial support responses: Federal CARES money/Education Opportunity funds to students, laptop and hotspot loans, food box delivery. 

Other successful interventions were “low tech” moves as simple as advising by phone and email, as well as “high tech” software accelerators (self-paced, online learning courseware). Although specific data-informed actions differed by an institution and had a range of results, successful initiatives can be categorized into 3 programmatic themes that proved effective in supporting student persistence. 

Pandemic Response Intervention Themes

In addition to the three programmatic themes we describe above, we also identified several intervention themes, including:

1. Efficacy Drop for Traditionally In-Person Programs

Programs that used to rely on in-person engagement or group-based activities (lab- or group-based learning) experienced an efficacy drop during the pandemic. This reduction in effectiveness is possibly due to the lack of personal interactions or operational challenges in substituting engaging online learning experiences for previously in-person engagements

2. Personal Outreach and Engagement Matter for Advising

Advising communication modalities moving to asynchronous and digital mattered little in program efficacy as long as there was personal outreach and engagement (email, video chats, phone). This is particularly true when there were existing advisor-advisee relationships. 

3. Helping Vulnerable Students in Times of Crisis Works

Among the most effective programs, there was a theme of helping vulnerable students in times of crisis, resulting in significant lifts in persistence impact results. Examples include food link/delivery, CARES act, free WiFi access, counseling, free laptop loaners, and flexible grading.  As expected, there was an element of intersectionality, where diverse students in the lower prediction quartiles benefitted far more from these interventions. 

4. Social Support Programs Keep Students Connected

Programs that connect students to peers or mentors (even virtually), that promote remote collaborative study groups, and encourage social cohesion, such as fraternity-sorority living (FSL), were found to be highly effective pre-and post-COVID. In contrast, an adaptive remote learning lab initiative exhibited somewhat lower efficacy during the pandemic, possibly because of the lack of personal contact, community engagement, or forced learning in isolation.

5. Continuous Evaluation and Refinement of Initiative Effectiveness is Critical 

An efficacy trend comparison with baseline matching offers insights into how to improve program efficacy continuously. Institutions that embarked on term-based impact analysis discovered causal links between operational changes and changes in program efficacies for continuous process improvement. Such a closed feedback loop with rapid measurement is a recipe for strong collaboration between human intelligence and artificial intelligence and can result in sustainable improvement of student outcomes. 

6. Predication Scores Revealed Student Risk Level Better Than Phenotypic Measures

Drill-down impact results highlight the fact that student segments facing greater adversities and challenges benefit more from interventions. Instead of relying on narrow phenotypic metrics, campuses using the Civitas Learning Student Impact Platform were able to leverage prediction scores as an amalgamation of academic, behavioral, social-psychological, and demographic factors contributing to student success. In general, there is a strong negative correlation between prediction scores and intervention efficacy results, meaning the lower the prediction scores, the greater the benefits from such interventions.

Engagement Matters for Persistence

With the emergency shut down of physical campuses due to COVID, supporting students remotely became a challenge. Data analytics allowed prepared campuses to see and support students in real time, and to be able to analyze the results of student needs along with new digital modalities, supports, and services.

Each student is unique, each campus is unique, and the homogeneity we assumed of “traditional” students of the past will not be seen again.  Supporting students must accommodate both population and environmental diversity to be effective. We can no longer look to student success practices from five (or 25) years past. 

What did we learn as a rapid response collective that should guide us going forward? 

Engagement and belonging matter in persistence.

But engagement and belonging must be approached differently as “new traditionals” (including older, working, financially challenged, living at home students) spend less time on campus. During the pandemic, across the higher education landscape, services that pivoted to technology supporting online learners improved retention.  When students feel seen, heard, supported for who and where they are – not for who students were in years past, the students are more apt to stay. 

Initiatives shared by the seventeen institutions in this study highlight the importance of identifying and helping students in times of crisis. Visibility into current student data analytics allowed institutions to not only identify which students were in crisis but more importantly understand the underlying risk factors so they could connect those students to the most relevant resources to help them succeed. 

Data from multiple campuses showed a significant lift in retention where faculty and student services (advising, enrollment, financial aid) used technology to stay connected to students. A number of campuses put student connection teams together and immediately started calling or texting students about enrollment opportunities.  Advisors and faculty worked to stay in touch, sending increased communications to students.  Personalized systems allowed for more targeted messaging (nudges) to keep students engaged and on-task during chaotic times.  

Technology services that never existed before kicked into gear, putting laptops and wi-fi hot spots in the hands of students that didn’t have equipment at home. A move to digital tools, whether in providing hardware to connect from anywhere, or moving meetings to virtual meeting platforms or the phone, showed not only a rise in participation but a rise in persistence. Through hardship and frustrations, students stay when they feel seen, heard, and supported in their journey. Furthermore, such a persistence lift stayed consistent through the pandemic.  Innovative campuses will look toward diverse means of providing that sense of belonging to the “new majority” of students now going to college, including young, digital natives that find belonging online.

We have in the past also seen traditional lifts for students involved in clubs, student government, and on-campus jobs. Moving student workers online during the pandemic for mentoring/tutoring/peer support and more provided a significant lift in persistence both in the student workers and the students served.

Moving Forward

Whether campuses continue to strategically evolve, or whether they’ll return to a culture that long ago ceased to be effective or fair, will be the choice of institutional leadership. Analytics tools now exist to provide insights via discovery, interpretation, and communication of meaningful patterns in data—especially for today’s more time-challenged, intersectional, and diverse groups of students and the campus services that best support their needs.

An ongoing examination of persistence trends allows data-driven leaders a better understanding of access, engagement, and affordability and promotes strategic planning. Necessity is the mother of invention, they say, and higher education experienced much invention and innovation in its response to COVID-19. Colleges and universities went fully online while leaders and staff reimagined lectures, labs, discussions, collaborations, and assessments/grading. Support services also went fully online while staff used phones, emails, and texts to reach students. And higher education acknowledged inequities and provided food and emergency funding as well as technology (computers, Wi-Fi, online materials and tools) to students at home.

Campuses prepared with data analytics, tools, and skills were able to reach out and rapidly take action with the right resources to help data-identified students through their unprecedented struggles. The challenge will continue to be how best to move data to insights and then to action that supports persistence and mitigates barriers.

Data analytics allows campus leaders and staff to know their students via evidence, in real time, and helps them to be prepared to act on this knowledge. Data analytics creates belonging, leading to persistence and student success. The use of analytics in moving campus data to insights, and insights to effective action, will be key to an institution’s support and practice of student success.

Notes

1. Ernest Pascarella, et al. Long-term persistence of two-year college students. Research in Higher Education, 24,#1 (1986): 47–71.

2. Ed Glantz, et al. Improved Student Engagement in HigherEducation’s Next Normal. Educause Review: 2021, #3

3. Stay Informed. National Student Clearinghouse Research Center. (Spring 2021).

4. Beth McMurtie. Teaching: After the Pandemic, What Innovations Are Worth Keeping? The Chronicle of Higher Education.(April 1, 2021).

5.Society for College and University Planning . Planning for: Effective Data Analytics (2020).

Contributors