Student retention software comes under microscope

By Carl Straumsheim, for Inside Higher Ed

November 11, 2013

Purdue University in Indiana has for years touted the ability of its early-warning system, Signals, to improve student retention, but a series of blog entries analysing the institution’s claims has not found a causal connection between students who use the system and their tendency to stick with their studies.

Purdue’s method of structuring its early warning system has permeated the industry, and research invalidating its results could have sent shockwaves through its competitors. The university’s approach is not being called into question, however, only its claims to boost retention - which, on one hand, is likely to come as a relief to the many software providers that have attempted to recreate what Signals does. On the other hand, the rationale given by many for using such early-warning systems is in fact to improve retention.

Signals combines demographic information with online engagement and produces a red, yellow or green light to show students how well they are doing in their courses - and provides that information to their professors so they can provide help to students before they drop or fail. Ellucian, which provides administrative software, sells it as the commercial product Course Signals, while educational software providers Blackboard and Desire2Learn offer many of the same features through Retention Center and Student Success System, respectively.

Michael Caulfield, director of blended and networked learning at Washington State University at Vancouver, decided to take a closer look at Signals after Purdue in a September press release claimed taking two Signals-enabled courses increased students’ six-year graduation rate by 21.48 per cent. Mr Caulfield described Purdue research scientist Matt Pistilli’s statement that “two courses is the magic number” as “maddening”.

Comparing the retention rates of the 2007 and 2009 cohorts, Mr Caulfield suggested much of what Purdue described as data analysis just measured how many courses students took. As Signals in 2008 left its pilot and more students across campus enrolled in at least one such course, Mr Caulfield found the retention effect “disappeared completely”.

Put another way, “students are taking more…Signals courses because they persist, rather than persisting because they are taking more Signals courses”, Mr Caulfield wrote.

His findings were last month corroborated by Alfred Essa, McGraw-Hill Education’s vice-president of research and development and analytics, who wrote a simulation that substituted “received a piece of chocolate” for “took a Signals-enabled class”.

”The simulation data shows us that the retention gain for students is not a real gain (i.e., causal) but an artifact of the simple fact that students who stay longer in college are more likely to receive more chocolates,” he concluded. “So, the answer to the question we started off with is ‘No.’ You can’t improve retention rates by giving students chocolates.”

Mr Essa helped design the Student Success System as a strategy director for Desire2Learn, but said it and other products that have been inspired by Signals don’t face an existential crisis.

“The aim of these early warning alert systems at the course level is just to make sure that students are performing well,” Mr Essa said. “It’s a huge leap to go from that and say, ‘Oh, and we’re also going to improve your retention rates directly.’ ”

Dr Pistilli defended the claims about Signals’ ability to increase retention - with the caveat that more research needs to be done. “The analysis that we did was just a straightforward analysis of retention rates,” he said. “There’s nothing else to it.”

To ensure an empirically grounded analysis of Signals, Mr Essa urged Purdue to give researchers access to as much data as possible. Dr Pistilli said he is open to participate in that conversation, but pointed out that granting open access could violate students’ privacy rights.

With Signals marking its fifth anniversary this year, Dr Pistilli said “it was probably just a matter of time for people to start looking for these pieces and begin to draw conclusions”. In that sense, the discussion about early warning systems resembles that of other ed-tech innovations, like flipping the classroom and massive open online courses, where hype drowns out any serious criticism.

“I think part of the answer is we’re really bad at statistical reasoning,” Mr Essa said. “Even experts get tripped up by statistics, and it’s very easy to make claims like this, but it’s difficult to dig in and try to make sense of it.”

He added: “Maybe one of the conclusions that could be derived from this is that we really don’t have a strong community to test and validate these claims? Maybe that’s really the starting point of discussion in the academic community. As we move forward with new technologies in learning analytics, how and who will be evaluating the claims that people put forward?”

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Assistant Dean (International) UNIVERSITY OF THE WEST OF SCOTLAND - PAISLEY CAMPUS
Assistant Dean (Research & Enterprise) UNIVERSITY OF THE WEST OF SCOTLAND
Professor of New Media UNIVERSITY OF THE WEST OF SCOTLAND
Professor of Sport UNIVERSITY OF THE WEST OF SCOTLAND
Professor of Strategic Management UNIVERSITY OF THE WEST OF SCOTLAND - PAISLEY CAMPUS

Most Commented

James Fryer illustration (8 September 2016)

Some lecturers will rightly encourage forms of student interaction that are impossible for those covering their faces, Eric Heinze argues

Handwritten essay on table

Universities must pay more attention to the difficulties faced by students, says Daniel Dennehy

University of Oxford students walking on campus

University of Oxford snatches top spot from Caltech in this year’s World University Rankings as Asia’s rise continues

Theresa May entering 10 Downing Street, London

The prospect of new grammar schools on the horizon raises big questions for HE, writes Nick Hillman

Nosey man outside window

Head of UK admissions service Mary Curnock Cook addresses concerns that universities might ‘not hear a word’ from applicants