Tackle ‘reporting bias’ in pedagogy research, paper warns

Author of new research says journals are ‘not doing’ enough to combat the phenomenon

February 1, 2017
man looking through giant rose tinted sunglasses
Source: Reuters
Rose-tinted: learning and teaching research is particularly susceptible to reporting bias owing to factors such as the ‘positive agendas’ of funding bodies, found the study

Learning and teaching journals are “not doing enough” to combat reporting bias in research from the field, according to the author of a new paper on the subject.

Phillip Dawson, associate director of the Centre for Research in Assessment and Digital Learning (Cradle) at Deakin University, also said that research funding organisations must help to reduce the phenomenon, which occurs when researchers selectively report significant positive results and omit non-significant or negative results.

“When higher education research is used to inform practice, positive biases can falsely inflate our confidence that a particular learning and teaching approach is going to achieve the desired outcome,” his paper, “Sharing successes and hiding failures: ‘reporting bias’ in learning and teaching research”, states.

The study found reporting bias during a meta-analysis that compared lectures and active learning in science, technology, engineering and maths (STEM) disciplines. The analysis incorporated two approaches to understanding the prevalence of reporting bias: sourcing authors’ publication habits and examining publication patterns around recently completed doctorates.

“This meta-analysis produced [results] which equated to students in lecture classes being 1.5 times more likely to fail compared to their active learning peers,” the paper in the journal Studies in Higher Education states. After re-examining the data according to the dissemination type of the studies – published studies or dissertations – the “strength of support for active learning differed”.

Published studies were “far more supportive of active learning than dissertations” and there was “statistical evidence of four missing studies…likely to have been conducted” but not published or captured by the researchers. These studies would have “reached the opposite conclusion” to those included. This is “problematic”, Professor Dawson writes, because it “represents bias”.

The analysis suggests that learning and teaching research is particularly susceptible to reporting bias because of a range of factors, including the “positive agendas” of funding bodies.

“Funding bodies want to fund innovation in learning and teaching, which is a positive thing,” he told Times Higher Education. “[Australia’s] Office for Learning and Teaching (OLT), and the [UK’s] Higher Education Academy (HEA) and Jisc, want to fund studies that work.

“We didn’t find evidence of projects funded by [these organisations] publishing negative results. Literature that’s just full of success stories…is not a full picture.”

The paper suggests there are also issues around academic freedom, and teaching and learning researchers may not publish poor teaching practice at their institution for fear of reprisal. Publishing negative results would be “bold” for a professional staff member who lacks academic freedom. By encouraging this, however, universities may be “institutionalising reporting bias”.

Professor Dawson suggested that higher education needs to do more to combat reporting bias because “left unchecked” it could result in “policymakers and educators distrusting supposedly evidence-based improvements to teaching and learning”.

“Journals are not doing enough to address reporting bias, in fact they are doing nothing I am aware of,” he said. “Funding bodies [also] have a role to play. At the end of the day, the OLT and HEA, they want to advance teaching and learning. People within them…believe sharing bad results is a good thing.

“[These groups] need to say: we want the full story. Journals should encourage authors to include all results from their studies, not just the sexy results. [And] universities need to say that your academic freedom extends to both your teaching and research, but also to the nexus between them.”

In response, Mark Jones, HEA’s chief operating officer, said it was in everybody’s interests that research gives a “balanced and accurate picture”, and “we strongly encourage and expect researchers to follow this principle”.

Phil Richards, chief innovation officer at Jisc, added that the organisation “no longer funds research in this space”, but was “very aware” of reporting bias being a potential problem in research of this kind and was compiling an evidence base to help universities understand what works in technology enhanced learning.

“For this to work for the benefit of students paying £9K fees, it is essential that the underlying research is sound and balanced; anything that calls that into question must potentially be a matter of concern,” he said.

“When we funded development and evaluation projects, we were very aware of reporting bias as a potential problem and were always at pains to emphasise to projects that [a] failure to achieve a positive result was an entirely valuable conclusion, and one to be shared.”

john.elmes@tesglobal.com

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

PhD Scholar in Medicine

University Of Queensland

Manager, Research Systems and Performance

Auckland University Of Technology

Lecturer in Aboriginal Allied Health

University Of South Australia

Lecturer, School of Nursing & Midwifery

Western Sydney University

College General Manager, SHE

La Trobe University
See all jobs

Most Commented

women leapfrog. Vintage

Robert MacIntosh and Kevin O’Gorman offer advice on climbing the career ladder

Woman pulling blind down over an eye
Liz Morrish reflects on why she chose to tackle the failings of the neoliberal academy from the outside
White cliffs of Dover

From Australia to Singapore, David Matthews and John Elmes weigh the pros and cons of likely destinations

Mitch Blunt illustration (23 March 2017)

Without more conservative perspectives in the academy, lawmakers will increasingly ignore and potentially defund social science, says Musa al-Gharbi

Michael Parkin illustration (9 March 2017)

Cramming study into the shortest possible time will impoverish the student experience and drive an even greater wedge between research-enabled permanent staff and the growing underclass of flexible teaching staff, says Tom Cutterham