Do PhD supervisors make the grade?

Poor theses often betray inexpert supervision, say external examiners David Alexander and Ian Davis

February 20, 2014

Source: Getty

It is not acceptable simply to rely on supervisors who understand research methods and have a passing understanding of a given field

Over the past few years, we have often been asked to act as external examiners for doctoral theses, both in Britain and overseas. Between us we have conducted 16 such examinations, acting variously as external examiner, head of the examination commission, faculty opponent, corresponding examiner and so on, according to the system under which the thesis has been submitted.

We have dealt with both monograph dissertations and those that are collections of papers linked by an introductory essay. In short, we have a good body of evidence to ponder.

Our experience does not lead us to criticise any particular system of examination or type of thesis. However, it does raise serious issues about the quality of work submitted for the PhD degree (or its equivalent) and the standards employed to judge such work.

To cut to the chase, a significant number of the theses we have examined did not deserve to pass – at least, not in the form in which they were submitted. One of us has examined six doctoral theses in the past year and believes that not one of them was worthy of the degree. Yet he had the means at his disposal to fail only two of them. Administrative conventions and examination procedures, not to mention social pressures, simply did not allow the possibility of failure.

The other has been the external examiner for 10 PhD dissertations in the UK, Sweden and the Netherlands in the past two decades, of which six fell well below the appropriate standard, requiring extensive additional work. A seventh had to be rejected outright, requiring a total rewrite.

We do not believe that we set impossible, unrealistic or unfair standards for doctoral work. Study at PhD level provides opportunities for candidates to learn how to carry out research and is, above all, about training in the management and documentation of knowledge.

However, we do believe that successful candidates need to demonstrate a certain mastery of their field, especially the topic they have set out to study, in order to demonstrate that the training has borne fruit. The thesis should also offer a contribution to the advancement of knowledge – perhaps modest, but nevertheless robust against criticism.

Under the classic model, the PhD candidate conducts research under the supervision of an expert in his or her field and the guidance of a committee of other experts.

It is usual, although not essential, for supervisors to hold the qualification, demonstrating that they are aware of what is involved in attaining it.

Once the dissertation is complete and has been fully scrutinised by the supervisor and committee members, and once the candidate has incorporated all necessary modifications, it is submitted for examination.

At this point, the supervisor’s judgement is that the work is ready and adequate for the test, which is conducted by an impartial external expert (in some cases with the support of an internal examiner who did not take part in supervising the candidate).

No doubt there are many doctoral theses that reach this stage in fine condition and are fully worthy of the accolades that they will eventually receive. Indeed, we have both enjoyed assessing such examples.

However, it is a concern that a significant number that reach this stage in what should be a rigorous process are clearly substandard.

Although we have no statistical data, our own experiences leave us feeling distinctly uneasy about the state of doctoral studies in the second decade of the 21st century.

The PhD should be the gold standard and there should be no excuse for poor performance. Even if bachelor’s and master’s degrees are being devalued by grade inflation or other examples of falling standards, the doctorate should be different.

The criterion for acceptance is clear: that it be a piece of research that withstands all the tests and criticisms of science, one that is defensible in the face of expert and peer review.

Yet in recent years, the quality of work judged ready for final assessment has put us both in difficult situations.

Doctorate qualifications obtained at UK institutions 1994-95 to 2012-13

We have had to explain to candidates that their work is unacceptable – in some cases irredeemably so. We have sat in closed rooms while a committee desperately searches for a face-saving formula by which a dreadful piece of work can be awarded a PhD degree.

Meanwhile, the corridor outside throngs with members of the candidate’s family, waiting for the inevitable happy announcement that follows a meaningless deliberation in camera.

On other occasions, we have even refused to travel to the candidate’s university on the day of his or her defence. We informed the university that the thesis was simply not capable of being examined and that we were resigning from the post of external examiner.

A bad PhD thesis is labour-intensive to examine, as every criticism needs to be carefully substantiated. It often feels as if one is doing the candidate’s research for him or her, especially when the person in question has not read and reviewed essential literature – an elementary but not infrequent fault. More importantly, by doing so, one is doing the supervisor’s work, because a substandard thesis is evidence of inadequate supervision; the candidate’s supervisor should have picked up the deficiencies and ensured that they were put right.

It can be intensely embarrassing when the supervisor – or the whole supervisory committee – is present at the examination and the faults are revealed one by one.

For example, some years ago, one of us (Ian Davis) was asked to examine a PhD thesis in a geography department in a university in the North West of England. During a pre-exam lunch with the candidate’s supervisors, he queried why the most basic texts on the subject had not been referred to, and asked whether there was a simple explanation for the candidate’s failure to consult with experts in any of the key centres in the UK and in the country where the fieldwork was conducted. One of the examiners candidly admitted that the reason the texts had been ignored and centres not consulted might have been due to his own ignorance of their existence.

But, as the difficult conversation developed, the other supervisor heatedly asked him who he thought he was examining – the student or his supervisors?

In this case there was evidently no one in the department with knowledge of the field of disaster recovery, yet despite this, the university had accepted the doctoral candidate.

The incentives to do so are clear. It is, of course, a matter of prestige for an academic department to have PhD students. It is one factor that betokens a research agenda, and possibly a serious dedication to research. However, research studentships come with obligations.

Moreover, independent, impartial external examination should be exactly this, free of pressure to pass work if it is not acceptable, whether this pressure is administrative, social or scientific in origin.

In the past it was probably broadly acceptable for the outcome of the PhD examination to be a foregone conclusion: by convention, a work was not submitted unless it was ready. Standards were determined by a combination of acceptable scientific practice and tradition from past PhDs, and candidates who were not capable of writing an acceptable thesis never made it to the examination room. As we have argued here, this is no longer the case.

There have, of course, been momentous changes to the research environment in recent years. For example, in our field, disaster studies, two of the leading peer-reviewed journals, Natural Hazards and Natural Hazards and Earth System Sciences, have increased their publication rate 10-fold in the past decade or so. Hence, there are more articles to read, more advances in research that must be taken into account, and greater competition in the whole academic process.

On the other hand, since the arrival of the internet literature searches have become vastly easier; word processing has revolutionised the production of the thesis itself; and advances in information technology have done likewise to the calculations and computations in the analysis on which the candidate reports.

Whatever the pressures of modern life, we need to revert to older standards. We will continue to examine theses, if requested, and will carry on striving to be fair, impartial and reasonable in our judgements. But at the same time we will be rigorous.

In our view, two changes would raise the standards of PhD research. First, however tempting in times of financial pressure, no university research committee or individual should permit PhD students to be accepted unless there are acknowledged, internationally recognised experts among their staff who can supervise. It is not acceptable simply to rely on supervisors who understand research methods and have a passing understanding of a given field. This is particularly important in the case of overseas students, who may not be familiar with a university’s strengths when applying.

Second, there is one change we will be making when we are asked to examine PhDs: we will not accept the task of examining any thesis without having looked at it, at least in the form of a late draft, to ascertain whether it is worth examining at all.

For the good of science, we urge all those involved in the production and examination of PhD work to do the same and to ensure that the “gold standard” of the PhD is upheld.

Male and female lifeguards (black and white)

Qualified views: are doctorate standards improving or deteriorating?

This is not the first time that concerns about PhD standards have been raised in the pages of Times Higher Education.

Writing in 2002, for example, Lyndon Smith, then a senior lecturer in the Faculty of Computing, Engineering and Mathematical Sciences at the University of the West of England, argued that the British PhD was “in danger of becoming a joke”.

He said that respect for the qualification was being undermined by “a tendency to award PhDs to candidates who have not published nor been subjected to truly independent appraisal”.

In 2009, a vox pop of THE readers identified concerns about mounting pressure to accept PhD students, the level of support available to candidates, and about supervisors being given too many students to supervise.

“It is inevitable that a push for quantity of PhD students leads to diminution of quality because resources never match the increases, leading to poor supervision, disappointed students and lower-quality work, inadequately challenged,” said Roger Cowell, then at the University of Leeds’ Centre for Innovation in Health Management.

“Some PhD work is a marker of original scholarship, but it is the exception. This is sad for the students concerned; but, ironically, this number-crunching approach may well prepare them better than rigorous scholarship for the realities of publication-driven standards in modern academic life.”

Some respondents, however, argued that standards seemed to be rising, while others said that it was impossible to judge (and futile to discuss) whether standards had risen or fallen owing to lack of data.

Clare Bambra, then a lecturer in the department of geography at Durham University, said that “for students in research-intensive universities, who are usually externally funded, I would say that the expectations and therefore the quality required to pass are higher than they used to be. Most are expected to produce peer-reviewed journal papers during their PhD as well as to produce a traditional thesis.”

Constantine Sandis, then a senior lecturer in philosophy at Oxford Brookes University, said: “One cannot help but be suspicious of claims that current PhDs are generally lacking in quality when these claims are not backed by any evidence comparing samples of recently submitted PhD theses to appropriate samples of past ones – from whatever alleged ‘golden age’ we are supposed to be nostalgic about.”

Others have argued that the development of professional doctorates, also known as taught doctorates, has undermined the reputation of the UK PhD. The 2010 Vitae researcher development conference in Manchester heard claims that there were “too many” taught doctorates.

Meanwhile, data from the Higher Education Statistics Agency show the growth in the number of students achieving a doctorate in the UK each year (see chart/graph).

Not all students registered for PhDs leave with a doctorate, although completion rates have crept up in recent years. According to the Higher Education Funding Council for England’s most recent statistics, 72.9 per cent of the 11,625 students from the UK or the EU who began full-time doctorates at English universities in 2010-11 will obtain a degree within seven years. This compares with 70.1 per cent of those who started in 2009-10 and 70.5 per cent from 2008-09. However, qualification rates at some institutions are much lower. The lowest predicted pass rate is recorded by London Metropolitan University: only 12 per cent of its 2010-11 cohort of 45 students are expected to qualify within seven years, according to Hefce. London Met’s “benchmark” figure, calculated according to the background of its students, is 70 per cent. Another six universities are expected to have completion rates of less than 50 per cent.

Tara Brabazon, head of school and professor of education at Charles Sturt University, Australia, has written a series of popular articles for THE about PhDs. They include:

Times Higher Education reporters

Times Higher Education free 30-day trial

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Reader's comments (9)

I found this article to be negative in its tone and demeaning of those who have invested an awful lot of hard work into their phd. It would have been more useful to provide some constructive criticism rather than list numerous negative anecdotes as if they represent the 'state' of Doctoral theses and supervisory relationships today. I would recommend any other phd students reading this to look at this link for helpful insights and tips of how to do well in the viva and how to avoid many common pitfalls: http://patthomson.wordpress.com/page/2/
When undertaking my doctorate a few years back, I took the time to read through a range of PhD/EdD thesis to get a sense of what was required. Some were excellent, but I distinctly recall wondering how some of them had ever passed - in the worst there were referencing errors, glaring methodological inconsistencies and lots of irrelevant padding.
I rather agree that we focus on the negatives as many PhDs are good. Given that we are in a mass system for PhDs compared to when I was a student, it is to be expected that the dynamic range (as per undergraduates) is larger than in past times. I speak as an experienced supervisor and examiner who has examined at a range of old and new universities.
I've hardly spent any time with my PhD supervisors for two reasons; they're not experts in my chosen field (one has knowledge related to it) and secondly, they're never around. It can take months to receive a reply to any query as they are busy with day to day business and other work, and by the time I do receive a reply, I've moved on to something else. It may be due to the cross discipline nature of my project, but useful guidance is a rare thing.
I too found this article something of a rant and not particularly revealing about the state of doctoral study. In fact I was left with the impression that the authors of the article must have been the inspiration for this excellent article from the Thesis Whisperer. http://thesiswhisperer.com/2013/02/13/academic-assholes/
To pass the thesis should contain work suitable for a paper in a respected, refereed journal, or copies of papers already accepted or published.
Oh dear! I'd been looking forward to reading this but what I hoped was going to be an interesting and balanced look at the state of UK Doctoral education, was, in fact, a rant from a couple of professors who obviously don't want to examine anyone anymore?! I can't see any other reason for such a pompous piece and I doubt that they would be high on anyones list of potential examiners! As one of the 20'000 people who got a PhD in 2010/11 (yes, lovely arbitrary and completely meaningless graph) I worked hard for that and it wouldn't take much effort for you to find evidence that I had some interesting supervisory moments! I deserved my PhD as I am sure the majority do. However, there is a really nasty undertone to this article suggesting that those of us who have gained a PhD in the last 10 years or so are somewhat inferior. I expect better from people in their position - if this was a PhD it would need major revisions. I also expect better from the THE, yes be provocative but don't print sub-standard, poor quality polemics just to massage the ego of Professor so and so!
I've examined 114 students in over 10 countries (not just Europe either), and I find very little to support this article. Of course, I might have only been asked to examine better PhD work, so my sample is really only 1 data point, but that's 50% of the authors' of this article....I am (almost always) pleasantly surprised by the consistency of what makes a PhD (in Computer Science) in Austria, Australia, America, France, Spain, Italy, Finland Hungary, Switzerland and Sweden, amongst others...
The authors pose an important question and focus on what they perceive as a problem (the negative answer to the title) rather than celebrating success, which for them should be the rule. I note a minor contradiction in that the older way of doing things asked more preparation from the student and more diligence from the supervisor, but wasn't necessarily associated with more rejections. "Whatever the pressures of modern life…" As much as I agree with the authors that they have raised an important problem (and address the "more experienced" audience and those inexperienced supervisors who will take the time to read them, but not so much the Ph.D. students themselves) I do not particularly think that it can be solved simply by enhancing the examination part of the equation. The whole question of how much pressure is put on the academic and whether such pressure is beneficial to science is way more significant, in my view. Yet, overall, I have welcomed this contribution, as much as I sympathise with the student reader. The time limit on their studentships, for example, is bad policy. Others may disagree...

Have your say

Log in or register to post comments

Featured Jobs

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

Microlight pilot flies with flock of cranes

Reports of UK-based researchers already thinking of moving overseas after Brexit vote

Portrait montage of Donald Trump and Nigel Farage

From Donald Trump to Brexit, John Morgan considers the challenges of a new international political climate