THE China Subject Ratings 2020: methodology

Browse the full results of the China Subject Ratings 2020

The Times Higher Education China Subject Ratings are the only performance tables that judge Chinese institutions against their global peers based on the Chinese Ministry of Education’s classification of subjects. The ratings measure research-intensive universities across all their core missions: teaching, research, international outlook and knowledge transfer. We use 11 carefully calibrated performance indicators, listed below, to provide the most comprehensive and balanced comparisons, trusted by students, academics, university leaders, industry and governments. With 89 subjects included, the ratings provide grades across more subjects than any other major rankings or ratings provider.

The performance indicators are grouped into five areas: Teaching (the learning environment); Research (volume, income and reputation); Citations (research influence); International outlook (staff, students and research); and Industry Income (knowledge transfer).

The data used for the THE China Subject Ratings 2020 are drawn from the THE World University Rankings (WUR) 2020, the THE Reputation Survey 2020 and Elsevier bibliometrics. We have published 12 pillar-level ratings, which measure performance against 89 subjects in total. We have also published an overall table that provides an overview of how institutions have performed across the different areas.

The metrics

Teaching (the learning environment)

• Reputation survey
• Academic staff-to-student ratio
• Institutional income per academic staff member

The most recent Academic Reputation Survey (run annually) that underpins this metric was carried out from November 2018 to March 2019. It examined the perceived prestige of institutions in teaching. The responses were statistically representative of the global academy’s geographical and subject mix. The 2019 data are combined with the results of the 2018 survey, giving more than 21,000 responses.

The academic staff-to-student ratio is defined as total full time equivalent (FTE) number of staff employed in an academic post divided by FTE number of students in all years and of all programmes that lead to a degree, certificate, university credit or other qualification.

The measure of institutional income indicates an institution’s general status and gives a broad sense of the infrastructure and facilities available to students and staff. This metric is generated by dividing the institutional income adjusted to PPP (purchasing power parity), by the total number of academic staff members.

Research (volume, income and reputation)

• Reputation survey
• Research income per academic staff
• Publications

This reputation survey metric measures a university’s reputation for research excellence among its peers, based on the responses to our annual Academic Reputation Survey (see above.

Research income is scaled against academic staff numbers and adjusted for purchasing power parity (PPP). This is a controversial indicator because it can be influenced by national policy and economic circumstances. But income is crucial to the development of world-class research, and because much of it is subject to competition and judged by peer review, our experts suggested that it was a valid measure. This indicator is fully normalised to take account of each university’s distinct subject profile, reflecting the fact that research grants in science subjects are often bigger than those awarded for the highest-quality social science, arts and humanities research.

To measure productivity we count the number of publications published in the academic journals indexed by Elsevier’s Scopus database per scholar, scaled for institutional size and normalised for subject. This gives a sense of the university’s ability to get papers published in quality peer-reviewed journals. Last year, we devised a method to give credit for papers that are published in subjects where a university declares no staff.

Citations (research influence)

Our research influence indicator looks at universities’ role in spreading new knowledge and ideas.

We examine research influence by capturing the average number of times a university’s published work is cited by scholars globally. This year, our bibliometric data supplier Elsevier examined 77.4 million citations to 12.8 million journal articles, article reviews, conference proceedings, books and book chapters published over five years. The data include more than 23,400 academic journals indexed by Elsevier’s Scopus database and all indexed publications between 2014 and 2018. Citations to these publications made in the six years from 2014 to 2019 are also collected.

The citations help to show us how much each university is contributing to the sum of human knowledge: they tell us whose research has stood out, has been picked up and built on by other scholars and, most importantly, has been shared around the global scholarly community to expand the boundaries of our understanding, irrespective of discipline.

The data are normalised to reflect variations in citation volume between different subject areas. This means that institutions with high levels of research activity in subjects with traditionally high citation counts do not gain an unfair advantage.

We have blended equal measures of a country-adjusted and non-country-adjusted raw measure of citations scores.

This metric is not used for five subjects where the number of papers produced globally is very low. These subjects are:

• History of science and technology
• Music and dance studies
• Drama and film studies
• Fine arts
• Design

International outlook (staff, students and research)

• Proportion of international students
• Proportion of international academic staff
• International co-authorship (proportion of international publications)

The ability of a university to attract undergraduates, postgraduates and faculty from all over the planet is key to its success on the world stage.

International students and staff are those whose nationality differs from the country where the institution is based. The metrics are calculated as the total FTE number of international students or academic staff divided by the total FTE number of students or academic staff.

In the third international indicator, we calculate the proportion of a university’s total research journal publications that have at least one international co-author and reward higher volumes. This indicator is normalised to account for a university’s subject mix and uses the same five-year window as the “Citations: research influence” category.

Industry income (knowledge transfer)

A university’s ability to help industry with innovations, inventions and consultancy has become a core mission of the contemporary global academy. This category seeks to capture such knowledge transfer activity by looking at how much research income an institution earns from industry (adjusted for PPP), scaled against the number of academic staff it employs.

The category suggests the extent to which businesses are willing to pay for research and a university’s ability to attract funding in the commercial marketplace – useful indicators of institutional quality.

Metric weightings

The metric weightings for each subject are calculated in line with the related subject weightings used in the WUR. Two of the metrics used in the WUR are not used in this rating: doctorate-to-bachelor’s ratio and doctorates-awarded-to-academic-staff ratio. The weights of these unused metrics are redistributed to increase the weighting of the remaining metrics by a constant ratio, bar the two reputation metrics, which remain static. A full list of the metric weightings for the various subjects are included in the detailed methodology document attached to the bottom of this article.

Grade production

Once the overall scores have been produced, a grade is calculated for each university within each subject using a grading system of A+ to C-. These grades are evenly split across the ranked universities across the world, such that the top 11.11 per cent of world universities in a subject receive an A+, the next 11.11 per cent receive an A, etc.

Subjects included

China’s Ministry of Education categorises 111 subjects across 13 pillars. The ratings measure performance across 89 of these subjects within 12 pillars. These pillars are:

• Agriculture
• Arts
• Economics
• Education
• Engineering
• History
• Law
• Literature
• Management
• Medical
• Philosophy
• Physical Science

We excluded 14 subjects because they had a strong military and/or national security theme, while a further seven subjects were excluded because they were too specific to China and therefore difficult to compare internationally. While we have included a law pillar, which includes subjects related to the area of law, we also excluded the subject of law because we did not have sufficient data for the international comparisons. A full list of the subjects we have included can be viewed in the detailed methodology document attached to the bottom of this article.

Data collection

The data used for the THE China Subject Ratings 2020 are drawn from the THE World University Rankings (WUR) 2020, the THE Reputation Survey 2020 and Elsevier bibliometrics. Institutional data have been provided and signed off by universities.

On the rare occasions when an institution does not provide a data point, thus making it impossible to generate a metric score, the missing metric score is calculated using whichever is the highest value generated from the following:

• The average of the two lowest metric scores for an institution; or
• The minimum score awarded across the whole population for that metric.

Getting to the final result

Moving from a series of specific data points to indicators to a total score for an institution requires us to match values that represent fundamentally different data. To do this we use a standardisation approach for each indicator, and then combine the indicators in the proportions indicated above.

The standardisation approach we use is based on the distribution of data within a particular indicator for each subject, where we calculate a cumulative probability function, and evaluate where a particular institution’s indicator sits within that function.

For all indicators we calculate the cumulative probability function using either a version of Z-scoring, Exponential, Generalised Normal, or Weibull component.

Exclusions

There are four key criteria for universities to be included in the subject ratings:

• They must be included in the World University Rankings 2020 and have applied for inclusion in the World University Rankings 2021
• They must have been eligible for the subject rankings related to the World University Rankings 2020
• They must have selected the relevant subject during the World University Rankings 2020 submission
• They must meet a minimum threshold for the number of papers published between 2014 and 2018 for each specific subject

Universities that meet these four criteria are included in the ratings for a given subject. All institutions that feature in at least one of the subjects are included in the overall table, which is designed to provide an overview of performance across the different subjects.