Validating learning in the age of generative AI: on the need to keep credentials credible
Words by Oliver Vettori, dean of accreditation and quality management, and director of programme management and teaching and learning at Vienna University of Economics and Business.
There is a wonderful comic strip from Charles M. Schulz’s Peanuts, first published in March 1989, where Charlie Brown’s sister Sally is preparing for a history test. As people familiar with Peanuts know, school is not Sally’s favourite pastime. But, “fortunately”, as she states, the test will probably be “mystical choice”. When her brother corrects her with a brief “multiple choice”, her reply is quite typical: “Whatever.”
As with many Peanuts strips, this one allows for different interpretations. The wordplay brilliantly puts its finger on something we are all dealing with in higher education right now: the meanings of assessment have become elusive, or at least ambiguous. The increasing variety of challenges that digitalisation, including generative AI, brings to our institutions adds to this ambiguity. How do we assess learning at a time when the very notion of knowledge has become, in the words of social theorist W.B. Gallie, “an essentially contested concept”?
But this is not just a matter of the (ab)use of technology or the correct assessment approach, and not even a matter of finally getting constructive alignment right: it is a matter of higher education’s survival.
Today’s incredible variety of higher education institutions has little in common with the first medieval universities, even though some of the original characteristics are experiencing a renaissance (student co-creation and personalised study paths had been largely forgotten for some centuries, despite arguably being a timely reinvention of the involvement that the first students had in creating their own curricula). But there is one core element that has survived over hundreds of years and can be understood as the very basis of higher education’s endurance and global success: the provision of standardised academic credentials.
As the Latin roots of the word belie, for credentials to work, society has to trust in them: trust in the authority they instil in the person bearing the credential and trust in the legitimacy of the institution that has issued the credential. Usually, there is a basis for this trust. Credentialing typically entails the verification and validation of knowledge and skills. So, what happens if higher education cannot guarantee this validation? There are several ways this could play out, and none of them appeals to me.
It would be too easy to blame AI for this situation. Many higher education institutions and professionals stopped prioritising the validation of learning long before the proliferation of AI. The “mystical” quality Sally found in her tests might be related to the ritualism that pervades much of assessment. Assessment has become a chore rather than core. And many students have, understandably so, a far greater interest in grades than grading.
At the same time, the rapid infiltration of generative AI across all aspects of higher education has certainly aggravated the related challenges. Are all unsupervised assessments compromised? Will we see a resurgence of on-campus exams, both written and oral, even in cases where we abandoned them for good reason? Can we build AI-resilient or even AI-embracing assessment schemes? Higher education conferences, seminars and online discussion boards are overflowing with debates on these and similar questions.
This is a great opportunity, too, of course: to deeply reflect on what kind of higher education is needed in today’s societies. The post-truth era coincides rather uncomfortably with the era of ‘no need to know’. And a sector of expert organisations should be well equipped to tackle these kinds of challenges.
But there are two aspects we need to make sure of when fighting for trust in our credentials. First, decisions on this cannot be delegated to the individual. There are plenty of institutional policies on AI and assessment that leave the handling of the problem to the individual teacher. If policies are supposed to guide behaviour, they are unlikely to succeed if they resist providing guidance.
I have been researching the issue with a colleague, examining how teachers in higher education are dealing with generative AI, assessment and academic integrity in their course syllabi. The landscape of attitudes, approaches and justifications in our research material was so fragmented that for any learner only one message seemed to shine through: anything goes. There is a need to come to an agreement across academic communities and higher education institutions. If higher education’s ability to validate learning is an institutional priority, it needs to be addressed at the institutional level.
Second, effective validation of learning in the age of generative AI requires a holistic rather than atomistic approach. Assessment might be something which, for organisational, technical and various other reasons, is typically handled at course level, but the validation of learning is a pan-curricular undertaking and needs to be considered across the student lifecycle.
Admittedly, this also raises complexity, particularly in institutions with many disciplines, academic programmes and teaching cultures. A meso-level compromise would be to approach the validation of learning at programme level, aiming for coherence between the different building blocks of the programme. This, again, calls for alignment, and thus a constructive dialogue between the educators involved. I am well aware that this is time-consuming and fraught with conflict. It is, unfortunately, much easier to ask an AI agent for advice, even when it comes to quandaries that at least partly originate in generative AI. Yet I have severe doubts about the effectiveness of such an approach, particularly at system level. Individual solutions are not ideal for institutional conundrums.
We need to keep our credentials credible – degree-level ones as much as any microcredential. Doing so calls for concerted efforts and for longitudinal learning validation. And, referring to another Peanuts strip, we might want to avoid making Peppermint Patty happy. After complaining about being tasked with an essay rather than a multiple choice or, even better, true or false test, Patty ends the comic with the words: “I hate it when you have to know what you’re writing about…” The joke leaves an uneasy feeling: of course, we want our students to understand what they are doing and why. So should we.
