Using AI to think with students, not for them

• This guest blog was kindly authored by Mary Curnock Cook CBE, who chairs the Emerge/Jisc HE Edtech Advisory Board, and Bess Brennan, Chief Marketing Officer at Cadmus. Cadmus is running a series of collaborative roundtables with UK university leaders about the challenges and opportunities of generative AI in Higher Education.

Last month’s invitation-only meeting focused on assessment, with four higher education leaders—from the University of Greenwich, Imperial College London, University of Glasgow and Queen’s University Belfast—sharing their institutional frameworks and practical solutions. Together they took us through the cycle of assessment from the grassroots of assessment design through student and educator AI literacy to tech-led solutions that support students’ authentic progression and engagement with assessment tasks.

Central to all the discussions were the practical ways in which universities are working with students to “use AI to think with you, not for you”. The concept goes to the heart of these leaders’ efforts to move the emphasis from detection of AI misuse to recognising the place AI will have in both education and the future working lives of today’s students.

Sharing solutions

There was a clear consensus among the leaders that there isn’t a silver bullet that will resolve the spectre of pieces being written or part-written by generative AI—there is no backend checker that can prove beyond doubt that a student has or has not misused AI in completing an assessment.

However, as the discussion showed, with care and thought backed by institutional policy and strategy, there are actions that leaders can take: looking at assessment design and how it aligns constructively with learning outcomes; supporting students and scaffolding them through assessment; delivering encouraging, timely and helpful feedback; and ensuring that assessment is iterative and inclusive, and integrates with learning activities and resources.

University of Greenwich: checking assessment vulnerability

For example, the University of Greenwich is considering all its assessments from the bottom up to ensure they are relevant and authentically reflect what graduates might encounter in the workplace. As a professional technical university with a large widening participation intake, Greenwich is dedicated to developing skills in students that will be of use in the workplace—and the ability to use generative AI effectively and ethically is certainly a skill employers will increasingly value. 

The University has developed a simple but effective tool to consider the vulnerability of its assessments to generative AI being misused by students. Using the AI Risk Measure Scale (ARMS), programme leaders assign a numerical risk to all the assessments in their courses and highlight those which are very high risk so they may then modify them. 

At the same time, Greenwich is also aware that such a high risk/low risk framework can be a blunt tool. Although in-person exams are generally very low risk in terms of academic misconduct, moving all assessments to in-person exams would be a retrograde step strategically for Greenwich when it is keen to move towards more authentic assessments and a policy of "exam by exception". 

Imperial College London: AI stress testing assessments

At Imperial College London, a similarly close examination of the purpose and suitability of assessments is taking place. An AI stress test is being used to drive educator AI literacy and explore the robustness and resilience of existing assessments, as well as extend innovation in assessment design and foster a culture of sharing and collaboration within the college.

Taking a team approach, individual members of staff worked with educational experts from Imperial’s educational development unit and Ed Tech Lab to stress test their assessments, learning basic AI prompt engineering to explore how robust the assessments really are.

All the cases where an assessment has changed substantially have been collected into an Anatomy of Assessment resource, to share learnings and as part of a policy to try to reduce the assessment burden on students. Where an assessment might be susceptible to GenAI, course leaders are asked if it needs to take place at all, and the extent to which what is being assessed aligns with what is being taught. 

University of Glasgow: building educator AI literacy

Educator AI literacy is also high up the agenda at the University of Glasgow. The Practice Enhancement Tool is a 10-minute survey for any staff member involved in teaching and assessment, which asks questions associated with meaningful, iterative, programmatic and inclusive assessment, and prompts faculty to reflect on their own practice.

The results produce a dashboard for the individual staff member to see how their own practice relates to Glasgow’s learning through assessment framework, and then steers them towards resources and people who can help. It also provides leaders, such as Professor Moira Fischbacher-Smith, Vice-Principal (Learning and Teaching), with an overview of the situation in the whole university.

Queen’s University Belfast: piloting AI-augmented feedback

Taking a slightly different tack, Queen’s University Belfast is exploring whether an AI-augmented approach could enhance feedback to students by making it more encouraging and engaging. Co-designed and co-delivered with 100 psychology students at the university—who will be working on the project over the long term—the study is trialling whether "human in the loop" AI-augmented feedback can ease staff workload and improve student experience and success. The results have surprised staff.

The success metric

While there is no silver bullet for assessment assurance, there are certainly tech enablers to support universities and students. For example, Herk Kailis, Cadmus CEO, explained how its timeline tool is being used.

Within the Cadmus authentic assessment online platform, end-to-end reconstruction reports can replay the construction of an assessment from start to finish, crucially looking at the process rather than the end product. For educators, the platform provides access to real-time analytics, which monitors the process around students’ development of their assessment rather than trying to catch academic misconduct at the point of submission. Educators can clearly identify how many hours a student has spent writing their assessment in Cadmus, if the work was copied and pasted or transcribed into Cadmus, and if the pasted work appears to be inauthentically created.

However, while the timeline tool can be used to show that a large chunk of text has (or has not) simply been pasted into the interface from ChatGPT, Herk Kailis revealed that when it was released to academics they used it in class with students as an exemplar rather than a punitive tool. 

By highlighting to students the process they need to go through for an assessment, from how long it should take and how to get started to building references and working through plans and checklists, it showed students how they could break down the process, giving them confidence, scaffolding them through the various steps and, ultimately, lowering barriers to student engagement with the assessment.

This summary draws from an article by Bess Brennan and Mary Curnock Cook, published on HEPI.