Browse Comments — Clean (de-noised)

Back to basics ( written examinations ) with higher weighing % of assessment than reports or such

Dear Michele Loi can you explain what they are in simple terms?

Paolo Spada two elements: 1) an ontology of human-AI interaction artifacts: two examples: prompt development logs to trace how the human interacted when writing a complex prompt; modification logs to track the human requests to revise draft. Give a LLM a schema, and it will apply it coherently, it's called a "skill", it will document everything. Bonus effect: the original bits introduced by the LLM are also tracked and these are becoming more interesting every month; 2) an AI "harness" for logging: a set of tools (deterministic, but any AI can assist in creating them) that take care of ensuring that all chat sessions are stored appropriately (systematically, searchable) and map them to the interaction artifacts above; slighly more complex to build, automatic mapping of input and output flows across different sessions (necessary when students' context-engineering workflows become sophisticated, which I asssume must be the case for every "obviously AI assisted but pretty good" essay you graded). Feel free to contact me if you want to talk more.

Alternative assessment system will surely emerge soon

I hope so!

I've said before and I will say it again - the gold standard of university assessment will be the in-person timed exam. I did that in the early 00s and I am only now starting to appreciate how rigorous that was.

In Italy, undergraduate assessment is mostly based on oral exams and in-class exams, but there is much less institutional attention to inclusion and student mental health. Paradoxically, less advanced systems may now be more resilient to the AI storm. We have just had a conversation with the AI faculty group, and many colleagues feel that oral and timed exams are not inclusive. I personally think this is exactly why we have special considerations, and that we could find a way to make oral exams and timed exams ubiquitous again while still offering reasonable adjustments. But this is not a small change. It requires rooms, new procedures, and probably more staff. In a sector that has been systematically disinvested in for decades, it is unclear whether the average university, already barely afloat, has the capacity to adjust.

Do you think it will go to exam only degrees?

Lisa Perrett I do not know. Part of me thinks this is epiphenomenal, and similar to what statistics went through when we switched from pen and paper to software. I did university in that period: all my introductory classes were still using paper tables to calculate statistics, while my advanced classes were using a kaleidoscope of software: Gauss, Matlab, Stata, and EViews. So part of me thinks that a mix of bans plus explorations will be stage one of the change (example year 1 and 2 of uni pen and paper and exams and orals, year 3 we teach prompting and AI augmented research techniques similar to stats teaching in 1995-2000), and then slowly a new pedagogy emerges, like in modern stats, at the same time of new software (such as R) that is free and controllable and fit the purpose of free and open education. But part of me thinks it is more fundamental, and more similar to when humankind introduced writing as a technology and the oral transmission of knowledge disappeared. If we are in this second scenario there is an imperative to develop a UK and EU stack of technology as Francesca Bria explains. I don't know, and I still have 30 take home essays to grade from my other class that hopefully are not surreal AI slop...

There can’t be anything more demoralising to a lecturer than marking work you can clearly see is AI-generated but it can’t be conclusively proved by available integrity software. How is it even possible to allocate a fair mark in those circumstances, especially when there is a clear disjoint between the standard of the student’s usual performance in class and the standard of the dissertation submitted. A possible solution is to call for an oral defence of the work.

Gerald Ramsden we have the academic integrity procedure that reviews the work and ask the students to explain certain parts that appear suspicious. But it used to be relatively easy and focused on plagiarism, cheating, buying the dissertation etc. With AI it gets very difficult. Because the smoking gun of a plagiarised piece work can be assessed by anybody, the fact that the student used byzantine statistical code that is correct but is not what I taught and we have the suspucion is generated by AI requires a subject expert and it's way easier to defend from. And the code might actually be more efficient and elegant than the one I taught. The student can say they saw a youtube video that suggested the unusual code and they used it, or they asked a friend that taught them about the coding strategy but did not help them directly. And if the class does not restrict the domain (use only my code, use only the approved texts) it is very hard to discern AI violation from actual extra work. So the risk of false positive is not a minor one

Paolo Spada interesting. I too remember my days of statistical tables in my A levels. I loved my book! It will be interesting to see how it all unfolds.

Not sure how this would evolve, but from a student’s perspective, I think assessments should be more pen-and-paper based. To prepare for the future, students should also be taught how to use AI to optimise their work and output and how to self evaluate when using AI. A combination of both would enable us to remain engaged with learning and be better prepared for the real world with modern tools.

Two years ago I proposed to go luddite in year 1 and 2 of university (pen and paper only, exams, orals), and then in the 3d year introduce classes that try to develop an AI augmented pedagogy. Some of us are starting to use more and more AI so we are actually getting a lot of ideas on how to teach it. But the political economics and climate impact of AI and the privacy issues around its systematic usage in the university is a crucial barrier. I think we need local models, specifically designed for university learning, that are free, and designed to promote critical thinking and less cognitive offloading.

I think AI may help those borderline students achieve a pass or boost students' grades a bit when they are near boundaries. Particularly, those students falling between 48-62%, especially. And even some of those around 68-69%. I do not think AI will do the magic to trasform someone that would have scored 48% into a >70%. But it may suffice to bring them over the edge of a pass mark, at Master level, i.e. 50-53%. Similarly a 68% may be boosted to a 70-72%, unlikely it will go to a 85%. Often students ask AI for help in the structure of their essays. AI will suggest sections with titles; allocation of topics to each sections; bullet points of content to cover for each theme; etc. Thus, if our marksheets are too prescriptive, for example if they require to allocate x-many points to structure, then of course there will be a general boost of final marks. We need to change our marksheets to make them fit for purpose in the AI era! Or switch to in person exams on paper!

Paolo Spada did you see this: I thought this concept of the AI favouring the never-skilling was interesting in the wider context of the use of AI by HE students combined for example with the approach of Sweden that is reverting to learning materials in paper (rather than digital), at least in schools

Paolo Spada an ‘oral defence’ of the submitted work should expose that i.e. a one-on-one discussion between lecturer and student where the student is asked to take the lecturer through the submitted work, explain its content and answer the lecturer’s questions. The outcome thereof can then be used to allocate a fair mark for the submitted work.

Thank you for writing this so honestly, Paolo. The 'raised the floor' line is the one I keep sitting with. The layer I'd add. We have spent two years asking 'did AI write this?' The harder question is the one you are already moving toward. What can this student actually do that they could not do before? When the answer to that has to show up live, in a draft we watch grow, in a defense, in a problem we put in front of them, the detection question quietly retires. Looking forward to what your department settles on.

I find it fascinating how consistently it has moved the needle back for us in terms of use of research. Yes - literature reviews are somewhat better - but students frequently draw on literature that is 30-50 years old, rather than anything more recent. It’s probably good in the sense they will at least have heard of some seminal authors, but it has not resulted in contemporary referencing. I’m also saddened in this transitional phase that I have seen the loss of personal voice in student work. Two years ago work was flawed but personal. Now it is fine, but generic. Marks are smoothed out. Similarity in Turn It In is way down.

We have to find alternative ways to ensure the use of AI is limited to copiloting and not replacing the writing process. An approach, maybe, is to ask for regular review chapter by chapter and incorporating this in the assessment grading. Also a viva is a must and potentially has to carry a higher weighting where students unable to defend what they've written will give them out as potentially using AI irresponsibly. But it will take a lot of honest admission and "thinking out of box" and do away with some academic orthodoxy.