Browse Comments — Clean (de-noised)

I don't have a large sample to draw on, but I'd say the dissertations I've seen this year have been a pretty poor bunch.

my department introduced oral thesis examinations. i do not think that solves the issue (entirely), but let's see how it will unfold. we are given only 15 minutes for examination (total, not per examiner). in macedonia, where I come from, the theses were always orally defended, but what helped not to block the examiners' work for a month, was that the students are given an entire year (the year after their studies) to hand in their thesis. once they hand in, the oral examination is scheduled, so it happened that there is only one or two scheduled examinations per month, since not everyone was handing in the theses at the same time. my bachelor thesis examination was an hour, the masters-two hours. before handing in, your supervisor needs to approve your submission.

Ana Pop Stefanija yes that was the Italian system when I graduated eons ago. I am very curious to know how the 15 minutes orals go. In the UK there is a very strong stigma against orals due to EDI concerns.

Paolo serious games and simulations could finally (rightly) be making their way up the food chain!

At the Uni Potsdam where I studied in Germany, 20 min of defense + 30 min of questions are mandatory (Masters). I just found out that every university here has their own way, and that it isn’t the standard, but it seems reasonable. Although 15-20 minutes seem so short to me to find out if someone has done the work, it could be that experience and domain knowledge compound and then it suffices :)

I agree 15 minutes is very short, so it will catch only the most blatant usage and the most unprepared students only. The difficult hallucinations to spot are difficult to spot in any form. If a student is using an obscure reference I have not read, and the hallucination is a missinterpretation or concept stretching of such obscure reference, I need time to study it and understand it to be able to catch the mistake. So a restriction on the domain of methods and references might be one key ingredient to avoid having to do quite a bit of studying of often very useless obscure papers before grading each assessment.

What does it mean "stats output we doubt are real"? I mean either is real or it is fabricated. But AI is by now pretty good at econometrics, certainly way better than writing.

Francesco Nicoli it means that without a replication package with data and code I can run and check I have no way to know if it is completely fabricated. It looks good but it uses stats and models nobody teaches and has all the visuals styleme of a specific AI.

Very interesting way of looking at it.

One thing I have found with marking is that even when students have used generative "AI" and produced work with references to sources that actually exist, the material being cited isn't actually found in those sources. I have required that students use page numbers for all in-text citations, whether they're quoting directly or paraphrasing, and I check that the information is actually found on the relevant page. Often, in essays with that bland writing style, the information cited is not actually there.

From what I’ve seen, academic integrity officers are purely operating for lip service, they are not being sufficiently trained, have limited resources, and aren’t being supported as necessary. Such a shame really.

Yasir Abbas Pollyanna docet

Francesco Nicoli but harder to use AI in a fast paced simulation in which you have to think in real time and the grade is on the performance in the simulation. So the assesment is not to deliver a game/simulation, is the performance in the game. In a sense orals are a game of this kind. It has rules, victory/losing conditions, score, entry and exit points.

Paolo Spada yes sure. I was thinking more in terms of research. I recently used codex to build a custom simulation software that implements a formal model of European integration I had developed over the past 10 years or so. Without codex I probably would have never finished it, nor build an entire dedicated simulation software.

Dr. Dean Fido CPsychol AI review is a large amount of work, often outside personal expertise at least in politics that is heterogenous as a field, often operating from pilot processes (we have a new AI declaration the students sign that is the basis of the review this year) and if it goes rampant, I had 26 cases in a class last year, it is a month of work that is not properly workloaded. The advantage of our new AI declaration is that it allows for class level customisation. So in one of my classes I piloted a -30 points flat punishmrnt for hallucination they can challenge with an office hour discussion with me that would determine if it was a honest mistake instead. Got a few, nobody challenged them. I did not involve AI officers. But agentic AI can mechanically cross-check references, particularly if you load all the sources, so this way of catching AI misuse is disappearing. An agent writes the essay or corrects it adfing references, one downloads the pdfs, one does the cross-check, ranks probability of hallucinations, and then the student does a manual cross-check. Maybe a couple of hours of work, perfect referencing, zero reading and learning (apart for a bit of AI agent setup following youtube videos).

Paolo Spada 'Given we do not ask for replication packages we have very reasonable looking stats output that we doubt are real. ' nailed it! Nice read and thanks!

Since the floor has been raised, maybe it makes sense to start asking for replication packages from undergraduate dissertations.

what about requring students to implement semi-automated transparency workflows? I have buid/experimented with one myself: but this is a 8 month scientific paper project where I was using an imperfect system (the in-development one) to fully document the very design thinking behind it. For a student use case the result will be much neater and clearer.

Thomas Carruthers I do that in my intro to RCT class, BUT I have multiple domains restrictions: 1) they need to use only the commands I teach in R, 2) they need to use only the statistical models I teach, 3) they can only do 3 experiments types, 4) they need to explain the code line by line. They still use AI but I delude myself that I force them to be more deliberate about it. It is also a lot of extra work to grade it, even with the restricted domain.

Do you think a series of conversations with the students who submitted the dissertations would help you assess them more accurately?