Why AI Detection Doesn’t Work: Insights from a Classroom Experiment

Article 1 

Why AI Detection Doesn’t Work: Insights from a Classroom Experiment

As educators, we’ve been grappling with the impact of generative AI tools like ChatGPT in our classrooms. Many of us have turned to AI detection software in an attempt to maintain academic integrity, some even encouraged by our institutions or exam boards.

This is bemusing, as the evidence is quite clear: At best, AI detection software doesn’t work, and at worst, it persecutes the innocent, especially those with English as a second language.

The Experiment

While many research papers have examined the efficacy of AI detectors with undergraduate students, little has been studied at the school level. Existing studies prove these detectors are ineffective, but I believed they would be even less effective in schools where children write less, giving detectors less to work with. Moreover, most studies retrospectively test prior written work, ignoring the fact that in 2024, children are aware of detection methods and actively trying to beat them. A quick check of social media reveals children are bombarded with videos explaining how to outsmart these detectors.

I decided to put these AI detectors to the test myself. I conducted a simple experiment involving about 100 students aged 12 to 17. The task was straightforward: complete their homework using only generative AI tools like ChatGPT, with the challenge of avoiding detection by a paid AI detector. (The task is included below for anyone wishing to test this with their own students)

I ran all of their AI-written homework through one of the leading AI detection sites. The results were not just eye-opening; they were alarming:

When children set themselves the specific aim of avoiding detection:

• 50% of the 12-year-olds completely bypassed detection
• 38% of the 13-year-olds completely bypassed detection
• An astonishing 100% of the 17-year-olds completely bypassed detection

Let that sink in. Half of the 12-year-olds, over a third of 13-year-olds, and every single 17-year-old in the study managed to fool the AI detector. The detector reported clear confidence that their work had been written entirely by humans, unaided by AI.

What’s more, 62% of students who successfully evaded detection reported either first-time use or very limited experience with generative AI. This wasn’t a feat achieved by tech-savvy students with extensive AI knowledge – it was accessible to novices.

These results underscore a crucial point: AI detectors are fundamentally unreliable, especially when students are actively trying to bypass them.

Time and Effort

You might think beating these detectors would require significant time and effort. However, 44% of students completed their AI-generated homework and successfully bypassed detection in just 5-10 minutes, demonstrating that AI detectors pose little challenge to determined students.

The Accusation Dilemma

Some educators argue these detectors are still useful as deterrents or conversation starters about AI use. However, this creates a troubling dynamic where we accuse students without concrete proof, and they have no way to definitively disprove the accusation. This approach erodes trust and ultimately as the detectors results have absolutely no legal standing social media is already advising students to simply deny, deny, deny.

Staying Ahead of the Curve

As educators seek more innovative ways to detect AI use – such as tracking Google Docs history or using “Trojan horse” techniques – students are quickly adapting. Again social media is rife with ways to circumvent these methods, and students are and will always remain one step ahead in this technological arms race.

Conclusion

The AI revolution in education is here to stay. The current situation is best summed up by a leading AI detection website. Despite proudly claiming they “Empower students to do their best original work” on their homepage, hidden away on a subpage comes the reality:
“Zero Cheating is a Pipedream, but we still need to push academic integrity.”

However, this reality poses significant problems for teachers. It’s an admirable sentiment we’d all like to support, but when exam boards formally place the responsibility for upholding academic integrity on teachers, I believe it’s time for schools to push back. Coursework, Internal Assessments, and Extended Essays have always been vulnerable to abuse, and AI only exacerbates this. Exam boards know that true integrity comes at a high cost — secure exams, invigilators, spot checks — yet they still ask teachers to tick an ‘authentication box’ for work completed outside controlled conditions. Forgive me for being more than a little cynical.