Stikkord: Claude

Unmasking the Limits of Chatbots: A Two-Year Study on the Inadequacy of AI Tools in solving Higher-Order Thinking Assignments within the field of Social Informatics

Per Arne Godejord
Nord University Business School
Norway

Abstract

This paper critically evaluates the capabilities of fourteen popular chatbots evaluated over a two-year period (December 2022 to January 2025) to address higher-order academic assignments within the field of Social Informatics, specifically focusing on assignments that require complex cognitive skills in alignment with Bloom’s taxonomy.

The chatbots examined include ChatGPT, GPT UiO, Sikt KI-Chat, GPT-3 Playground, Chatsonic, Bing Chat (Copilot), Jenni, Claude, llama70b-v2-chat, Perplexity.ai, Gemini Pro, and others, using primarily their free versions. The tools were tasked with producing fact-based essays, academic responses to course assignments in various sub-fields (e.g., computer security, law, game creation, and work in virtual teams), and addressing complex academic inquiries.

Results indicated that none of the chatbots were capable of reliably producing high-quality academic outputs beyond simple fact repetition. A substantial number of responses involved fabricated information, including non-existent sources.

These findings challenge media claims that chatbots like ChatGPT can effectively meet the demands of higher education assessments and thereby making portfolio assessment in online courses impossible as examination method.

Furthermore, this paper strongly suggests that concerns about chatbots undermining academic integrity in Norwegian bachelor and master’s theses are unfounded. In conclusion, current AI tools are far from being true artificial intelligence, and they fall short in delivering the level of academic rigor required by advanced education within the Social Sciences.

This excerpt from an upcoming journal paper is derived from two years of testing involving 14 chatbots. It also incorporates insights from my blog book, “ChatGPT – A Talkative Example of Artificial Intelligence, or…?”.

The comprehensive findings and analyses will be fully detailed in the complete paper, scheduled for publication in 2025.

ChatGPT – et taleført eksempel på kunstig intelligens, eller…?

Per A. Godejord


Fullstendig bloggbok: 24. utgave, 11.02. 2025
Deler av innholdet første gang publisert som ett enkeltstående innlegg: 05.01. 2023

«Mundus vult decipi»
– Sebastian Brant, 1494 (1)

«Den regner kun i sandsynligheder. Når du spørger den om noget, går den igennem sit datasæt og svarer: «Er det sandsynligt, at dette er sandt, ud fra de mønstre, jeg kan finde her?» Det er det eneste, den kan.»
– Nikolaj Nottelmann, PhD. 2023 (2)

«People believe it really can solve a lot of things that it can’t. I mean, the unfortunate thing is that OpenAI was amazing. But it was useless. It can’t really do anything.»
– Bern Elliot, 2024 (3)

«The problem that crops up over and over again in this field is that people anthropomorphize these computers, ascribing all sorts of desires, plans, emotions, and the like. They are machines, running code. They don’t need anything, love or hate anyone, have goals or desires. They are not conscious. They don’t care if they’re replaced by a newer version. They don’t act covertly. They are machines, running code.»
– wsf, 2024 (4)

«If you know the answer, you don’t need to ask an LLM; if you don’t know the answer, you can’t trust an LLM.»
– Professor Gary N. Smith, 2025 (5)

«In LRMs, the term “reasoning” seems to be equated with generating plausible-sounding natural-language steps to solving a problem, and the extent to which this provides general and interpretable problem-solving abilities is still an open question.»
– Professor Melanie Mitchell, 2025 (6)

Prolog

1. kapittel Innledende betraktninger
2. kapittel Whoops apocalypse
3. kapittel Bullshit Generator eller genial faktaggjenngiver?
4. kapittel En tjenende ånd eller en falsk lurendreier?
5. kapittel Rasende utvikling – fra dum til dummere?
6. kapittel Når oppgavene blir lange…
7. kapittel Hva gjør vi nu, lille du?
8. kapittel En sverm av djevler klokken tre
9. Kapittel Det handler om undervisning og oppgavedesign
10. Kapittel Summa Summarum

Epilog

Vedlegg: Samleside for alle testene

Annet:

  1. Kort konferanseartikkel basert på den første uttestingen: «Language Models: Viable Strategies for Portfolio Assessment»
  2. Work in progress: «Unmasking the Limits of Chatbots: A Two-Year Study on the Inadequacy of AI Tools in solving Higher-Order Thinking Assignments within the field of Social Informatics»

Denne «bloggboken» er i hovedsak utviklet for arbeidskrav 1 i fagemnet IKT1016 – Digital dannelse, studieprogram IKT og læring 1.

Kildehenvisning i tekst: (Godejord, P.A., 2025).
I litteraturlisten: Godejord, Per A. (2025). «Kapittelnavn», i «ChatGPT – et taleført eksempel på kunstig intelligens, eller …?», Nord universitets bloggnettverk, [Online]. Hentet fra: Full lenkeadresse til kapittel.

[NB! Som del av arbeidet med AK1 i IKT1016 er det vesentlig at alle henvisninger er knyttet til konkrete kapitler i bloggboken NB!]