Per Arne Godejord
Nord University Business School
Norway
Abstract
This paper critically evaluates the capabilities of fourteen popular chatbots evaluated over a two-year period (December 2022 to January 2025) to address higher-order academic assignments within the field of Social Informatics, specifically focusing on assignments that require complex cognitive skills in alignment with Bloom’s taxonomy.
The chatbots examined include ChatGPT, GPT UiO, Sikt KI-Chat, GPT-3 Playground, Chatsonic, Bing Chat (Copilot), Jenni, Claude, llama70b-v2-chat, Perplexity.ai, Gemini Pro, and others, using primarily their free versions. The tools were tasked with producing fact-based essays, academic responses to course assignments in various sub-fields (e.g., computer security, law, game creation, and work in virtual teams), and addressing complex academic inquiries.
Results indicated that none of the chatbots were capable of reliably producing high-quality academic outputs beyond simple fact repetition. A substantial number of responses involved fabricated information, including non-existent sources.
These findings challenge media claims that chatbots like ChatGPT can effectively meet the demands of higher education assessments and thereby making portfolio assessment in online courses impossible as examination method.
Furthermore, this paper strongly suggests that concerns about chatbots undermining academic integrity in Norwegian bachelor and master’s theses are unfounded. In conclusion, current AI tools are far from being true artificial intelligence, and they fall short in delivering the level of academic rigor required by advanced education within the Social Sciences.
This excerpt from an upcoming journal paper is derived from two years of testing involving 14 chatbots. It also incorporates insights from my blog book, “ChatGPT – A Talkative Example of Artificial Intelligence, or…?”.
The comprehensive findings and analyses will be fully detailed in the complete paper, scheduled for publication in 2025.
