ChatGPT in exam rooms: preliminary insights into student performance with and without AI assistance

Abstract

Reactions to artificial intelligence, particularly ChatGPT in the field of education, have been varied all across the world. We watched institutions, cities, and even entire countries prevent it in the educational context, and we also experienced the opposite extreme, which was institutions supporting it and integrating AI into their routine operational workflow. However, what is the most effective method for students? Does having access to AI allow them to benefit from it, or does it hinder their ability to perform well in school? Our interest in answering these questions resulted with a study carried out with the participation of 65 college students who were studying computer science and engineering. Structured programming and object-oriented programming were the two types of programming classes that we looked into. Both were administered in the same manner and lasted for the same amount of time; however, whether or not students had access to ChatGPT throughout the examination was different. When taking tests in structured programming, they were permitted to use ChatGPT; but, when taking exams in object-oriented programming, they were specifically prohibited from using it. Within the scope of this study, the preliminary findings are provided. Results show that when students had subsequent exams under the same circumstances and with access to ChatGPT, they improved on the second exam by around 53% and displayed a strong positive correlation between the two subjects. However, restricting AI access on the second exam, led to a 39% decrease in results and more unpredictability in their outcomes.