Assessment of pedagogical and Technical Quality of Generative AI Response in Java OOP

Abstract

In this study, we examine the quality of answers returned by AI language models (GenAI) to Java Object-Oriented Programming (OOP) questions. A 20-question package was developed, including questions on Classes, Objects, Encapsulation, Inheritance Polymorphism and Constructors. Each GA was then rated with respect to the MAs and SAs in terms of (a) how correct it was; (b) how much understanding it revealed; (c) how clear it is; (d) if included code, whether the code quality was acceptable; and also, in terms of potential “hallucination.” The experimental results demonstrate that GenAI generates high-quality answers, especially in the correctness and conceptual depth aspects, but may fail to make a clear response for questions of complex semantics. The research offers a transferable model for AI evaluation in education.