Research Paper: A Solution for Automated Grading of QDA Homework
Abstract: Teaching research methods is of vital importance in any curriculum that prepares students for an academic career. While the theoretical frameworks underpinning qualitative theory building research can be adequately conveyed through lecturing and text books, experiential learning plays a particularly important role in understanding the challenges and best practices for the analysis methods for qualitative data such as interview transcripts. However, using experiential learning techniques for teaching qualitative data analysis (QDA) methods to large numbers of students presents a challenge to the instructor due to the effort required for the grading of homework that scales with the number of class participants. Any homework involving the coding of qualitative data using computer assisted qualitative data analysis software (CAQDAS) will result in a myriad of different interpretations of the same data with varying degrees of quality and veracity. Hence, grading such a homework assignment requires significant effort per participating student which prevents such courses from being available to a large number of students, or leads to the exclusion of such practical exercises. Within our own course on research methods we solved this problem by using methods of inter-rater agreement and a model solution as a proxy for quality of the submission. Within this paper we demonstrate that this proxy has high correlation with manual grading of submissions, as evidenced by a data set of 1254 coded interviews in 627 homework submissions.
Keywords: Teaching, QDA, qualitative research methods, homework evaluation
Reference: Andreas Kaufmann, Dirk Riehle, Julia Krause and Nickolay Harutyunyan. 2023. A Solution for Automated Grading of QDA Homework. In Proceedings of the 56th Hawaii International Conference on System Sciences (HICSS 2023), forthcoming. Maui, USA.
The paper is available as a PDF.