Question classification according to Bloom's Revised Taxonomy

Abstract

FeedbackFruits is a company that provides tools for educators to organize their courses. The company is currently working on aiding teachers in aligning course material and assessment. Aligning the two provides students with clear expectations and can lead to an increase in learning [1]. Aligning course material and assessment is usually done by comparing what the students are taught to how students are assessed. When a student is assessed by an exam consisting of questions, the alignment process involves classifying these questions according to the cognitive process categories needed to answer them[2].
This process can be time consuming if an exam contains many questions it, and it can be easy to lose oversight of whether the questions in the assessment are representative of what is taught in the course material. The task of classifying questions into categories that represent the cognitive processes needed to answer them can be facilitated by providing a classification tool. This tool also gives educators insight by displaying a summary of the different question categories present in a set of questions. As part of the solution to the problem of course alignment, FeedbackFruits requested the development of a question
classifier which classifies questions according to the cognitive process required to answer them. Bloom’s revised taxonomy (subsection 2.1.1) is a taxonomy that categorizes questions and learning objectives into six distinct classes in the cognitive process domain. We propose a software solution that uses machine learning techniques to classify a courses’ questions and provides a clear overview of the classes in Bloom’s revised taxonomy present in these courses. To achieve this, we built a training set and test set by combining a preexisting labeled dataset from Anwar Ali Yahya, Addin Osama, et al. [5] and a self labeled
dataset of over 1500 samples. We engineered a set of features specific to short text samples and questions. We adopted an experimental approach in selecting the classifier model: we tested several different models throughout the project and picked the best performing models as final step. When looking up Bloom’s taxonomy it is often presented with lists of class specific keywords. We replicated a study [3] that makes use of keywords that are indicative of the class in Bloom’s taxonomy to set a baseline to compare our model to. We ran our model and the model of the baseline study on the same test set. Our model scored an accuracy of 75% compared to the baseline model which scored an accuracy of 40%.