Reproducibility of Published Educational Recommendation Systems: Systematic Review

Abstract

The purpose of this paper is to conduct a systematic review of the available literature on explainable recommendation systems in education and their reproducibility, particularly when recommendations are integrated as part of learning management systems. The first part of the paper’s methodology employs an NLP-powered toolkit that automates a large portion of the review process by automatically analyzing articles indexed in the IEEE Xplore, PubMed, Springer, Elsevier, and MDPI digital libraries. A quantitative review of all available literature is carried, followed by a qualitative review of the few selected articles that do indeed focus on the explainability approach when implementing recommendation systems. The relevant articles are thoroughly analyzed and compared based on a variety of indicators such as the purpose of the recommendations, tools and techniques used, and whether the research is easy or hard to reproduce. The findings show that, while the amount of available research is increasing and new learning management systems are continuously being developed in recent years, the explainability of the machine learning techniques used in recommendation systems isn’t a primary focus among researchers and developers, and the scope of the available literature is quite limited.