A New Web Crawler for Gathering Source Code Solutions for Data Mining Purposes

Abstract

Teaching programming in the modern educational environment imposes the need for fast assessment of large sets of programming solutions. The same applies to programming competitions, especially the international ones. Although the problem is typically solved by use of computer aided or automated assessment (CAA) systems that perform grading based on test cases, this dominant approach for source code assessment has its own serious drawbacks. In this paper we present our research in this field, with focus on one important preparatory step before conducting our experiments. We review our proposed model for semiautomatic student source code assessment, and we explain the need for gathering a variety of source code sets. Then, we discuss the problem we have encountered when trying to collect source code sets from a very famous web-based CAA system. Finally, we describe the creation process for a web crawler – as a proposed solution to this particular problem.