Please use this identifier to cite or link to this item: http://hdl.handle.net/20.500.12188/17065
Title: Feature Ranking Based on Information Gain for Large Classification Problems with MapReduce
Authors: Zdravevski, Eftim 
Lameski, Petre 
Kulakov, Andrea 
Jakimovski, Boro 
Filiposka, Sonja 
Trajanov, Dimitar 
Issue Date: Aug-2015
Publisher: IEEE
Conference: 2015 IEEE Trustcom/BigDataSE/ISPA
Abstract: In classification problems the large number of features can pose a significant challenge from many aspects. This is particularly the case in the context of Big Data. In order to address this issue we propose a distributed and parallel computation of information gain based on MapReduce. The proposed implementation on Hadoop can be used for ranking features of large datasets and furthermore for feature selection. The data-parallelism is achieved by uniformly distributing it using HBase tables with proper row keys. Performance evaluations are made by estimation of the speed-up of multi-node clusters against a one-node cluster. The framework was deployed on a on-premises Hadoop cluster. The results show that by parallelization and distribution of the computations on a cluster significant speedup can be achieved. The main contribution of this paper is that we have demonstrated how the higher level scripting language Pig Latin can be used for writing MapReduce jobs instead of directly writing a separate map and reduce function. Additionally, we have proposed the use of manually pre-splitted HBase tables instead of HDFS files for data fragmentation in order to set the degree of parallelism on a higher level.
URI: http://hdl.handle.net/20.500.12188/17065
DOI: 10.1109/trustcom.2015.580
Appears in Collections:Faculty of Computer Science and Engineering: Conference papers

Files in This Item:
File Description SizeFormat 
2015_InfoGain_EftimZdravevski.pdf650.44 kBAdobe PDFView/Open
Show full item record

Page view(s)

12
checked on Jun 23, 2022

Download(s)

10
checked on Jun 23, 2022

Google ScholarTM

Check

Altmetric


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.