HARP

@Copyright 2013-2017 Inidana University Apache License 2.0

@Author: Bingjing Zhang

WHAT IS HARP?

Harp is a framework for machine learning applications.

FEATURES

1. A Hadoop plugin. It currently supports hadoop 2.6.0 ~ 2.7.3 version.
2. Hierarchical data abstraction (arrays/objects, partitions/tables)
3. Pool based memory management
4. Collective + event-driven programming model (distributed computing)
5. Dynamic Scheduler + Static Scheduler (multi-threading)

COMPILATION & INSTALLATION

####1. Install Maven by following the maven official instruction ####2. Enter "harp" home directory ####3. Install third party jar file. This javaml jar is required by randomforest application. It's not required by harp project itself. mvn install:install-file -Dfile=third_party/javaml-0.1.7.jar -DgroupId=net.sf -DartifactId=javaml -Dversion=0.1.7 -Dpackaging=jar ####4. Compile harp mvn clean package

####5. Install harp plugin to hadoop cp harp-project/target/harp-project-1.0-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/mapreduce/ cp third_party/fastutil-7.0.13.jar $HADOOP_HOME/share/hadoop/mapreduce/

####6. Configure Hadoop environment for settings required to run Hadoop

####7. Edit mapred-site.xml in $HADOOP_HOME/etc/hadoop, add java opts settings for map-collective tasks. For example:

 <property>
   <name>mapreduce.map.collective.memory.mb</name>
   <value>512</value>
 </property>
 <property>
   <name>mapreduce.map.collective.java.opts</name>
   <value>-Xmx256m -Xms256m</value>
 </property>

####8. To develop Harp applications, remember to add the following property in job configuration: jobConf.set("mapreduce.framework.name", "map-collective");

EXAMPLE

####1. copy harp examples to $HADOOP_HOME cp harp-app/target/harp-app-1.0-SNAPSHOT.jar $HADOOP_HOME

####2. Start Hadoop cd $HADOOP_HOME sbin/start-dfs.sh sbin/start-yarn.sh

####3. Run Kmeans Map-collective job

The usage is

hadoop jar harp-app-1.0-SNAPSHOT.jar edu.iu.kmeans.regroupallgather.KMeansLauncher <num of points> <num of centroids> <vector size> <num of point files per worker> <number of map tasks> <num threads> <number of iteration> <work dir> <local points dir>

For example:

hadoop jar harp-app-1.0-SNAPSHOT.jar edu.iu.kmeans.regroupallgather.KMeansLauncher 1000 10 100 5 2 2 10 /kmeans /tmp/kmeans