This repository was archived by the owner on Jan 8, 2020. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 35
/
Copy pathREADME
46 lines (38 loc) · 2.52 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
Data relationships play a vital role in various data analytics, structured machine
learning, and data mining applications. These relationships can be expressed as
graphs. GraphBuilder is a Java library for preparing graphs from large volumes of
unstructured and semi-structured data.
Processing large graphs in a distributed computing environment presents multiple
system-level challenges, from parallelizing the graph construction algorithms to
achieving balanced system utilization. In addition, the resultant graph must be
carefully partitioned to ensure that subsequent application processing is properly
balanced. GraphBuilder addresses these challenges with a carefully-designed
multi-stage pipeline that leverages the Apache Hadoop framework to achieve
scalability.
How to build the GraphBuilder?
------------------------------
The GraphBuilder is using Maven for building the package, ensure maven is
installed on the system. To build package type:
$> mvn package
How to use the GraphBuilder?
----------------------------
The GraphBuilder library can be used in multiple ways. Library groups multiple
MapReduce jobs in four major groups: a) GraphCreation b) ComputeNetworkInfo
c) GraphNormalization d) GraphPartitioning. These group can be chained to form
graph construction pipeline depending on an application requirements (e.g. Please
refer construction pipeline for link and doc-word graph in demoapps dir).
Library can be used as tool to construct link and doc-word graph for Wikipedia
using following command:
* Link graph construction - $> hadoop jar target/graphbuilder-0.0.1-SNAPSHOT-hadoop-job.jar com.intel.hadoop.graphbuilder.demoapps.wikipedia.linkgraph.LinkGraphEnd2End <# of partition> <hdfs input directory> <hdfs output directory>
* Doc-word graph construction - $> hadoop jar target/graphbuilder-0.0.1-SNAPSHOT-hadoop-job.jar com.intel.hadoop.graphbuilder.demoapps.wikipedia.docwordgraph.TFIDFGraphEnd2End <# of partition> <hdfs input directory> <hdfs output directory> <optional stop word filter list>
Sample application code
-----------------------
Early stage in graph building pipeline is data dependent, hence we abstracted
out the data dependent code from the library and provided interface to plug-in
data-specific code. Sample code for those interface can be found in demoapps
directory. It contains code for generating link-graphs and doc-word
graph out of Wikipedia XML dump. Interfaces are documented, please refer
a Javadoc in the doc directory.
Contact
-------
More information visit www.01.org/graphbuilder