Skip to content

Latest commit

 

History

History
38 lines (33 loc) · 1.2 KB

README.md

File metadata and controls

38 lines (33 loc) · 1.2 KB
Raw

anamnesis

OVERVIEW

An in-memory, location aware, HDFS based file system.

BUILD

EACH COMPONENT (CLIENT, NAMENODE, DATANODE)

gradle build

EXECUTE

NAMENODE

./namenode.sh src/main/resources/config.properties

DATANODE

./datanode.sh src/main/resources/config1.properties
./datanode.sh src/main/resources/config2.properties
./datanode.sh src/main/resources/config3.properties
./datanode.sh src/main/resources/config4.properties

CLIENT (currently not supported)

./anamnesis.sh mkdir foo/bar
./anamnesis.sh ls foo
./anamnesis.sh upload /tmp/test.txt foo/test.txt -b 2 -f 0
./anamnesis.sh download foo/test.txt /tmp/test2.txt

HDFS

./bin/hdfs dfs -mkdir /user/hamersaw
./bin/hdfs dfs -D dfs.block.size=1024 -copyFromLocal /tmp/test.txt /user/hamersaw/test.txt
./bin/hdfs dfs -copyToLocal /user/hamersaw/test.txt /tmp/test2.txt

SPARK

$bin/spark-shell
scala> val rdd = sc.textFile("hdfs://localhost/user/hamersaw/MOCK_DATA.csv")

TODO

  • change DatanodeService to add Datanode not it's elements
  • setup some rock solid logging

RPC SERVER

  • rpc response error handling (fix up)

CLIENT

  • fix everything (works with hdfs native client)