cluster-dt066a

This does only work on different IP addresses, different machines

There's no need to setup ip addresses, The master program finds all active slaves on the same network. The slaves and master has a global udp communication link on port 9999 and then creates a separate udp link on individual ports. These private communication links are ran by different threads from Master node's perspective.

This is taking pretty long time, and cannot handle to big of files. My assumption is that its because it has to send each packet thrue the network, it would be more optimized to connect via ssh and work within the same file directory.

Operation Order

• Import file (Master)
• Split (Master)
• Map (Slave)
• Reverse/Shuffle (Slave)
• Reduce (Master)

LOGS

Every operation the master end is logged into different files, could be turned off.

BUILDING

I used Apache ant to build the jars in console (on the RPi:s), since the rpi had lower compile version than my own computer.

SLAVE BUILD AND RUN

in build.xml make sure that first line is <project name="Slave" default="main" basedir="."> and <property name="main-class" value="src.Main"></property>

java -jar Slave.jar <should log, leave this parameter if you dont want it to log>

MASTER BUILD AND RUN

in build.xml make sure that first line is <project name="Master" default="main" basedir="."> and <property name="main-class" value="main.Main"></property>

to run the project make sure that Data folder is created with your testfiles, also make sure to create a folder LogFiles. THen run it like

java -jar Master.jar <filepath> <lines per split> <message window> <should make file, leave this if you want it to create log files>

TODO

• implement package loss, it is needed for larger line config and window sizes. currently working only for 20 of window sizes and 50 of line sizes

// pi perspective

TESTS

1x Master pc 3x Raspberry Pi 3 model B+

NATIVE

1 Master 3 Slave Raspberry Pi's.

File Size	Total Msgs	Total lines	Total words	Splits	window	Import (OP)	Split (OP)	Map (OP)	Reverse (OP)	Reduce (OP)	Total
11.0MB	40000	100000	1593582	10	1	0,191s	0,64s	31,31466s	38,38147s	0,108s	71,71471s
11.0MB	40000	100000	1593582	10	5	0,189s	0,71s	14,14582s	17,17962s	0,95s	34,34413s
11.0MB	40000	100000	1593582	10	10	0,178s	0,63s	14,14516s	17,17977s	0,89s	34,34197s
11.0MB	20000	100000	1593582	20	1	0,171s	0,92s	15,15882s	24,24570s	0,106s	42,42187s
11.0MB	20000	100000	1593582	20	5	0,185s	0,97s	7,7578s	14,14144s	0,96s	23,23454s
11.0MB	20000	100000	1593582	20	10	0,187s	0,95s	7,7677s	14,14384s	0,123s	24,24071s
11.0MB	13332	100000	1593582	30	1	0,198s	0,128s	11,11753s	21,21063s	0,89s	34,34741s
11.0MB	13332	100000	1593582	30	5	0,176s	0,119s	7,7039s	13,13263s	0,103s	22,22216s
11.0MB	13332	100000	1593582	30	10	0,196s	0,102s	5,5434s	13,13062s	0,87s	20,20365s
11.0MB	10000	100000	1593582	40	1	0,172s	0,107s	9,9413s	17,17904s	0,91s	29,29023s
11.0MB	10000	100000	1593582	40	5	0,165s	0,106s	5,5377s	12,12724s	0,117s	19,19969s
11.0MB	10000	100000	1593582	40	10	0,187s	0,116s	4,4504s	12,12393s	0,85s	18,18636s
11.0MB	8000	100000	1593582	50	1	0,183s	0,107s	8,8124s	16,16720s	0,87s	26,26475s
11.0MB	8000	100000	1593582	50	5	0,187s	0,104s	4,4285s	12,12014s	0,89s	17,17905s
11.0MB	8000	100000	1593582	50	10	0,187s	0,101s	4,4272s	12,12033s	0,89s	17,17895s
29.0MB	100000	250000	3987025	10	1	0,286s	0,85s	68,68838s	95,94934s	0,203s	168,167982s
29.0MB	100000	250000	3987025	10	5	0,266s	0,130s	36,36330s	43,43062s	0,176s	83,83912s
29.0MB	100000	250000	3987025	10	10	0,268s	0,101s	40,40782s	42,42750s	0,194s	88,87923s
29.0MB	50000	250000	3987025	20	1	0,317s	0,134s	37,37819s	59,59683s	0,180s	101,101810s
29.0MB	50000	250000	3987025	20	5	0,277s	0,157s	18,18709s	35,35446s	0,195s	58,58665s
29.0MB	50000	250000	3987025	20	10	0,276s	0,172s	18,18719s	35,35784s	0,227s	59,58984s
29.0MB	33332	250000	3987025	30	1	0,290s	0,148s	28,28106s	49,49863s	0,211s	82,82303s
29.0MB	33332	250000	3987025	30	5	0,271s	0,153s	13,13359s	33,33348s	0,170s	50,50918s
29.0MB	33332	250000	3987025	30	10	0,313s	0,139s	12,12899s	32,32601s	0,196s	49,49794s
29.0MB	25000	250000	3987025	40	1	0,270s	0,191s	23,23836s	46,46116s	0,217s	74,74132s
29.0MB	25000	250000	3987025	40	5	0,276s	0,170s	10,10706s	30,30662s	0,146s	45,45279s
29.0MB	25000	250000	3987025	40	10	0,264s	0,165s	12,12962s	31,31058s	0,163s	47,47832s
29.0MB	20000	250000	3987025	50	1	0,271s	0,183s	20,20282s	45,45725s	0,176s	70,70158s
29.0MB	20000	250000	3987025	50	5	0,280s	0,195s	10,10738s	30,30117s	0,214s	45,45762s
29.0MB	20000	250000	3987025	50	10	0,394s	0,254s	11,11029s	30,30349s	0,210s	46,46005s
59.0MB	200000	500000	7973661	10	1	0,568s	0,133s	128,128197s	180,180769s	0,285s	317,316897s
59.0MB	200000	500000	7973661	10	5	0,461s	0,130s	74,74751s	87,87446s	0,271s	170,169842s
59.0MB	200000	500000	7973661	10	10	0,524s	0,117s	69,69231s	87,86998s	0,627s	166,166216s
59.0MB	100000	500000	7973661	20	1	0,474s	0,211s	75,74968s	120,120262s	0,260s	202,202432s
59.0MB	100000	500000	7973661	20	5	0,475s	0,243s	36,36422s	71,71129s	0,268s	114,114834s
59.0MB	100000	500000	7973661	20	10	0,481s	0,208s	39,39358s	71,70975s	0,241s	117,117771s
59.0MB	66664	500000	7973661	30	1	0,462s	0,251s	56,55975s	102,102543s	0,262s	166,166520s
59.0MB	66664	500000	7973661	30	5	0,473s	0,254s	26,26303s	65,65622s	0,246s	99,99552s
59.0MB	66664	500000	7973661	30	10	0,459s	0,168s	24,24341s	66,66571s	0,239s	98,98043s
59.0MB	50000	500000	7973661	40	1	0,442s	0,210s	45,45788s	91,91677s	0,253s	145,145360s
59.0MB	50000	500000	7973661	40	5	0,472s	0,224s	22,22331s	62,62597s	0,280s	93,93111s
59.0MB	50000	500000	7973661	40	10	0,501s	0,323s	21,21086s	62,62311s	0,283s	91,91876s
59.0MB	40000	500000	7973661	50	1	0,498s	0,383s	41,41158s	86,86418s	0,238s	135,135168s
59.0MB	40000	500000	7973661	50	5	0,455s	0,328s	20,20665s	61,61174s	0,251s	89,89448s
59.0MB	40000	500000	7973661	50	10	0,462s	0,334s	20,20319s	60,60631s	0,252s	88,88869s
119.0MB	400000	1000001	15948061	10	1	0,840s	0,185s	261,261675s	363,363377s	0,351s	643,643286s
119.0MB	400000	1000001	15948061	10	5	0,765s	0,172s	150,149987s	169,169294s	0,356s	338,338408s
119.0MB	400000	1000001	15948061	10	10	0,797s	0,245s	149,149324s	173,173575s	0,355s	342,341811s
119.0MB	200000	1000001	15948061	20	1	0,859s	0,327s	157,156882s	246,246271s	0,354s	420,420289s
119.0MB	200000	1000001	15948061	20	5	0,825s	0,348s	84,84697s	143,143582s	0,427s	247,247655s
119.0MB	200000	1000001	15948061	20	10	0,950s	0,345s	85,85083s	147,146959s	0,557s	253,252928s
119.0MB	133332	1000001	15948061	30	1	1,1030s	0,492s	125,125261s	214,213875s	0,446s	358,357830s
119.0MB	133332	1000001	15948061	30	5	1,1396s	0,672s	58,58790s	134,133996s	0,490s	209,209237s
119.0MB	133332	1000001	15948061	30	10	0,910s	0,461s	51,51813s	130,130258s	0,369s	198,198256s
119.0MB	100000	1000001	15948061	40	1	0,928s	0,636s	94,94499s	177,177720s	0,346s	289,288756s
119.0MB	100000	1000001	15948061	40	5	0,804s	0,457s	43,43666s	126,126752s	0,346s	185,185210s
119.0MB	100000	1000001	15948061	40	10	0,819s	0,500s	42,42515s	122,122177s	0,287s	178,177880s
119.0MB	80000	1000001	15948061	50	1	0,718s	0,447s	81,81465s	171,171774s	0,311s	267,267315s
119.0MB	80000	1000001	15948061	50	5	0,727s	0,407s	40,40148s	119,119697s	0,271s	173,173773s
119.0MB	80000	1000001	15948061	50	10	0,760s	0,417s	40,40574s	120,119880s	0,344s	174,174754s

HYPERVISOR

1 Master 3 Slave Raspberry Pi's.

File Size	Total Msgs	Total lines	Total words	Splits	window	Import (OP)	Split (OP)	Map (OP)	Reverse (OP)	Reduce (OP)	Total
11.0MB	40000	100000	1593582	10	1	0,215962s	0,723641s	35,407162s	43,39753s	0,122114s	81,087079s
11.0MB	40000	100000	1593582	10	5	0,210073s	0,789163s	15,723033s	19,095092s	1,055922s	38,17339s
11.0MB	40000	100000	1593582	10	10	0,195116s	0,690579s	15,505323s	18,831733s	0,97558s	37,644206s
11.0MB	20000	100000	1593582	20	1	0,182003s	0,979199s	16,134242s	25,805834s	0,112821s	45,151584s
11.0MB	20000	100000	1593582	20	5	0,195836s	1,026813s	8,212178s	14,969711s	1,016228s	24,595398s
11.0MB	20000	100000	1593582	20	10	0,213578s	1,085024s	8,871726s	16,154109s	0,140482s	27,686051s
11.0MB	13332	100000	1593582	30	1	0,220895s	0,142801s	12,403057s	23,663228s	0,992911s	38,319022s
11.0MB	13332	100000	1593582	30	5	0,192596s	0,130221s	8,430361s	14,371008s	0,112713s	24,31766s
11.0MB	13332	100000	1593582	30	10	0,208203s	0,10835s	5,888524s	13,948114s	0,924165s	21,461501s
11.0MB	10000	100000	1593582	40	1	0,181212s	0,112731s	10,47375s	18,099139s	0,958739s	30,858997s
11.0MB	10000	100000	1593582	40	5	0,18955s	0,121772s	6,36165s	13,931643s	0,134408s	22,056397s
11.0MB	10000	100000	1593582	40	10	0,208622s	0,129413s	4,964981s	13,52577s	0,948282s	20,289174s
11.0MB	8000	100000	1593582	50	1	0,199821s	0,116835s	9,622438s	17,653294s	0,949971s	28,679013s
11.0MB	8000	100000	1593582	50	5	0,198495s	0,110393s	4,70073s	12,865192s	0,94471s	18,235084s
11.0MB	8000	100000	1593582	50	10	0,1979s	0,106887s	4,685254s	12,826804s	0,941877s	18,180283s
29.0MB	100000	250000	3987025	10	1	0,326602s	0,97067s	78,439726s	109,570788s	0,231819s	192,04195s
29.0MB	100000	250000	3987025	10	5	0,295607s	0,14447s	40,410742s	48,264694s	0,19559s	93,17089s
29.0MB	100000	250000	3987025	10	10	0,29417s	0,110863s	44,353668s	46,570571s	0,212944s	97,558341s
29.0MB	50000	250000	3987025	20	1	0,338154s	0,142942s	39,872483s	63,573801s	0,192012s	107,84846s
29.0MB	50000	250000	3987025	20	5	0,291339s	0,165127s	19,128538s	37,184571s	0,205094s	61,619366s
29.0MB	50000	250000	3987025	20	10	0,316336s	0,197137s	20,845182s	40,52526s	0,260175s	68,298679s
29.0MB	33332	250000	3987025	30	1	0,321961s	0,164311s	31,397894s	54,953836s	0,234254s	91,950892s
29.0MB	33332	250000	3987025	30	5	0,297459s	0,167938s	14,415894s	36,58801s	0,186598s	55,440668s
29.0MB	33332	250000	3987025	30	10	0,331942s	0,147412s	12,863011s	34,282312s	0,207862s	52,493451s
29.0MB	25000	250000	3987025	40	1	0,285224s	0,20177s	24,54869s	49,080943s	0,229236s	78,955722s
29.0MB	25000	250000	3987025	40	5	0,312304s	0,192361s	11,436506s	34,293042s	0,165204s	51,431484s
29.0MB	25000	250000	3987025	40	10	0,294668s	0,184167s	13,538664s	34,947791s	0,181935s	52,993665s
29.0MB	20000	250000	3987025	50	1	0,295881s	0,199801s	22,057671s	49,630747s	0,192159s	77,192796s
29.0MB	20000	250000	3987025	50	5	0,298838s	0,208119s	10,787397s	32,33981s	0,228398s	48,515975s
29.0MB	20000	250000	3987025	50	10	0,415625s	0,267941s	11,720075s	31,966688s	0,221526s	49,009997s
59.0MB	200000	500000	7973661	10	1	0,648519s	0,151854s	146,291578s	205,723093s	0,325401s	362,299561s
59.0MB	200000	500000	7973661	10	5	0,51241s	0,144497s	83,083158s	97,673991s	0,301221s	189,14674s
59.0MB	200000	500000	7973661	10	10	0,57307s	0,127956s	76,218612s	96,09852s	0,685715s	181,726768s
59.0MB	100000	500000	7973661	20	1	0,506187s	0,225328s	80,893429s	128,276976s	0,277655s	215,932899s
59.0MB	100000	500000	7973661	20	5	0,499785s	0,255679s	38,261659s	75,453095s	0,281984s	120,069202s
59.0MB	100000	500000	7973661	20	10	0,54997s	0,237825s	45,04215s	81,992073s	0,275557s	133,911063s
59.0MB	66664	500000	7973661	30	1	0,517346s	0,281069s	63,33538s	114,334016s	0,293387s	186,072599s
59.0MB	66664	500000	7973661	30	5	0,520181s	0,279336s	28,882705s	72,205272s	0,270538s	109,969836s
59.0MB	66664	500000	7973661	30	10	0,491091s	0,179746s	25,938374s	71,326605s	0,25571s	105,90059s
59.0MB	50000	500000	7973661	40	1	0,467333s	0,222036s	48,063321s	97,185025s	0,267501s	153,464437s
59.0MB	50000	500000	7973661	40	5	0,536233s	0,254484s	25,247628s	71,148591s	0,318105s	106,713974s
59.0MB	50000	500000	7973661	40	10	0,560933s	0,36164s	23,748265s	70,114565s	0,316855s	102,914785s
59.0MB	40000	500000	7973661	50	1	0,545279s	0,419361s	45,343121s	95,110909s	0,260595s	147,964658s
59.0MB	40000	500000	7973661	50	5	0,486751s	0,350889s	21,616719s	65,911156s	0,268515s	96,167534s
59.0MB	40000	500000	7973661	50	10	0,488424s	0,353103s	21,358718s	64,072706s	0,266413s	93,972705s
119.0MB	400000	1000001	15948061	10	1	0,958715s	0,211145s	298,184975s	414,716392s	0,400606s	734,607389s
119.0MB	400000	1000001	15948061	10	5	0,851987s	0,191558s	167,223288s	188,405249s	0,39648s	376,810297s
119.0MB	400000	1000001	15948061	10	10	0,8709s	0,267717s	162,978843s	189,230686s	0,387917s	374,084647s
119.0MB	200000	1000001	15948061	20	1	0,915984s	0,348692s	167,58222s	262,581544s	0,377483s	448,309767s
119.0MB	200000	1000001	15948061	20	5	0,873462s	0,368442s	89,831081s	151,552173s	0,452083s	261,771564s
119.0MB	200000	1000001	15948061	20	10	1,080715s	0,39247s	97,663494s	167,393677s	0,633641s	288,09932s
119.0MB	133332	1000001	15948061	30	1	1,224612s	0,546246s	138,921059s	237,832218s	0,495174s	397,868894s
119.0MB	133332	1000001	15948061	30	5	1,252935s	0,738831s	64,414537s	147,473782s	0,538731s	230,015345s
119.0MB	133332	1000001	15948061	30	10	0,967507s	0,490133s	54,773811s	138,353821s	0,392319s	210,723366s
119.0MB	100000	1000001	15948061	40	1	0,97597s	0,668876s	99,852836s	186,336297s	0,363885s	304,242518s
119.0MB	100000	1000001	15948061	40	5	0,916675s	0,521045s	49,524006s	143,802539s	0,394489s	211,137629s
119.0MB	100000	1000001	15948061	40	10	0,915672s	0,559019s	47,432889s	136,537117s	0,320877s	199,209469s
119.0MB	80000	1000001	15948061	50	1	0,789601s	0,491576s	89,973438s	188,241506s	0,342014s	293,919966s
119.0MB	80000	1000001	15948061	50	5	0,773435s	0,432996s	42,981991s	126,728073s	0,288309s	184,234674s
119.0MB	80000	1000001	15948061	50	10	0,801534s	0,439789s	42,613939s	126,684508s	0,3628s	183,693515s