Multiplying-Sparse-Matrices-with-MapReduce

There are two big sparse matrices M and N (each is 100k * 10k), compute the multiplication of them by using MapReduce. Matrices are stored with the following format:
<i><TAB><j><TAB><m_ij>
<j><TAB><k><TAB><n_jk>
If you want to install Hadoop cluster by yoursrlf, you can refer to my tutorial (https://zhuanlan.zhihu.com/p/58968191)
I use Hadoop streaming with Python to solve this problem. There are three MapReduce jobs in total.

Stage 1

mappper: add tag to matrices M and N so as to differentiate them. Swap i and j in M. Output is
<j><TAB><m><TAB><i><TAB><m_ij>
<j><TAB><n><TAB><k><TAB><n_jk>
reducer: no reducer in this stage

Stage 2

mapper: identity mapper reducer: Cartesian product of record with the same j from M and N matrices, output is
< i,k ><TAB>< m_ij * n_jk>

Stage 3

mapper: identity mapper reducer: sum record with same key (i,k)