DDStore

Enabling efficient distributed data loading for distributed data parallelism

Installation

pip install numpy mpi4py Cython # install pre-requisites

# for use with PYTHONPATH=$PWD:$PYTHONPATH
CC=mpicc CXX=mpicxx python setup.py build_ext --inplace

# or, for installing into your virtual environment using pip
CC=mpicc CXX=mpicxx pip install .

# or, as above but in development mode
CC=mpicc CXX=mpicxx pip install -e .

Test

mpirun -n 4 python -m mpi4py test/demo.py