Split Large dictionnary

[arita37]

Hello,

Sounds a good project.
Is there a way to pass / scatter a dictionnary of 500k keys ?
Dictionary is dictionnary of dictionnary... to handle complex data.

How the data are copied (because I have 5go and dont have to copy all....)

[mattja]

Thanks, that is a good suggestion.
Currently there is a special case for scattering of sequences, but not for dictionaries. It would be a useful feature and should be straightforward to add, though I do not have time to do it this month.

At the moment distob is implemented as an object layer on top of ipyparallel 4.x.
So it's really designed for computation on multiple hosts separated by network links, and in my experience
distob is very inefficient with wasted serialization, deserialization and communications overhead if using
multiple CPUs on a single host. Currently when scattering I believe all the data are serialized by dill then copied by ipyparallel across sockets, using zeromq.

In the future it should be possible for distob to use faster back-ends, including one suitable for
parallel processing on a single host without copying data needlessly (and also a back-end for GPU
computation).

[arita37]

I have one solution: similar to unix shell: very efficient and work on any code (no serialization !). Limitations: Mainly for data processing. Results should not depend on process results (independant compute). We can discuss if you want. On 19 Jan 2017, at 16:15, Matthew Aburn <[email protected]> wrote: Thanks, that is a good suggestion. Currently there is a special case for scattering of sequences, but not for dictionaries. It would be a useful feature and should be straightforward to add, though I do not have time to do it this month. At the moment distob is implemented as an object layer on top of ipyparallel 4.x. So it's really designed for computation on multiple hosts separated by network links, and in my experience distob is very inefficient with wasted serialization, deserialization and communications overhead if using multiple CPUs on a single host. Currently when scattering I believe all the data are serialized by dill then copied by ipyparallel across sockets, using zeromq. In the future it should be possible for distob to use faster back-ends, including one suitable for parallel processing on a single host without copying data needlessly (and also a back-end for GPU computation). ― You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[mattja]

Sure if you have some example code, I would be interested to have a look later, when I have some time available.

[arita37]

Lets have a chat over gitter ? Matthew Aburn wrote:

…

Sure if you have some example code, I would be interested to have a look later, when I have some time available. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AR10p_3flLVhGodPASwMOpZhgBdf9xBzks5rT_KzgaJpZM4LnG5P>.