different "launch time" with persist

[JohnBrouillet]

Hi !

I use map_blocks and persist(scheduler='multiprocessing') to parallelize the computation of a chunked dataset. It works great.
I ran my script on 2 VM:

1. 4 cores and 32Gb of memory
2. 8 cores and 169Gb.

Both on CentOS Linux release 7.9.2009 (Core).

xarray version : 0.18.2
dask version : 2.17.2 (we don't use the last version because of an error with the import of Client from dask.distributed)

In the parallelize function, there are prints to know where I am in the computation of the chunk.
On the first VM, when persist is called, the first prints arrive fastly. But with the second one, the prints come after 10min (but the computation is faster than on the first vm).
Do you have any idea of what could explain such a difference of time ?

Thank you !

[max-sixty]

Unfortunately there's not nearly enough information here — please see #5404.

I recognize these can be difficult to provide a full repro, but you'd at least need to provide code of an example — e.g. see #5499 for a version of this.

[JohnBrouillet]

Hi!

Sorry for the very very very late answer. I went to holiday at that moment and then completely forgot about this discussion.

I can give an answer. At that moment I was not aware that the "8 cores and 169Gb" computer had a very peculiar and quite slow shared file system. It explains why I experienced this difference of time to launch the computation.