Constant "downloading"

[sprig]

In the same setup as #133 (modified 0.6.4 from git on ubuntu 19.10, for anyone else is reading this), maestral now appears to constantly be in a "downloading X/600K files" state, after the initial downloade. Reading the logs, I see mostly that the files are resolved unchanged. I can't paste the log at the moment, but the vast majority of files state that the have "Equal Content Hashes".

I have seen it reach very low numbers - once below 10k remaining files, but when when I come back to the window, maestral appears to be syncing all over again. Since I have a rather large library, it means that the sync (if it even actually happens) takes over a day, which is rather long.

Is this normal/expected? From https://github.com/SamSchott/maestral/blob/09417eee2ba5e1d9bfb3a42ffe90a1e13d09ff34/maestral/sync.py#L1959 it seems like this is by design, but I'm not certain.

If it is indeed normal, I'm also curious about the reasoning - I would have expected "normal" syncs to be quick, after the initial slow bulk download.

[samschott]

This is definitely not by design! The method get_remote_folder is only called during the initial sync. Any future syncs will only apply changes which are provided by the Dropbox API. Additionally, a complete resync is performed once per week but you can modify this interval in the config file.

I suspect that this is a similar issue as with indexing: A connection timeout causes the downloads to be interrupted and to eventually restart from the beginning. The log output would be helpful to see which API call exactly is problematic.

I have so far chosen the easy route of not saving any download state on connection errors or crashes but just restarting from the last successful sync. This is typically not a problem because already downloaded files will be skipped if their content hashes are equal. However, it becomes problematic if the initial sync never completes. There are in principle two solutions: First, retry a certain number of times on timeouts for all API calls before giving up. Second, save the current state on a connection error and resume from there if not too much time has elapsed.

[samschott]

I had hoped increasing the timeout from 60 to 100 sec in v1.0.0 would take care of most problems due to a slow internet connection or a slowly responding API.

[sprig]

Thanks for the explanation! I'm retry the sync and post the logs.

I had hoped increasing the timeout from 60 to 100 sec in v1.0.0 would take care of most problems due to a slow internet connection or a slowly responding API.

I don't know 60 seconds is quite a long time to respond as it is.

[samschott]

Closing since this has been addressed in v1.4.0. Indexing now resumes from where is was interrupted.