Files over 4GB are not correctly synced

[Fabian42]

I have a folder with two copies of a 4.4GB (4669979776B) file. One of those copies got downloaded by GoSync as a 1000MB (1048576010B=1024²×1000B+10B) file, one as a 2.5GB (2726297626B=???) file. The copy in the cloud was not damaged.

[hschauhan]

This is because the download sometimes fails. The idea is to retry. Alain is working on adding exponential backoff algorithm for retries. Should be fixed soon.

[Fabian42]

The download failed after exactly 1000MB+10B? That seems to be too exact of a number to be a concidence.

[arentoine]

Hello,

a fix was added to GoSync to handle large file download.

Can you update GoSync with the current version and share your comments ?

Regards

[Fabian42]

Not worth its own report, but: "Debuging"→"Debugging"

[arentoine]

Hello,

does it work ? :-)

If not, do you have log ?

Thanks

[hschauhan]

Not worth its own report, but: "Debuging"→"Debugging"

Ouch!

[Fabian42]

I was testing it and doing other stuff at the same time, so I didn't notice that I had to click "resume sync". Now I did and it instantly ends again:

2020-05-26 20:44:05,701 - GoSync - DEBUG - SyncThread - run - Trying to acquire lock.
2020-05-26 20:44:05,701 - GoSync - DEBUG - SyncThread - run - Lock acquired.
2020-05-26 20:44:05,701 - GoSync - DEBUG - SyncThread - run - Validating sync settings
2020-05-26 20:44:05,702 - GoSync - INFO - SyncThread - run - validated
2020-05-26 20:44:05,702 - GoSync - INFO - SyncThread - run - Staring the sync now
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - Start - Syncing remote directory
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - End - Syncing remote directory
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################

2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - Start - Syncing local directory
2020-05-26 20:44:05,702 - GoSync - DEBUG - ###############################################
2020-05-26 20:44:05,702 - GoSync - DEBUG - ### SyncLocalDirectory: - Sync Started
2020-05-26 20:44:05,702 - GoSync - ERROR - SyncThread - run - Unknown exception
2020-05-26 20:44:05,702 - GoSync - INFO - SyncThread - run - Sync done

10/10 error message right there.

[arentoine]

Hello,

issue is in the upload section. That definitely need some fixing.

Did you test large file download ? From GoogleDrive to Local ?

Thanks for you comments.

[Fabian42]

I just starting syncing. Since large files didn't work before, that would have been the only thing to sync for it. But it didn't even start syncing properly.

…

On Tue, 26 May 2020 at 20:50, arentoine ***@***.***> wrote: Hello, issue is in the upload section. That definitely need some fixing. Did you test large file download ? From GoogleDrive to Local ? Thanks for you comments. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#38 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACP66YXVSBD5E6AFDWTDIY3RTQFOFANCNFSM4NGLNEEQ> .

[arentoine]

Do you know if you have a "half downloaded file" sitting in your local drive ?

[Fabian42]

I do, the big ones.

…

On Tue, 26 May 2020 at 20:57, arentoine ***@***.***> wrote: Do you know if you have a "half downloaded file" sitting in your local drive ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#38 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACP66YWGCIUKYH7YM42BQUTRTQGHVANCNFSM4NGLNEEQ> .

[arentoine]

Can you move the incomplete files in a safe place out of the sync directory. Then if you could update your tree with the most uptodate version. Then tell us if this will help you ?

Regards

[Fabian42]

It seems to have actually uploaded the small, broken version of one of the big files again, duplicating it on Drive. It did not do that for any other one.

I did what you said, moved the broken files somewhere else and tried syncing again. This is what happened last:

2020-05-27 15:22:56,384 - GoSync - DEBUG - Checking: file.mp4

2020-05-27 15:22:56,385 - GoSync - DEBUG - SyncRemoteDirectory: Checking file (file.mp4)
2020-05-27 15:22:56,392 - GoSync - DEBUG - DownloadFileByObject: Download Started - File (/home/fabian/Google Drive/[email protected]/path/to/file.mp4)
2020-05-27 15:22:56,392 - GoSync - DEBUG - DownloadFileByObject: Download Started - File size (1301474114)
2020-05-27 15:22:56,517 - GoSync - DEBUG - Observer: /home/fabian/Google Drive/[email protected]/path/to/file.mp4 created

2020-05-27 15:22:56,522 - GoSync - DEBUG - UploadObservedFile: File /home/fabian/Google Drive/[email protected]/path/to/file.mp4 is created but sync is running. Possibly created by sync. Skipping.
2020-05-27 15:26:24,527 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (file.mp4)
2020-05-27 15:26:24,528 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (to)
2020-05-27 15:26:24,528 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (path)
2020-05-27 15:26:24,529 - GoSync - ERROR - SyncThread - run - Unknown exception
2020-05-27 15:26:24,529 - GoSync - INFO - SyncThread - run - Sync done
2020-05-27 15:26:54,570 - GoSync - DEBUG - SyncThread - run - Waiting for Sync to be enabled
2020-05-27 15:27:14,716 - GoSync - INFO - SyncThread - run - Internet is down. Clearing running.

Now it's stuck at "Network is down" and nothing happens anymore.

[arentoine]

I will prepare a special python module to gather more information about the issue.

I'll get back to you in a couple of days ...

Greetings

[Fabian42]

Just a quick reminder that I only test this because it's fun and helps you, I don't really need the program. If nobody else has my issues, you don't have to spend all that time.

[hschauhan]

Hi Fabian, Your testing is quite important to this project. It has uncovered lot of problems and made the project quite robust as compared to previous versions. Please keep testing if you can. Regards Himanshu

…

On Thu, 28 May 2020 at 1:43 PM, Fabian Röling ***@***.***> wrote: Just a quick reminder that I only test this because it's fun and helps you, I don't really need the program. If nobody else has my issues, you don't have to spend all that time. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#38 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABXIY53IDXSB3HKYKQ4IKLRTYMJBANCNFSM4NGLNEEQ> .

[arentoine]

Exactly !

Keep having fun testing as much as we do resolving bugs.

And may be, someone will benefit from that !

Thanks again!

[hschauhan]

I found the problem, a silly one. Will commit tomorrow.

…

On Wed, 27 May 2020 at 12:20 AM, arentoine ***@***.***> wrote: Hello, issue is in the upload section. That definitely need some fixing. Did you test large file download ? From GoogleDrive to Local ? Thanks for you comments. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#38 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABXIY4AIZBTE4ZNZT7SNN3RTQFOFANCNFSM4NGLNEEQ> .

[hschauhan]

I have committed the changes and local sync is working fine.

[Fabian42]

2020-05-30 12:12:36,500 - GoSync - DEBUG - Checking: file.iso

2020-05-30 12:12:36,501 - GoSync - DEBUG - SyncRemoteDirectory: Checking file (file.iso)
2020-05-30 12:12:36,506 - GoSync - DEBUG - DownloadFileByObject: Download Started - File (/home/fabian/Google Drive/[email protected]/path/to/file.iso)
2020-05-30 12:12:36,507 - GoSync - DEBUG - DownloadFileByObject: Download Started - File size (4699979776)
2020-05-30 12:12:36,576 - GoSync - DEBUG - Observer: /home/fabian/Google Drive/[email protected]/path/to/file.iso created

2020-05-30 12:12:36,586 - GoSync - DEBUG - UploadObservedFile: File /home/fabian/Google Drive/[email protected]/path/to/file.iso is created but sync is running. Possibly created by sync. Skipping.
2020-05-30 12:14:57,565 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (file.iso)
2020-05-30 12:14:57,565 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (to)
2020-05-30 12:14:57,566 - GoSync - ERROR - SyncRemoteDirectory: Failed to sync directory (path)
2020-05-30 12:14:57,566 - GoSync - ERROR - SyncThread - run - Unknown exception
2020-05-30 12:14:57,566 - GoSync - INFO - SyncThread - run - Sync done
2020-05-30 12:15:27,600 - GoSync - DEBUG - SyncThread - run - Waiting for Sync to be enabled
2020-05-30 12:15:47,744 - GoSync - INFO - SyncThread - run - Internet is down. Clearing running.
2020-05-30 14:41:00,783 - GoSync - INFO - calculateUsage - GoSync is shutting down!

I just tried the "exit" menu option for the first time and after confirming, both the confirmation dialogue and the program froze and needed to be forcibly shut down.