Re-run failed uploads?

[caphector]

I've noticed that sometimes the daily files will fail to transfer (star trails, keograms, timelapses). Is there a way to re-run these uploads or should I use something like rsync?

[aaronwmorris]

The status of file transfers, S3 uploads, and SyncAPI transfers are all tracked in the database. The ./misc/upload_sync.py will reattempt any upload that failed.

source virtualenv/indi-allsky/bin/activate

./misc/upload_sync.py

[aaronwmorris]

The 10s timeout is the connect timeout. This means the remote side is not accepting the connection at Layer 4 (or you have a bad network connection). It sounds like the remote side may have firewall rules that limit a single connection from each source IP. Even with a single upload, it may take 1-2 seconds after each upload for the remote side to recognize your connection is no longer there.

I may need to introduce the ability to sleep between each transfer.

[aaronwmorris]

Actually, before I do this. Is indi-allsky running (and uploading) at the same time as trying to run the upload sync?

[caphector]

It was running when I tried to do the sync.

[aaronwmorris]

If indi-allsky is performing uploads, and your hosting provider is limiting connections, you may have to stop indi-allsky while you perform a sync.

[caphector]

I don't think I have limited connections; I upload to a VPS and I was able to make four separate SFTP connections via sftp on my CLI. I also tested the connection by making a direct sftp connection to my host while Indi-Allsky was running.

[gaitskell]

I have found that many .FITS files didn't successfuly copy from the local raspi, up to GCP (Google Cloud Platform). The much smaller .jpg's are present. I believe my WiFi link to the raspi may have been substandard for a few nights. I am going to try upload_sync.py suggestion to see if the FITS missing from GCP get filled in. (I will do this during daytime when no live images are uploading.)

[gaitskell]

Advice for others:
I am maintaining a GCP [Google Cloud] archive of all the FITS and JPG from nightly sessions, plus the diagnostic images/movies for night's viewing.

I need to fill in the files that are missing in GCP.

I used the following CLI commands. This also provides a log of transfers in .log file

cd ~/indi-allsky/
source virtualenv/indi-allsky/bin/activate
./misc/upload_sync.py sync

Note argument sync

Initially I had
webUI > Config > Object Storage > Connect Timeout
set to 10 secs. This created errors in the transfers for mp4.
So I increased this to 60 secs in the WebUI and the transfers could complete within that time.