Skip to content
This repository has been archived by the owner on Mar 9, 2021. It is now read-only.

Commit

Permalink
Prevent duplicate downloading.
Browse files Browse the repository at this point in the history
- If a file is already being used by another process, return a
successful state in the DownloadBinaryFile method of the Downloader
class. Otherwise the Downloader will try with a different host or file
size and might download a duplicate in a smaller dimension. This might
happen if two threads find the same url in the queue (duplicate posted
in the blog).
  • Loading branch information
johanneszab committed Jul 15, 2017
1 parent e54189b commit 219413b
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 4 deletions.
4 changes: 2 additions & 2 deletions src/TumblThree/SharedAssemblyInfo.cs
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,5 @@

[assembly: ComVisible(false)]
[assembly: NeutralResourcesLanguage("en-US", UltimateResourceFallbackLocation.MainAssembly)]
[assembly: AssemblyVersion("1.0.8.1")]
[assembly: AssemblyFileVersion("1.0.8.1")]
[assembly: AssemblyVersion("1.0.8.2")]
[assembly: AssemblyFileVersion("1.0.8.2")]
10 changes: 8 additions & 2 deletions src/TumblThree/TumblThree.Applications/Downloader/Downloader.cs
Original file line number Diff line number Diff line change
Expand Up @@ -229,17 +229,23 @@ protected virtual async Task<bool> DownloadBinaryFile(string fileLocation, strin
}
catch (IOException ex) when ((ex.HResult & 0xFFFF) == 0x27 || (ex.HResult & 0xFFFF) == 0x70)
{
// Disk Full, HRESULT: ‭-2147024784‬ == 0xFFFFFFFF80070070
Logger.Error("ManagerController:Download: {0}", ex);
shellService.ShowError(ex, Resources.DiskFull);
crawlerService.StopCommand.Execute(null);
return false;
}
catch (IOException ex) when ((ex.HResult & 0xFFFF) == 0x20)
{
// The process cannot access the file because it is being used by another process.", HRESULT: -2147024864 == 0xFFFFFFFF80070020
return true;
}
catch (WebException webException) when ((webException.Response != null))
{
var webRespStatusCode = (int)((HttpWebResponse)webException?.Response).StatusCode;
if (webRespStatusCode >= 400 && webRespStatusCode < 600) // removes inaccessible files: status code 400 -- 599
if (webRespStatusCode >= 400 && webRespStatusCode < 600) // removes inaccessible files: http status codes 400 to 599
{
try { File.Delete(fileLocation); }
try { File.Delete(fileLocation); } // could be open again in a different thread
catch { }
}
return false;
Expand Down

0 comments on commit 219413b

Please sign in to comment.