install_github() unexpectedly fails with no clear error message

[DanChaltiel]

Hi,

I'm trying to install my package from GitHub.

The package is up to date on GitHub and installs with no problem from RStudio and using install.packages("myfile.tar.gz", repos = NULL, type="source").

For a (hopefully) reproducible example, you can try remotes::install_github("DanChaltiel/EDCimport"), and you can verify it by cloning https://github.com/DanChaltiel/EDCimport and installing it from RStudio.

I'm using Windows 10 and the default options("download.file.method"") was "wininet" but I tried them all:

options("download.file.method" = "internal")
remotes::install_github("DanChaltiel/EDCimport")
#> Downloading GitHub repo DanChaltiel/EDCimport@HEAD
#> Error in utils::download.file(url, path, method = method, quiet = quiet,  : 
#>   impossible d'ouvrir l'URL 'http://api.github.com/repos/DanChaltiel/EDCimport/tarball/HEAD'
options("download.file.method" = "wininet")
remotes::install_github("DanChaltiel/EDCimport")
#> Downloading GitHub repo DanChaltiel/EDCimport@HEAD
#> Error: Failed to install 'EDCimport' from GitHub:
#>   incomplete block on file
options("download.file.method" = "libcurl")
remotes::install_github("DanChaltiel/EDCimport")
#> Downloading GitHub repo DanChaltiel/EDCimport@HEAD
#> Error in utils::download.file(url, path, method = method, quiet = quiet,  : 
#>   impossible d'ouvrir l'URL 'https://api.github.com/repos/DanChaltiel/EDCimport/tarball/HEAD'
options("download.file.method" = "wget")
remotes::install_github("DanChaltiel/EDCimport")
#> Downloading GitHub repo DanChaltiel/EDCimport@HEAD
#> Error in utils::download.file(url, path, method = "wget", quiet = quiet,  : 
#>   'wget' call had nonzero exit status
options("download.file.method" = "curl")
remotes::install_github("DanChaltiel/EDCimport")
#> Downloading GitHub repo DanChaltiel/EDCimport@HEAD
#> Error in utils::download.file(url, path, method = "curl", quiet = quiet,  : 
#>   'curl' call had nonzero exit status

Also, while they point to the exact same function, I get different results with devtools::install_github():

options("download.file.method" = "wininet") #the default on my computer
remotes::install_github("DanChaltiel/EDCimport")
#> Downloading GitHub repo DanChaltiel/EDCimport@HEAD
#> Error: Failed to install 'EDCimport' from GitHub:
#>   incomplete block on file
devtools::install_github("DanChaltiel/EDCimport")
#> Downloading GitHub repo DanChaltiel/EDCimport@HEAD
#> External tar failed with `--force-local`, trying without
#> Warning in utils::untar(tarfile, ...): 'tar.exe -xf "C:
#> \Users\D_CHAL~1\AppData\Local\Temp\RtmpYFljPN\file1d8858347533.tar.gz" -C "C:/
#> Users/D_CHAL~1/AppData/Local/Temp/RtmpYFljPN/remotes1d8829715056"' returned
#> error code 2
#> External tar failed with `--force-local`, trying without
#> Warning in system(cmd, intern = TRUE): running command 'tar.exe -tf "C:
#> \Users\D_CHAL~1\AppData\Local\Temp\RtmpYFljPN\file1d8858347533.tar.gz"' had
#> status 2
#> 
#>          checking for file 'C:\Users\d_chaltiel\AppData\Local\Temp\RtmpYFljPN\remotes1d8829715056\DanChaltiel-EDCimport-d474b07/DESCRIPTION' ...     checking for file 'C:\Users\d_chaltiel\AppData\Local\Temp\RtmpYFljPN\remotes1d8829715056\DanChaltiel-EDCimport-d474b07/DESCRIPTION' ...   v  checking for file 'C:\Users\d_chaltiel\AppData\Local\Temp\RtmpYFljPN\remotes1d8829715056\DanChaltiel-EDCimport-d474b07/DESCRIPTION'
#>       -  preparing 'EDCimport':
#>    checking DESCRIPTION meta-information ...     checking DESCRIPTION meta-information ...   v  checking DESCRIPTION meta-information
#>       -  checking for LF line-endings in source and make files and shell scripts
#>   -  checking for empty or unneeded directories
#>       -  building 'EDCimport_0.2.0.9001.tar.gz'
#>      
#> 
#> Warning in i.p(...): installation of package 'C:/Users/D_CHAL~1/AppData/Local/
#> Temp/RtmpYFljPN/file1d884d392ac9/EDCimport_0.2.0.9001.tar.gz' had non-zero exit
#> status

^{Created on 2022-11-07 with reprex v2.0.2}

Note that I can install other packages from GitHub with no problem.

[gaborcsardi]

Can you download that file?

download.file("https://api.github.com/repos/DanChaltiel/EDCimport/tarball/HEAD", tempfile())

[DanChaltiel]

Yes, I can download it but there indeed seems to be a problem here.
Using 7zip, I can uncompress the .gz but I cannot open the .tar. Same when I download from Chrome.
If I download any other repo tarball, there is no problem.

This may be a problem on the GitHub side, but my google search yielded no good results about it.
If this is beyond the scope of {remotes}, would you have any idea about how I could debug this?
If a test is easy to implement, maybe a custom error message in remotes would be interesting to add (but that may be a corner case).

[gaborcsardi]

Try setting the TAR env var to internal.

[jranke]

This looks similar to the problem I have with remotes specified in the DESCRIPTION of my pfm package:

https://app.travis-ci.com/github/jranke/pfm/builds/257655186

Somehow download.file as called from within remotes fails for these github api urls in these cases.

On my local Debian system, running remotes::install_deps(".") or remotes::install_deps(".", dependencies = TRUE) is fine, I am wondering why it does not succeed on Travis.

[michaeleekk]

I had downloading issue recently with install_github() also.

remotes::install_github('satijalab/seurat-wrappers')
Downloading GitHub repo satijalab/seurat-wrappers@HEAD
Error in utils::download.file(url, path, method = method, quiet = quiet, :
download from 'https://api.github.com/repos/satijalab/seurat-wrappers/tarball/HEAD' failed

I came across a stack overflow thread and changing the timeout to a larger number, options(timeout=9999999), solved my problem.

[williamboman]

Cross-posting #706 (comment) here in case it helps.

[gaborcsardi]

I am closing this now, but please reopen with more information if you still have this issue.

[DanChaltiel]

Sorry, I forgot this issue was open.

This was actually a very weird firewall/proxy problem.

In fact, my package repository contains some password-protected archives that are needed for some tests. My office firewall does not allow these files for security reasons.

What I didn't know was possible, is that the tarball would download fine until the password-protected archives were detected, which caused the download to stop unexpectedly, leaving the file incomplete.

As the archives are only used for tests, adding an export-ignore line in a .gitattributes file solved my problem.

This issue is probably not worth fixing from inside remotes, but you could address this specific case with a specific error message.