Skip to content

"cannot open URL" with remotes::install_github() #130

Closed
@wibeasley

Description

@wibeasley
Contributor

I have a vanilla installation of Windows (no proxies or anything similar) where remotes::install_github("yihui/testit") won't work (it fails similarly for many other packages that do and don't require compilation). I've tried the two most recent patched versions of R (3.4.3 and 3.4.4) and both the CRAN and GitHub master version of remotes (the CRAN version predates the most recent changes to install-github.R). It's been throwing this error for at least a week.

Furthermore, these same arrangements have been working fine on my other Windows machines. And working fine on this same machine with devtools::install_github("yihui/testit") . The error message is

> remotes::install_github("yihui/testit")
Downloading GitHub repo yihui/testit@master
Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
  cannot open URL 'https://api.github.com/repos/yihui/testit/zipball/master'

Stepping through after declaring debug(utils::download.file) leads to the following misbehaving line, and I don't have the skills to debug once it cross the R-C boundary.

.External(C_download, url, destfile, quiet, mode, cacheOK, method == "wininet")

I'm not seeing anything similar amoung the 18 SO hits. Is this something I should ask SO about instead? I'm not that worried for my own case, because I can keep using devtools::install_github(), but as more people migrate to remotes, this corner case might cause more problems.

traceback
> traceback()
11: utils::download.file(url, path, method = download_method(), quiet = quiet, 
        mode = "wb")
10: withCallingHandlers(expr, warning = function(w) invokeRestart("muffleWarning"))
9: suppressWarnings(status <- utils::download.file(url, path, method = download_method(), 
       quiet = quiet, mode = "wb"))
8: base_download(real_url, path, quiet)
7: download(dest, src, auth_token = x$auth_token)
6: remote_download.github_remote(remote, quiet = quiet)
5: remote_download(remote, quiet = quiet)
4: FUN(X[[i]], ...)
3: vapply(remotes, install_remote, ..., FUN.VALUE = logical(1))
2: install_remotes(remotes, ...)
1: remotes::install_github("yihui/testit")
session info
R version 3.4.4 Patched (2018-03-19 r74446)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.4.4 tools_3.4.4    yaml_2.1.18    remotes_1.1.1 

Activity

gaborcsardi

gaborcsardi commented on Mar 31, 2018

@gaborcsardi
Member

This is coming from utils::download.file so either that does not support HTTPS, or you need a GH API token because your IP address was rate limited, or some other error, like a bad proxy setup.

Unfortunately there is not much we can do about it in remotes, other than maybe trying to a different download method, or try multiple methods.

But for that we would need to be able to recreate your problem, so some questions:

  • does it always fail, or does it sometimes work? You can see your GH rate limit, and when it resets with sg like gh::gh("/rate_limit")
  • does it work if you call utils::download.file() directly, with the same URL?
  • does it work for the various utils::download.file() methods?
wibeasley

wibeasley commented on Apr 4, 2018

@wibeasley
ContributorAuthor

Your intuition was right. It appears to be influenced by the 'method' parameter in utils::download.file().

  1. It always failed -it seems unrelated to rate limits
  2. It failed with the default method (see the first entry under "'method' values that fail")
  3. The method="libcurl" option worked. All others failed.

What would be helpful for me to do next?

source_url <- "https://api.github.com/repos/yihui/testit/zipball/master"
capabilities("libcurl")
# libcurl 
#    TRUE 

utils::download.file(source_url, destfile="aa.zip", method="libcurl")
# trying URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# Content length 16294 bytes (15 KB)
# downloaded 15 KB
'method' values that fail
utils::download.file(source_url, destfile="aa.zip")
# trying URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# Error in utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  : 
# cannot open URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# In addition: Warning message:
# In utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  :
# InternetOpenUrl failed: 'A connection with the server could not be established'

utils::download.file(source_url, destfile="aa.zip", method="wininet")
# trying URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# Error in utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  :
# cannot open URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# In addition: Warning message:
# In utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  :
# InternetOpenUrl failed: 'A connection with the server could not be established'

utils::download.file(source_url, destfile="aa.zip", method="internal")
# Error in utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  : 
# scheme not supported in URL 'https://api.github.com/repos/yihui/testit/zipball/master'


utils::download.file(source_url, destfile="aa.zip", method="wget")
# Error in utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  : 
# 'wget' call had nonzero exit status
# In addition: Warning message:
# running command 'wget  "https://api.github.com/repos/yihui/testit/zipball/master" -O "aa.zip"' had status 127 

utils::download.file(source_url, destfile="aa.zip", method="curl")
# Error in utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  : 
# 'curl' call had nonzero exit status
# In addition: Warning message:
# running command 'curl  "https://api.github.com/repos/yihui/testit/zipball/master"  -o "aa.zip"' had status 127 

utils::download.file(source_url, destfile="aa.zip", method="auto")
# trying URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# Error in utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  : 
# cannot open URL 'https://api.github.com/repos/yihui/testit/zipball/master'
# In addition: Warning message:
# In utils::download.file("https://api.github.com/repos/yihui/testit/zipball/master",  :
# InternetOpenUrl failed: 'A connection with the server could not be established'
cderv

cderv commented on Apr 4, 2018

@cderv
Contributor

In remotes::install_github, the download method is determined by remotes::download_method() and set to "auto" except on older R version. On windows, it is not libcurl by defaut but wininet when auto is set.

FWIW, I can't reproduce the issue on my windows computer, but I am on windows 10.

Could it come from a different behaviour of wininet between different version of windows?
I will try to verify that.

One solution could be to check for libcurl availability by default but it was the previous behaviour that did not allowed windows proxy behavior offered by wininet.

Reprex

with defaults
tmp_zip <- tempfile(fileext = ".zip")
source_url <- "https://api.github.com/repos/yihui/testit/zipball/master"
utils::download.file(source_url, destfile=tmp_zip)
file.exists(tmp_zip)
#> [1] TRUE
with wininet explicitly
tmp_zip <- tempfile(fileext = ".zip")
source_url <- "https://api.github.com/repos/yihui/testit/zipball/master"
utils::download.file(source_url, destfile=tmp_zip, method = "wininet")
file.exists(tmp_zip)
#> [1] TRUE
with auto from remotes:::download_method()
tmp_zip <- tempfile(fileext = ".zip")
source_url <- "https://api.github.com/repos/yihui/testit/zipball/master"
utils::download.file(source_url, destfile=tmp_zip, method = remotes:::download_method())
file.exists(tmp_zip)
#> [1] TRUE

Created on 2018-04-04 by the reprex package (v0.2.0).

Session info
devtools::session_info()
#> - Session info ----------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.4.3 (2017-11-30)
#>  os       Windows 10 x64              
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                        
#>  collate  French_France.1252          
#>  tz       Europe/Paris                
#>  date     2018-04-04                  
#> 
#> - Packages --------------------------------------------------------------
#>  package     * version     date      
#>  ansistrings   1.0.0.9000  2018-03-31
#>  assertthat    0.2.0       2017-04-11
#>  backports     1.1.2       2017-12-13
#>  callr         2.0.2.9002  2018-03-31
#>  cli           1.0.0.9002  2018-03-31
#>  clisymbols    1.2.0       2018-02-04
#>  crayon        1.3.4       2017-09-16
#>  debugme       1.1.0       2017-10-22
#>  desc          1.1.1       2017-08-03
#>  devtools      1.13.5.9000 2018-03-31
#>  digest        0.6.15      2018-01-28
#>  evaluate      0.10.1      2017-06-24
#>  glue          1.2.0       2017-10-29
#>  hms           0.4.2       2018-03-10
#>  htmltools     0.3.6       2017-04-28
#>  knitr         1.20        2018-02-20
#>  magrittr      1.5         2014-11-22
#>  memoise       1.1.0       2017-04-21
#>  pkgbuild      1.0.0       2018-03-31
#>  pkgconfig     2.0.1       2017-03-21
#>  pkgload       1.0.0       2018-03-31
#>  prettyunits   1.0.2       2015-07-13
#>  progress      1.1.2.9002  2018-03-31
#>  R6            2.2.2       2017-06-17
#>  Rcpp          0.12.16     2018-03-13
#>  rlang         0.2.0.9001  2018-03-28
#>  rmarkdown     1.9         2018-03-01
#>  rprojroot     1.3-2       2018-01-03
#>  selectr       0.4-0       2018-03-27
#>  sessioninfo   1.0.1.9000  2018-01-11
#>  stringi       1.1.7       2018-03-12
#>  stringr       1.3.0       2018-02-19
#>  testthat      2.0.0.9000  2018-03-31
#>  usethis       1.3.0       2018-02-24
#>  withr         2.1.2       2018-03-28
#>  xml2          1.2.0       2018-01-24
#>  yaml          2.1.18      2018-03-08
#>  source                                 
#>  Github (r-lib/ansistrings@4e4d309)     
#>  CRAN (R 3.4.0)                         
#>  CRAN (R 3.4.3)                         
#>  Github (r-lib/callr@9c38827)           
#>  Github (r-lib/cli@53c129a)             
#>  Github (gaborcsardi/clisymbols@e49b4f5)
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.1)                         
#>  Github (r-lib/devtools@2f7bc84)        
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.3)                         
#>  Github (r-lib/pkgbuild@015f7f6)        
#>  CRAN (R 3.4.0)                         
#>  Github (r-lib/pkgload@f827a27)         
#>  CRAN (R 3.4.0)                         
#>  Github (r-lib/progress@72c9873)        
#>  CRAN (R 3.4.2)                         
#>  CRAN (R 3.4.4)                         
#>  Github (tidyverse/rlang@49d7a34)       
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.4)                         
#>  Github (r-lib/sessioninfo@c871d01)     
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.3)                         
#>  Github (hadley/testthat@5ed0bb1)       
#>  CRAN (R 3.4.4)                         
#>  Github (r-lib/withr@79d7b0d)           
#>  CRAN (R 3.4.3)                         
#>  CRAN (R 3.4.3)
gaborcsardi

gaborcsardi commented on Apr 4, 2018

@gaborcsardi
Member

@wibeasley I suspect that this is a TLS issue, GitHub does not support TLS v1 and v1.1

See e.g. https://githubengineering.com/crypto-deprecation-notice/

Older Windows systems only support TLS v1.x, so the system's builtin HTTP lib does not work with GitHub. It is possible that you can enable TLS v1.2 or you can download a patch from MS. E.g. https://cloudblogs.microsoft.com/microsoftsecure/2017/07/20/tls-1-2-support-added-to-windows-server-2008/

It depends on your windows version.

To make this more robust, remotes could try a couple of methods, at least on windows trying libcurl and wininet would make sense.

wibeasley

wibeasley commented on Apr 4, 2018

@wibeasley
ContributorAuthor

It is possible that you can enable TLS v1.2 or you can download a patch...

I don't mind doing that or either of the two other solutions (eg, specifying "libcurl", or calling devtools::install_github()). But these may not be feasible to expect other remotes::install_github() users to do without a lot of problems encountered.

If you think I have a weird computer failing on a trivial issue, I'm happy to drop the issue. Or, if you think I have a canary in a coalmine, I'm happy to provide more info to help make
remotes::install_github() more robust. I'm cool with whatever you feel is the best use of your time.

In case this helps create a more dynamic/robust, here's info on the three machines. Notice the failing one isn't the oldest OS.

I can't find a difference in these 'predictor' variables. Again, I don't care about this particular machine --but more about its value improving the robustness of remotes::install_github() (eg, finding an environmental setting/variable value that determine the best download method).

Windows machine 1 (which fails):

  1. download outcome: utils::download.file(source_url, destfile="aa.zip") see error message above
  2. Windows 8
  3. R.version 3.4.4 Patched
  4. getOption("download.file.method"): "wininet"
  5. capabilities(c("libcurl", "http/ftp")): TRUE TRUE
  6. utils::download.file(source_url, destfile="aa.zip", method="libcurl"): works
  7. devtools::install_github("yihui/testit") : works
  8. request <- httr::GET(source_url); writeBin(httr::content(request, "raw"), con="aa.zip"): works (ie, my interpretation of devtool's internals)

Windows machine 2 (which works):

  1. download outcome: utils::download.file(source_url, destfile="aa.zip") works
  2. Windows 7
  3. R.version 3.4.4 Patched
  4. getOption("download.file.method"): "wininet"
  5. capabilities(c("libcurl", "http/ftp")): TRUE TRUE
  6. utils::download.file(source_url, destfile="aa.zip", method="libcurl"): works
  7. devtools::install_github("yihui/testit") : works
  8. request <- httr::GET(source_url); writeBin(httr::content(request, "raw"), con="aa.zip"): works

Windows machine 3 (which works):

  1. download outcome: utils::download.file(source_url, destfile="aa.zip") works
  2. Windows 10
  3. R.version 3.4.4 Patched
  4. getOption("download.file.method"): "wininet"
  5. capabilities(c("libcurl", "http/ftp")): TRUE TRUE
  6. utils::download.file(source_url, destfile="aa.zip", method="libcurl"): works
  7. devtools::install_github("yihui/testit") : works
  8. request <- httr::GET(source_url); writeBin(httr::content(request, "raw"), con="aa.zip"): works

Follow question:

Ideas why devtools::install_github() doesn't have this problem? Is it because httr::GET() is used instead? (Notice I included that as the 8th characteristic in the list above.)

I see a lot of Google hits for ""InternetOpenUrl failed: 'A connection with the server could not be established'". Is this why you guys avoided utils::download.file() in devtools::install_github()? I understand the appeal of avoiding additional package dependencies (eg httr) and relying on required packages (like utils).

gaborcsardi

gaborcsardi commented on Sep 22, 2018

@gaborcsardi
Member

Ideas why devtools::install_github() doesn't have this problem? Is it because httr::GET() is used instead?

httr uses curl, which uses its own libcurl.

I am pretty sure that this is the GH TLS issue. If you still have that problematic machine, can you please try to install TLS 1.2 support? Note that the next release of devtools will use remotes, so the devtools workaround will soon stop working.

wibeasley

wibeasley commented on Sep 23, 2018

@wibeasley
ContributorAuthor

This guide recommended enabling TLS 1.2 in IE 10. That worked. But it's not very satisfying because when I then unchecked the box, it still worked. I'm guessing that checking the box the first time set something in the registry that helped utils::download.file()?

image

I didn't see anything to install in that link above that applies to Windows 8. I think part of the complexity was that the machine can't be upgraded to Windows 8.1.

This isn't my main computer, so again, I don't really care for my sake. But if there's something else I can run or do that might help you, please tell me. otherwise, I think this issue can be closed.

gaborcsardi

gaborcsardi commented on Sep 23, 2018

@gaborcsardi
Member

Thanks! I don't think we can do much more here.

ShaowenJ

ShaowenJ commented on Nov 27, 2018

@ShaowenJ

Hi guys, if I can use utils::download.file(,method="libcurl") to download the file like wibeasley did, how can I install it?

36 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @gaborcsardi@wibeasley@Baneeishaque@cderv@alinakamanina

        Issue actions

          "cannot open URL" with remotes::install_github() · Issue #130 · r-lib/remotes