-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.Rmd
162 lines (125 loc) · 5.42 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
---
output: github_document
editor_options:
chunk_output_type: console
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%",
message = TRUE
)
library(gitpins)
stopifnot(!file.exists(here::here("gitpins")))
```
# gitpins
<!-- badges: start -->
[](https://github.com/torfason/gitpins/actions)
[](https://github.com/torfason/gitpins/actions/workflows/R-CMD-check.yaml)
<!-- badges: end -->
`r desc::desc_get("Description")`
## The problem
You want to quickly and easily process an online resource using R functions,
some of which only accept local files. Thus you would like the following
properties for your workflow:
* Download to a local file
* But avoid downloading on every single run
* Refresh your data regularly from the online source
* Use a local copy if the online resource is not accessible
* Have the local copy be easily accessible in a predictable location
* Not ruin your local copy if the online version should change in a "bad" way
## The solution
The `gitpins` package downloads a URL to a local file in the `gitpins` folder
inside your project (the currently fixed path is determined by
`here("gitpins")`), and then returns the full file name name of the local file,
which can be passed as an argument to any function that expects to read such a
file.
```{r}
# Downloads on first try
pin("https://vincentarelbundock.github.io/Rdatasets/csv/openintro/country_iso.csv") |>
read.csv() |> head()
```
You can maintain as many resources as you need:
```{r}
# Another resource
pin("https://vincentarelbundock.github.io/Rdatasets/csv/datasets/sunspot.month.csv") |>
read.csv() |> head()
```
The file is downloaded the first time you run `pin()` on a given URL (the
actual download is done with `curl::curl_download()`). After that, it checks to
see the age of the local file and re-downloads if it is to old. The default
refresh interval is 12 hours, but is configurable with a parameter.
Note that the return value of the `pin()` function is simply the full path to
the local copy of the file. You can therefore use `pin()` with the original
URL wherever you would have used the local path of the resource. The exact name
of the file is constructed in a deterministic way based on the URL
(specifically, the base name is the `digest()` of the URL).
```{r}
# Uses a cached copy if a recent one is available (start of the url changed for privacy)
pin("https://vincentarelbundock.github.io/Rdatasets/csv/openintro/country_iso.csv") |>
gsub(pattern=".*/(gitpins/.*)", replacement="/home/user/project/\\1")
```
The refresh interval is configured with the `refresh_hours` parameter. Use
`refresh_hours=0` to force a download on every call, and `refresh_hours=Inf` to
always use the local copy (after the first download).
```{r}
# Force a reload by specifying zero refresh time
pin("https://vincentarelbundock.github.io/Rdatasets/csv/openintro/country_iso.csv",
refresh_hours = 0) |>
gsub(pattern=".*/(gitpins/.*)", replacement="/home/user/project/\\1")
# Always use local copy by specifying Inf refresh time
pin("https://vincentarelbundock.github.io/Rdatasets/csv/openintro/country_iso.csv",
refresh_hours = Inf) |>
gsub(pattern=".*/(gitpins/.*)", replacement="/home/user/project/\\1")
```
The `gitpins` directory is actually a local `git` repository, and each new
version is committed to the repository. That way, a complete history of the
downloads is kept, but if the resource is not changing a lot, this history will
not take up an inordinate amount of space (because of the deduplication
properties of `git`).
If the resource gets borked, you can retrieve older versions using git. A
function is provided to list available pins (with or without history), but
beyond that, the user is expected to use `git` directly for more complex
retrieval operations.
```{r, include=FALSE}
old_width <- options(width=130)
```
```{r}
list_pins()
list_pins(history = TRUE)
```
```{r, include=FALSE}
options(old_width)
```
## Installation
Install `gitpins` with either of the following commands:
``` r
pak::pak("torfason/gitpins")
# or alternatively
remotes::install_github("torfason/gitpins")
```
You can then load and use the package like this:
```r
library(gitpins)
pin(URL)
```
To use this concurrently with another package that also defines a `pin()`
function, exclude this function and use the alias `gitpin()` instead:
```r
library(gitpins, exclude="pin")
gitpin(URL)
```
Note that `gitpins` uses the native pipe operator (`|>`) and so depends on `R
(>= 4.1.0)`. If this is an issue for you, holler and I can probably be convinced
of changing it to make it compatible with older versions.
## Related packages and feedback
This package was inspired by the `pins` package, and in particular the
`pins::pin()` function. However, that function stores the actual local file in a
system location rather than inside the project, so using it did not prove
reliable reliable. Furthermore, it did not have the desired versioning
properties, and finally, it is now defined as a legacy function and is not part
of the new api for that package. As a result, `gitpins` was born.
For feature requests, bugs, or other feedback, feel free to file an issue.