Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build without warnings nor notes #7

Merged
merged 3 commits into from
Apr 23, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file removed .DS_Store
Binary file not shown.
3 changes: 3 additions & 0 deletions .Rbuildignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
^.*\.Rproj$
^\.Rproj\.user$
README.Rmd
^\.travis.yml
^tmp
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,5 @@
.Ruserdata
data
data/*
.DS_Store
tmp/
11 changes: 6 additions & 5 deletions DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,14 @@ Title: A package to search for and download data from the Canadian Open Governme
Version: 0.1.0
Author: Valentin Lucet [aut, cre]
Maintainer: Valentin Lucet <[email protected]>
Description: ropencan allows users to serach for existing dataset on the Governement of Canada
open data portal.
Description: rgovcan allows users to search for existing dataset on the Canadian Open Government portal (<https://open.canada.ca/en>).
Imports:
ckanr, dplyr, tidyr, purrr, data.table
ckanr, dplyr, purrr, stringr
License: GPL-3 + file LICENSE
Encoding: UTF-8
LazyData: true
RoxygenNote: 6.1.1
Authors@R: person("Valentin", "Lucet", email = "[email protected]",
RoxygenNote: 7.1.0
Authors@R: person("Valentin", "Lucet", email = "[email protected]",
role = c("aut", "cre"))
person("Kevin", "Cazelles", email = "[email protected]",
role = c("ctb"))
3 changes: 2 additions & 1 deletion R/ckan_package_stack-class.R
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@
new_ckan_package_stack <- function(x = list()){
structure(x,
class = "ckan_package_stack",
dim = length(x))}
dim = length(x))
}

#' @export
print.ckan_package_stack <- function(x, ...) {
Expand Down
1 change: 1 addition & 0 deletions R/ckan_resource_stack-class.R
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#' ckan_resource_stack class and helpers
#'
#' @param x a list
#' @param ... ignored.
#' @keywords internal

new_ckan_resource_stack <- function(x = list()){
Expand Down
43 changes: 23 additions & 20 deletions R/govcan_dl_resources.R
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#' Download the resources attached to a specific record or (i.e. a CKAN package)
#' or to a stack of packages,
#'
#' @description Download resources attached to a specific record or (i.e. a CKAN package)
#' or to a stack of packages
#'
#' @description Download resources attached to a specific record or (i.e. a CKAN
#' package) or to a stack of packages.
#'
#' @param resources An object of the class ckan_package_stack or ckan_package, or an id of
#' a specific record or (i.e. a CKAN package), or an object of type ckan_resource or
#' ckan_resource_stack
Expand All @@ -16,59 +16,62 @@
#' }
#' @param where (string) One of "session" is files have to be charged in the session
#' or a path to the folder in which to download the files
#' @param ... extra argument(s)
#'
#' @export
govcan_dl_resources <- function(resources,
govcan_dl_resources <- function(resources, file_formats, where,
...) {
UseMethod("govcan_dl_resources")
}

#' @describeIn govcan_dl_resources Method for ckan_resource objects.
#' @export
govcan_dl_resources.ckan_resource <- function(resource,
govcan_dl_resources.ckan_resource <- function(resources,
file_formats = c("CSV"),
where = getwd()){
all_formats <- unlist(resource$format)
where = getwd(), ...){
all_formats <- unlist(resources$format)
wanted_indices <- (all_formats %in% file_formats)

if (wanted_indices == TRUE){
if (where == "session"){

message("Warning: the session option is currently not working well due to issues in ckanr")
ckanr::fetch(resource$url, store = "session")
write_import_messgae(resource)
ckanr::ckan_fetch(resources$url, store = "session")
write_import_message(resources)

} else if (where != "session"){

resource_name <- get_resource_name(resource)
resource_name <- get_resource_name(resources)
path <- create_storing_path(where, resource_name)

ckanr::fetch(resource$url, store = "disk", path = path)
write_dl_messgae(resource, path)
ckanr::ckan_fetch(resources$url, store = "disk", path = path)
write_dl_message(resources, path)

}
} else {
message("Nothing matching, no download")
}
}

#' @describeIn govcan_dl_resources Method for ckan_resource_stack objects.
#' @export
govcan_dl_resources.ckan_resource_stack <- function(resources,
file_formats = c("CSV"),
where = getwd()){
where = getwd(), ...){

all_formats <- unlist(purrr::map(resources, ~.x$format))
wanted_indices <- which(all_formats %in% file_formats)

if (length(wanted_indices) > 0 ){
if (length(wanted_indices) > 0){
if (where == "session"){

message("Warning: the session option is currently not working well due to issues in ckanr")

for (resource in wanted_indices){
resource_tmp <- resources[[resource]]

ckanr::fetch(resource_tmp$url, store = "session")
write_import_messgae(resource_tmp)
ckanr::ckan_fetch(resource_tmp$url, store = "session")
write_import_message(resource_tmp)
}

} else if (where != "session"){
Expand All @@ -80,8 +83,8 @@ govcan_dl_resources.ckan_resource_stack <- function(resources,
resource_name <- get_resource_name(resource_tmp)
path <- create_storing_path(where, resource_name)

ckanr::fetch(resource_tmp$url, store = "disk", path = path)
write_dl_messgae(resource_tmp, path)
ckanr::ckan_fetch(resource_tmp$url, store = "disk", path = path)
write_dl_message(resource_tmp, path)

}
}
Expand Down Expand Up @@ -112,11 +115,11 @@ create_storing_path <- function(where, resource_name){
path
}

write_import_messgae <- function(resource_tmp){
write_import_message <- function(resource_tmp){
cat("Dataset ", resource_tmp$name, " imported successfully to session")
}

write_dl_messgae <- function(resource_tmp, path){
write_dl_message <- function(resource_tmp, path){
cat(" ---------------------------------------------------------------- \n")
cat("",resource_tmp$format, "file named", resource_tmp$name, "downloaded successfully \n")
cat(" path to file is:", path, "\n")
Expand Down
2 changes: 1 addition & 1 deletion R/govcan_get_record.R
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
#' @param ... More arguments to be passed on to ckanr::package_show()
#'
#' @return If only_resources is TRUE, will return only the list of data files (resources)
#' associated with the record querried else it will return all the output of the CKAN query.
#' associated with the record queried else it will return all the output of the CKAN query.
#' If format_resources is TRUE, the resources are formatted to a tibble.
#'
#' @export
Expand Down
6 changes: 3 additions & 3 deletions R/govcan_get_resources.R
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
#' @description Display resources attached to a specific record or (i.e. a CKAN package)
#' or to a stack of packages
#'
#' @param x An object of the class ckan_package_stack or ckan_package, or an id of
#' a specific record or (i.e. a CKAN package)
#' @param x An object of the class ckan_package_stack or ckan_package, or an id
#' of a specific record or (i.e. a CKAN package).
#'
#' @return A object of class ckan_resource_stack or list of ckan_resource_stack objetcs
#'
Expand All @@ -30,6 +30,6 @@ govcan_get_resources.ckan_package <- function(x){
govcan_get_resources.character <- function(x){
resource_list <- govcan_get_record(record_id = x, only_resources = TRUE,
format_resources = FALSE)
resource_stack <- map(resource_list, ckanr::as.ckan_resource)
resource_stack <- purrr::map(resource_list, ckanr::as.ckan_resource)
new_ckan_resource_stack(resource_stack)
}
6 changes: 3 additions & 3 deletions R/govcan_search.R
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
#' @param only_results (logical) Whether the function should return only the results
#' without the query metadata (default is TRUE)
#' @param format_results (logical) Whether the function should return a formatted output
#' of the results as a tibble or an unformatted version under the form of a list of
#' of the results as a tibble or an unformatted version under the form of a list of
#' CKAN packages (default is FALSE)
#' @param ... More arguments to be passed on to ckanr::package_search()
#'
Expand All @@ -30,8 +30,8 @@ govcan_search <- function(keywords,
paste(keywords, collapse = ", "))

# Collate all keywords
keywords_collated <- paste0(keywords,"+", collapse = "")
keywords_collated <- substr(keywords_collated,1, nchar(keywords_collated)-1)
keywords_collated <- paste0(keywords, "+", collapse = "")
keywords_collated <- substr(keywords_collated, 1, nchar(keywords_collated)-1)

# Perform query
if (format_results == TRUE) {
Expand Down
6 changes: 4 additions & 2 deletions R/govcan_setup.R
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
#' Set or reset the ckanr URL to the Open Canada portal URl "https://open.canada.ca/data/en"
#' Set or reset the ckanr URL to the Open Canada portal URL "https://open.canada.ca/data/en"
#'
#' @description Set or reset the ckanr URL to the Open Canada portal URl
#' @description Set or reset the ckanr URL to the Open Canada portal URL
#' "https://open.canada.ca/data/en"
#'
#' @param url Open Canada portal URL.
#'
#' @export
govcan_setup <- function(url = "https://open.canada.ca/data/en"){
message(paste0("ckanr url set to ", url))
Expand Down
29 changes: 17 additions & 12 deletions README.Rmd
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
---
output: github_document
output:
md_document:
variant: gfm
---

```{r setup, include=FALSE}
Expand All @@ -12,26 +14,26 @@ knitr::opts_chunk$set(echo = TRUE)
[![License: GPL v3](https://img.shields.io/badge/License-GPL%20v3-blue.svg)](http://www.gnu.org/licenses/gpl-3.0)
[![Travis-CI Build Status](https://travis-ci.org/VLucet/rgovcan.svg?branch=master)](https://travis-ci.org/vlucet/rgovcan)

A R package to interact with the Open Canada API, to search and download datasets. It is our hope that we will be able to bring this package up to the standard of a `ropensci` packages (see this issue on `ropensci/wishlist` https://github.com/ropensci/wishlist/issues/27).
A R package to interact with the Open Canada API, to search and download datasets. It is our hope that we will be able to bring this package up to the standard of a `ropensci` packages (see this issue on `ropensci/wishlist` https://github.com/ropensci/wishlist/issues/27).

This package makes extensive use of `ckanr` to access the canadian government's CKAN REST API.
This package makes extensive use of `ckanr` to access the Canadian government's CKAN REST API.

The code is under GPL-3 license.
The code is under GPL-3 license.
All the data is under Open Government License (http://open.canada.ca/en/open-government-licence-canada).

Hex Logo done with `hexSticker`: https://github.com/GuangchuangYu/hexSticker

## Installation

Untill release to CRAN, you will need to use `devtools` to install from source.
Until release to CRAN, you will need to use `devtools` to install from source.

```r
devtools::install_github("vlucet/rgovcan")
```

## Usage

1. First, load the package. The default `ckanr` url will be set to the Open Canada Portal.
1. First, load the package. The default `ckanr` url will be set to the Open Canada Portal.
```{r}
library("rgovcan")
```
Expand All @@ -40,16 +42,16 @@ If you happen to change the default url, you can reset it back to the default wi
govcan_setup()
```

2. A typical workflow with `rgovcan` can start with running `govcan_search` on a given set of keywords. This yiels a `stack` of `ckan_packages` (object of class `ckan_package_stack`).
2. A typical workflow with `rgovcan` can start with running `govcan_search` on a given set of keywords. This yields a `stack` of `ckan_packages` (object of class `ckan_package_stack`).
```{r}
?govcan_search
dfo_search <- govcan_search(keywords = c("dfo"), records = 10)
dfo_search # outputs a `ckan_package_stack`
```
see `?govcan_search` for further details.

3. Another possibility is to start with a package id corresponding to an actual record and retrieve a `ckan_package`.
```{r}
id <- "7ac5fe02-308d-4fff-b805-80194f8ddeb4" # Package ID
id <- "7ac5fe02-308d-4fff-b805-80194f8ddeb4" # Package ID
id_search <- govcan_get_record(record_id = id)
id_search # outputs a `ckan_package`
```
Expand All @@ -65,10 +67,13 @@ dfo_resources <- govcan_get_resources(dfo_search)
dfo_resources # outputs a list of `resource_stack`s
```

5. Finally, you can download the resources with `govcan_dl_resources`. These can either be stored to a certain directory or load into session (* this option might fail due to current issues with `ckanr::fetch`).
5. Finally, you can download the resources with `govcan_dl_resources`. These can either be stored to a certain directory or load into session (* this option might fail due to current issues with `ckanr::ckan_fetch`).

```{r}
?govcan_dl_resources
path <- paste0(getwd(),"/data/")
path <- "tmp/data/"
dir.create(path, recursive = TRUE)
govcan_dl_resources(id_resources, file_formats = c("JSON", "CSV", "SHP"), where = path)
# govcan_dl_resources(res_pel, file_formats = c("JSON", "CSV", "SHP"), where = "session")
```

see `?govcan_dl_resources` for further details.
Loading