datapackager.yml file.library(DataPackageR)
@@ -353,11 +359,14 @@ Set up a new data package.
force = TRUE,
code_files = processing_code,
r_object_names = "cars_over_20",
- path = tempdir()
- )
-Adding DataVersion string to DESCRIPTION
-Creating data and data-raw directories
-configuring yaml fileThis has created a datapackage source tree named “mtcars20” (in a temporary directory). For a real use case you would pick a path on your filesystem where you could then initialize a new github repository for the package.
The two main pieces of information in the configuration are a list of the files to be processed and the data sets the package will store.
This example packages an R data set named cars_over_20 (the name was passed in to datapackage_skeleton()). It is created by the subsetCars.Rmd file.
The objects must be listed in the yaml configuration file. datapackage_skeleton() ensures this is done for you automatically.
Raw data (provided the size is not prohibitive) can be placed in inst/extdata.
The datapackage_skeleton() API has the raw_data_dir argument, which will copy the contents of raw_data_dir (and its subdirectories) into inst/extdata automatically.
In this example we are reading the mtcars data set that is already in memory, rather than from the file system.
As stated in the README, in order for your processing scripts to be portable, you should not use abosolute paths to files. DataPackageR provides an API to point to the data package root directory and the inst/extdata and data subdirectories. These are useful for constructing portable paths in your code to read files from these locations.
For example: to construct a path to a file named “mydata.csv” located in inst/extdata in your data package source tree:
DataPackageR::project_extdata_path("mydata.csv") in your R or Rmd file. This would return: e.g., /var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T//RtmpilrC9a/mtcars20/inst/extdata/mydata.csvDataPackageR::project_extdata_path("mydata.csv") in your R or Rmd file. This would return: e.g., /var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T//RtmpO9s7M4/mtcars20/inst/extdata/mydata.csvSimilarly:
DataPackageR::project_path() constructs a path to the data package root directory. (e.g., /var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T//RtmpilrC9a/mtcars20)DataPackageR::project_data_path() constructs a path to the data package data subdirectory. (e.g., /var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T//RtmpilrC9a/mtcars20/data)DataPackageR::project_path() constructs a path to the data package root directory. (e.g., /var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T//RtmpO9s7M4/mtcars20)DataPackageR::project_data_path() constructs a path to the data package data subdirectory. (e.g., /var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T//RtmpO9s7M4/mtcars20/data)Raw data sets that are stored externally (outside the data package source tree) can be constructed relative to the project_path().
If your processing scripts are Rmd files, the usual yaml header for rmarkdown documents should be present.
+If you have Rmd files, you can still include a yaml header, but it should be commented with #' and it should be at the top of your R file. For example, a test R file in the DataPackageR package looks as follows:
#'---
+#'title: Sample report from R script
+#'author: Greg Finak
+#'date: August 1, 2018
+#'---
+data <- runif(100)This will be converted to an Rmd file with a proper yaml header, which will then be turned into a vignette and properly indexed in the built package.
Once the skeleton framework is set up,
-# Run the preprocessing code to build cars_over_20
-# and reproducibly enclose it in a package.
-DataPackageR:::package_build(file.path(tempdir(),"mtcars20"))
-INFO [2018-07-31 11:26:17] Logging to /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpilrC9a/mtcars20/inst/extdata/Logfiles/processing.log
-INFO [2018-07-31 11:26:17] Processing data
-INFO [2018-07-31 11:26:17] Reading yaml configuration
-INFO [2018-07-31 11:26:17] Found /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpilrC9a/mtcars20/data-raw/subsetCars.Rmd
-INFO [2018-07-31 11:26:17] Processing 1 of 1: /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpilrC9a/mtcars20/data-raw/subsetCars.Rmd
-processing file: subsetCars.Rmd
-output file: subsetCars.knit.md
-/usr/local/bin/pandoc +RTS -K512m -RTS subsetCars.utf8.md --to html4 --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash+smart --output /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpilrC9a/mtcars20/inst/extdata/Logfiles/subsetCars.html --email-obfuscation none --self-contained --standalone --section-divs --template /Library/Frameworks/R.framework/Versions/3.5/Resources/library/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable 'theme:bootstrap' --include-in-header /var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T//RtmpilrC9a/rmarkdown-str1228b57905547.html --mathjax --variable 'mathjax-url:https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML'
-
-Output created: /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpilrC9a/mtcars20/inst/extdata/Logfiles/subsetCars.html
-INFO [2018-07-31 11:26:17] 1 required data objects created by subsetCars.Rmd
-INFO [2018-07-31 11:26:17] Saving to data
-INFO [2018-07-31 11:26:17] Copied documentation to /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpilrC9a/mtcars20/R/mtcars20.R
-
-✔ Creating 'vignettes/'
-✔ Creating 'inst/doc/'
-INFO [2018-07-31 11:26:17] Done
-INFO [2018-07-31 11:26:17] DataPackageR succeeded
-INFO [2018-07-31 11:26:17] Building documentation
-First time using roxygen2. Upgrading automatically...
-Updating roxygen version in /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpilrC9a/mtcars20/DESCRIPTION
-Writing NAMESPACE
-Loading mtcars20
-Writing mtcars20.Rd
-Writing cars_over_20.Rd
-INFO [2018-07-31 11:26:17] Building package
-'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file \
- --no-environ --no-save --no-restore --quiet CMD build \
- '/private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpilrC9a/mtcars20' \
- --no-resave-data --no-manual --no-build-vignettes
-
-Reloading installed mtcars20
-[1] "/private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpilrC9a/mtcars20_1.0.tar.gz"# Run the preprocessing code to build cars_over_20
+# and reproducibly enclose it in a package.
+DataPackageR:::package_build(file.path(tempdir(),"mtcars20"))
+INFO [2018-08-01 11:21:24] Logging to /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpO9s7M4/mtcars20/inst/extdata/Logfiles/processing.log
+INFO [2018-08-01 11:21:24] Processing data
+INFO [2018-08-01 11:21:24] Reading yaml configuration
+INFO [2018-08-01 11:21:24] Found /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpO9s7M4/mtcars20/data-raw/subsetCars.Rmd
+INFO [2018-08-01 11:21:24] Processing 1 of 1: /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpO9s7M4/mtcars20/data-raw/subsetCars.Rmd
+processing file: subsetCars.Rmd
+output file: subsetCars.knit.md
+/usr/local/bin/pandoc +RTS -K512m -RTS subsetCars.utf8.md --to html4 --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash+smart --output /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpO9s7M4/mtcars20/inst/extdata/Logfiles/subsetCars.html --email-obfuscation none --self-contained --standalone --section-divs --template /Library/Frameworks/R.framework/Versions/3.5/Resources/library/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable 'theme:bootstrap' --include-in-header /var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T//RtmpO9s7M4/rmarkdown-str1527273c3b6f8.html --mathjax --variable 'mathjax-url:https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML'
+
+Output created: /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpO9s7M4/mtcars20/inst/extdata/Logfiles/subsetCars.html
+INFO [2018-08-01 11:21:24] 1 required data objects created by subsetCars.Rmd
+INFO [2018-08-01 11:21:24] NEWS.md file not found, creating!
+Enter a text description of the changes for the NEWS file.
+INFO [2018-08-01 11:21:30] Saving to data
+INFO [2018-08-01 11:21:30] Copied documentation to /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpO9s7M4/mtcars20/R/mtcars20.R
+
+✔ Creating 'vignettes/'
+✔ Creating 'inst/doc/'
+INFO [2018-08-01 11:21:30] Done
+INFO [2018-08-01 11:21:30] DataPackageR succeeded
+INFO [2018-08-01 11:21:30] Building documentation
+First time using roxygen2. Upgrading automatically...
+Updating roxygen version in /private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpO9s7M4/mtcars20/DESCRIPTION
+Writing NAMESPACE
+Loading mtcars20
+Writing mtcars20.Rd
+Writing cars_over_20.Rd
+INFO [2018-08-01 11:21:31] Building package
+'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file \
+ --no-environ --no-save --no-restore --quiet CMD build \
+ '/private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpO9s7M4/mtcars20' \
+ --no-resave-data --no-manual --no-build-vignettes
+
+Reloading installed mtcars20
+Next Steps
+1. Update your package documentation.
+ - Edit the documentation.R file in the package source data-raw subdirectory and update the roxygen markup.
+ - Rebuild the package documentation with document() .
+2. Add your package to source control.
+ - Call git init . in the package source root directory.
+ - git add the package files.
+ - git commit your new package.
+ - Set up a github repository for your pacakge.
+ - Add the github repository as a remote of your local package repository.
+ - git push your local repository to gitub.
+[1] "/private/var/folders/jh/x0h3v3pd4dd497g3gtzsm8500000gn/T/RtmpO9s7M4/mtcars20_1.0.tar.gz"When you build a package in interactive mode, you will be prompted to input text describing the changes to your data package (one line).
+These will appear in the NEWS.md file in the following format:
+DataVersion: xx.yy.zz
+========
+A description of your changes to the package
+
+[The rest of the file]If the processing script is time consuming or the data set is particularly large, then R CMD build would run the code each time the package is installed. In such cases, raw data may not be available, or the environment to do the data processing may not be set up for each user of the data. DataPackageR decouples data processing from package building/installation for data consumers.
After the first build, the R directory contains mtcars.R that has autogenerated roxygen2 markup documentation for the data package and for the packaged data cars_over20.
The processed Rd files can be found in man.
The autogenerated documentation source is in the documentation.R file in data-raw.
You should update this file to properly document your objects. Then rebuild the documentation:
+document(file.path(tempdir(),"mtcars20"))
+INFO [2018-08-01 11:21:32] Rebuilding data package documentation.
+Loading mtcars20
+[1] TRUEThis is done without reprocessing the data.
You should update the documentation in R/mtcars.R, then call package_build() again.
When the package is installed, these will be accessible via the vignette() API.
The vignette will detail the processing performed by the subsetCars.Rmd processing script.
The data set documentation will be accessible via ?cars_over_20, and the data sets via data().
# Let's use the package we just created.
-install.packages(file.path(tempdir(),"mtcars20_1.0.tar.gz"), type = "source", repos = NULL)
-if(!"package:mtcars20"%in%search())
- attachNamespace('mtcars20') #use library() in your code
-data("cars_over_20") # load the data
-
-cars_over_20 # now we can use it.
- speed dist
-44 22 66
-45 23 54
-46 24 70
-47 24 92
-48 24 93
-49 24 120
-50 25 85
-?cars_over_20 # See the documentation you wrote in data-raw/documentation.R.
-
-vignettes <- vignette(package = "mtcars20")
-vignettes$results
- Package
-Topic "mtcars20"
- LibPath
-Topic "/Library/Frameworks/R.framework/Versions/3.5/Resources/library"
- Item Title
-Topic "subsetCars" "A Test Document for DataPackageR (source, html)"# Let's use the package we just created.
+install.packages(file.path(tempdir(),"mtcars20_1.0.tar.gz"), type = "source", repos = NULL)
+if(!"package:mtcars20"%in%search())
+ attachNamespace('mtcars20') #use library() in your code
+data("cars_over_20") # load the data
+
+cars_over_20 # now we can use it.
+ speed dist
+44 22 66
+45 23 54
+46 24 70
+47 24 92
+48 24 93
+49 24 120
+50 25 85
+?cars_over_20 # See the documentation you wrote in data-raw/documentation.R.
+
+vignettes <- vignette(package = "mtcars20")
+vignettes$results
+ Package
+Topic "mtcars20"
+ LibPath
+Topic "/Library/Frameworks/R.framework/Versions/3.5/Resources/library"
+ Item Title
+Topic "subsetCars" "A Test Document for DataPackageR (source, html)"Your downstream data analysis can depend on a specific version of the data in your data package by tesing the DataVersion string in the DESCRIPTION file.
We provide an API for this:
-# We can easily check the version of the data
-DataPackageR::data_version("mtcars20")
-[1] '0.1.0'
-
-# You can use an assert to check the data version in reports and
-# analyses that use the packaged data.
-assert_data_version(data_package_name = "mtcars20",
- version_string = "0.1.0",
- acceptable = "equal") #If this fails, execution stops
- #and provides an informative error.# We can easily check the version of the data
+DataPackageR::data_version("mtcars20")
+[1] '0.1.0'
+
+# You can use an assert to check the data version in reports and
+# analyses that use the packaged data.
+assert_data_version(data_package_name = "mtcars20",
+ version_string = "0.1.0",
+ acceptable = "equal") #If this fails, execution stops
+ #and provides an informative error.You can migrate an old package by constructing such a config file using the construct_yml_config() API.
# assume I have file1.Rmd and file2.R located in /data-raw,
-# and these create 'object1' and 'object2' respectively.
-configuration:
- files:
- file1.Rmd:
- enabled: yes
- file2.R:
- enabled: yes
- objects:
- - object1
- - object2
- render_root:
- tmp: '841701'# assume I have file1.Rmd and file2.R located in /data-raw,
+# and these create 'object1' and 'object2' respectively.
+configuration:
+ files:
+ file1.Rmd:
+ enabled: yes
+ file2.R:
+ enabled: yes
+ objects:
+ - object1
+ - object2
+ render_root:
+ tmp: '448786'config is a newly constructed yaml configuration object. It can be written to the package directory:
path_to_package <- tempdir() #e.g., if tempdir() was the root of our package.
-yml_write(config, path = path_to_package)path_to_package <- tempdir() #e.g., if tempdir() was the root of our package.
+yml_write(config, path = path_to_package)Now the package at path_to_package will build with version 1.12.0 or greater.
For example:
# read 'myfile.csv' from inst/extdata relative to data-raw where the Rmd is rendered.
-read.csv(file.path("../inst/extdata","myfile.csv"))# read 'myfile.csv' from inst/extdata relative to data-raw where the Rmd is rendered.
+read.csv(file.path("../inst/extdata","myfile.csv"))Now Rmd and R scripts are processed in render_root defined in the yaml config.
To read a raw data set we can get the path to the package source directory using an API call:
# DataPackageR::project_extdata_path() returns the path to the data package inst/extdata subdirectory directory.
-# DataPackageR::project_path() returns the path to the data package root directory.
-# DataPackageR::project_data_path() returns the path to the data package data subdirectory directory.
-read.csv(
- DataPackageR::project_extdata_path("myfile.csv")
- )# DataPackageR::project_extdata_path() returns the path to the data package inst/extdata subdirectory directory.
+# DataPackageR::project_path() returns the path to the data package root directory.
+# DataPackageR::project_data_path() returns the path to the data package data subdirectory directory.
+read.csv(
+ DataPackageR::project_extdata_path("myfile.csv")
+ )We can also perform partial builds of a subset of files in a package by toggling the enabled key in the config file.
This can be done with the following API:
-config <- yml_disable_compile(config,filenames = "file2.R")
-yml_write(config, path = path_to_package) # write modified yml to the package.
-configuration:
- files:
- file1.Rmd:
- enabled: yes
- file2.R:
- enabled: no
- objects:
- - object1
- - object2
- render_root:
- tmp: '841701'config <- yml_disable_compile(config,filenames = "file2.R")
+yml_write(config, path = path_to_package) # write modified yml to the package.
+configuration:
+ files:
+ file1.Rmd:
+ enabled: yes
+ file2.R:
+ enabled: no
+ objects:
+ - object1
+ - object2
+ render_root:
+ tmp: '448786'Note that the modified configuration needs to be written back to the package source directory in order for the changes to take effect.
The consequence of toggling a file to enable: no is that it will be skipped when the package is rebuilt, but the data will still be retained in the package, and the documentation will not be altered.
This is useful in situations where we have multiple data sets, and want to re-run one script to update a specific data set, but not the other scripts because they may be too time consuming, for example.
@@ -692,7 +745,7 @@