19-appendix.Rmd

# (PART) Appendix {.unnumbered}

# Appendix A - Use Case: Building an App from Start to Finish {.unnumbered}

This chapter aims at exemplifying the workflow developed in this book using a "real-life" example.
In this appendix, we will be building a `{shiny}` application from start to finish.
We've chosen to build an application that doesn't rely on heavy computation/data analysis, so that we can focus on the engineering process, not on the internals of the analytic methodology, nor on spending time explaining the dataset.

## About the application {.unnumbered}

In this appendix, we will build a "minify" application, an application that takes a CSS, JavaScript, HTML or JSON input, and outputs a minified version of this file.
We will rely on the `{minifyr}` package to build this application.

Here is an example of what the specifications for this app could look like:

``` {.txt}
Hello! 

We want to build a small application that can 
minify CSS, JavaScript, HTML and JSON. 

In this app, user will be able either to paste 
the content or to upload a file. 

Once the content is pasted/upladed, they select 
the type, which is pre-selected based on the 
file extension. Then they click on a button, 
and the content is minified. 

They can then copy the output, or download it as a file. 

Cheers!
```

## Step 1: Design {.unnumbered}

### Deciphering the specifications {.unnumbered}

#### General observations {.unnumbered}

-   As this app is pretty straightforward, it would be better to handle everything in the same page, *i.e* everything should happen on the same page (no tab).

-   It would be a plus to have the "before minification"/"after minification" gain, so that the users have a better idea of the purpose of the application.

#### User experience considerations {.unnumbered}

-   We should provide a link to an explanation of minification.

-   The user might get different results based on the minifying algorithm they use, which can be surprising at first.
    The application should alert about this.

-   For long printed outputs, if we use `verbatimTextOuput`, we should be careful about the page width, as these elements will natively overflow on the x-axis of the page.
    This should be doable with the following CSS: `pre{ white-space: pre-wrap; word-break: keep-all; }`.

-   We should be careful about using semantic HTML for the inputs and outputs.

#### Technical points {.unnumbered}

-   As `{minifyr}` wraps a NodeJS module, we will need to install NodeJS when deploying.

-   To be sure that the process works, we should check the validity of the file extension from the UI and from the server side.

### Building a concept map {.unnumbered}

Figure \@ref(fig:19-appendix-1) is the concept map for this application, using Xmind.

(ref:conceptmapminifyr) A concept map for the minifying application.

```{r 19-appendix-1, echo=FALSE, fig.cap="(ref:conceptmapminifyr)", out.width="100%"}
knitr::include_graphics("img/minifyr-map.png")
```

### Asking questions {.unnumbered}

#### About the end users {.unnumbered}

-   *Who are the end users of your app?*

This application is mainly useful for web developers.

-   *Are they tech-literate?*

Yes.

-   *In which context will they be using your app?*

Notably at work, or while building pet projects.

-   *On what machines?*

Laptop/personal computer.
Small chance of using this on a smartphone.

-   *What browser version will they be using?*

Hard to say, but given that we aim for a public of community developers, probably modern browsers.

-   *Will they be using the app in their office, on their phone while driving a tractor, in a plant, or while wearing lab coats?*

Nothing of the like: they should be using this application while developing, so chances are they are using it at a desk.

### Building personas {.unnumbered}

Let's pick two random names for our personas, and two fake companies where they might be working.

```{r 19-appendix-2, eval = TRUE}
nms <- withr::with_seed(
  608, {
    charlatan::ch_name(2)
    
  }
)
nms

company <- withr::with_seed(
  608, {
    charlatan::ch_company(2)
  }
)
company
```

#### `r nms[1]`: `{shiny}` developer at `r company[1]` {.unnumbered}

`r nms[1]` is a `{shiny}` developer at `r company[1]`.
She's been learning R in graduate school while working on her master's degree in statistics.
When she started at `r company[1]`, she was mainly doing data analysis in Rmd, but has gradually switched to building `{shiny}` applications full-time.
She discovered minification while reading the "Engineering Production-Grade Shiny App" book, and now wants to add this to her `{shiny}` application.

#### `r nms[2]`: web developer and trainer at `r company[2]` {.unnumbered}

`r nms[2]` is a web developer at `r company[2]`.
He studied web development at the university, where he learned about minification.
He is now also in charge of training new recruits for the company where he works, and also gives some lectures at the university he attended.
Most of the minification he does is automated, but he is looking for a tool he can use during training and classes to explain how minification works.

This step is available at <https://github.com/ColinFay/minifying/tree/master/step-1-design>.

## Step 2: Prototyping {.unnumbered}

In this step, we will be building the back-end of the application on one side, and the UI on the other side.
Once we have the back-end settled and the UI defined, we will be working on making the two work with each other.

### Back-end in Rmd {.unnumbered}

Our back-end will be composed of two functions:

```{r 19-appendix-3, echo = FALSE, eval = TRUE}
library(minifyr)
minif <- ls("package:minifyr", pattern = "minifyr_js_")
minif <- paste0("`", minif, "()`")
minif <- knitr::combine_words(minif)
```

-   `guess_minifier`, which will take a function and return the available algorithms for that file: for example, if you have a JavaScript file, you'll be able to use the `r minif` functions. If the type is not guessed based on the extension, the function should fail gracefully, and not make `{shiny}` crash. We'll chose to return an empty string if this extension is not guessed.

```{r 19-appendix-4}
library(minifyr)
guess_minifier <- function(file){
  # We'll start by getting the file extension
  ext <- tools::file_ext(file)
  # Check that the extension is correct, if not, return early
  # It's important to do this kind of check also 
  # on the server side as HTML manual tempering 
  # would allow to also send other type of files
  if (
    ! ext %in% c("js", "css", "html", "json")
  ){
    # Return early
    return(list())
  }
  # We'll then retrieve the available 
  # pattern based on the extension
  patt <- switch(
    ext, 
    js = "minifyr_js_.+",
    html = "minifyr_html_.+",
    css = "minifyr_css_.+",
    json = "minifyr_json_.+"
  )
  # List all the available functions to minify the file
  list(
    file = file,
    ext = ext,
    # We return this pattern so that 
    # it will be used to update the selectInput that 
    # is used to select an algo
    pattern = patt, 
    functions = grep(
      patt, 
      names(
        loadNamespace("minifyr")
      ), 
      value = TRUE
    )
  )
  
}

# minifyr comes with a series of examples, 
# so we can use them as tests
guess_minifier(
  minifyr_example("css")
)[2:4]
guess_minifier(
  minifyr_example("js")
)[2:4]
guess_minifier(
  minifyr_example("html")
)[2:4]
guess_minifier(
  minifyr_example("json")
)[2:4]
# Try with a non valid extension
guess_minifier(
  "path/to/text.docx"
)
```

-   A `compress()` function, which takes three parameters: the file as `input`, the `algo` outputted from our last function, and the selection, which is the one selected by the user. The compressed file will be outputted to a tempfile.

```{r 19-appendix-5, eval = FALSE}
compress <- function(algo, selection){
  # Creating a tempfile using our algo object
  tps <- tempfile(fileext = sprintf(".%s", algo$ext))
  # Getting the function with the selection
  converter <- get(
    grep(selection, algo$functions, value = TRUE)
  )
  # Do the conversion
  converter(algo$file, tps)
  # Return the temp file
  return(tps)
}

algo <- guess_minifier(
  minifyr_example("js")
)

compress(
  algo = algo, 
  selection = "babel"
) 
```

```{r 19-appendix-1-bis, echo = FALSE}
compress <- function(algo, selection){
  # Creating a tempfile using our algo object
  tps <- tempfile(fileext = sprintf(".%s", algo$ext))
  # Getting the function with the selection
  converter <- get(
    grep(selection, algo$functions, value = TRUE)
  )
  # Do the conversion
  converter(algo$file, tps)
  # Return the temp file
  return(tps)
}

algo <- guess_minifier(
  minifyr_example("js")
)
```

-   Finally, a `compare()` function, that can compare the size of two files, so that we can measure the minification gain. This function will take two file paths.

```{r 19-appendix-6}
compare <- function(original, minified){
  # Get the file size of both
  original <- fs::file_info(original)$size
  minified <- fs::file_info(minified)$size
  return(original - minified)
}

```

So, for the complete process:

```{r 19-appendix-7}
algo <- guess_minifier(
  minifyr_example("js")
)

compressed <- compress(
  algo = algo, 
  selection = "babel"
)
compare(
  minifyr_example("js"), 
  compressed
)
```

Now, time to move this into a vignette!

### UI prototyping {.unnumbered}

Let's start by drawing a small mockup of our front-end using [Excalidraw](https://excalidraw.com/), as seen in Figure \@ref(fig:19-appendix-8).

(ref:excalidraw) A mockup for the UI of our application, made with Excalidraw <https://excalidraw.com/>.

```{r 19-appendix-8, echo=FALSE, fig.cap="(ref:excalidraw)", out.width="100%"}
knitr::include_graphics("img/excalidraw.png")
```

We would love this application to be "full screen", and to do that, we'll take inspiration from the [split-screen layout](https://www.w3schools.com/howto/howto_css_split_screen.asp) available at W3Schools.
To mock up the UI, we will also use the `{shinipsum}` package.

In this first step, we will start generating the module skeleton for the application.
Here, we will have a `left` module for the left part of the app, and a `right` module for the right part.
Each will receive their corresponding `class`, based on the CSS from W3.
Now that these two spots are available, we'll add the two modules, with some fake output, to simulate our application behavior.
The left side will be functional in the sense that uploading a file will randomly add algorithms to the `selectInput()`, and clicking on `Launch the minification` will regenerate a fake text.

Now, let's pick a soft palette of colors, using [coolors.co](https://coolors.co/palettes/), and a font family from [fonts.google.com](https://fonts.google.com/).
We went for:

-   One of the monochrome palettes from [coolors.co](https://coolors.co/fbfbf2-e5e6e4-cfd2cd-a6a2a2-847577).
-   The `Sora` font [fonts.google.com/specimen/Sora](https://fonts.google.com/specimen/Sora). There is not that much text displayed on the screen, so this font should work well.

We then used CSS to arrange our page: size, padding, alignment, colors, etc. 
If you want to know more about this file, it's located in the `inst/app/www` folder of the package.

This step is available at [github.com/ColinFay/minifying, folder step-2-prototype](https://github.com/ColinFay/minifying/tree/master/step-2-prototype).

## Step 3: Build {.unnumbered}

Now we've got the back-end in an Rmd, the front end working with `{shinipsum}`. 
Now is the time to make the two work together!

Here is the logic we will be adding to the application:

-   When a file is uploaded, we also check the format from the server-side: the UI restriction with the `accept` parameter of `fileInput()` will not be enough to stop users who **really** want to upload something else.

-   If the file comes with the right extension, we update the algorithm selection and read it inside the "Original content" block.

-   Once the user clicks on "Launch the minification", we create a temp file and minify the original file inside this temp file.

-   When the file is minified, we update the gain output to reflect how many bytes have been gained from the minification, and add the result of this minification to the "Minified content" block.

-   Finally, when the "Download the output" button is clicked, the minified file is downloaded.

During this process, we will migrate the functions from the Rmd to files inside the `R/` folder, use external dependencies, and document our business logic functions.
You can refer to the `dev/02_dev.R` file if you want to read the exact steps taken here.

This step is available at <https://github.com/ColinFay/minifying/tree/master/step-3-build>.

## Step 4: Strengthen {.unnumbered}

As of now, we have a working application.
Time to strengthen it!

Here are the few steps we will be working on:

-   Turning our business logic into an R6 class.
    This R6 class will generate an object at the very start of our app, and it will be passed into the modules.

-   As the minification process takes a couple of seconds, we will add a small progress bar so that the user knows something is happening.

-   As we will use R6, we will need to manually set the reactive context invalidation.
    To do so, we will use `triggers` from `{gargoyle}`.

-   Chances are, users will be testing several algorithms when using the application, and we don't want the minification process to happen another time when it is called on the same file and with the same algorithm.
    This is even even more important because the process involves calling an external NodeJS process.
    To prevent that, we will be caching the function that does the computation.

-   Create an unseen input that will upload data, so that we can build an interactivity test using `{crrry}`.
    This input will look like the following on the server:

```{r 19-appendix-9}
observeEvent( input$testingtrigger , {
  
  if (golem::app_dev()){
    file$original_file <- minifyr::minifyr_example(
      ext = input$testingtext
    )
    file$guess_minifier()
    file$type <- input$upload$type
    file$minified_file <- NULL
    file$original_name <- input$upload$name
    gargoyle::trigger("uploaded")
  }
  
})
```

We use this pattern so that we can combine it with a testing suite with `{crrry}`, using the following pattern:

```{r 19-appendix-10, eval = FALSE}
test <- crrry::CrrryProc$new(
  chrome_bin = pagedown::find_chrome(),
  # Process to launch locally
  fun = "golem::document_and_reload();run_app()",
  # Note that you will need httpuv >= 1.5.2 for randomPort
  chrome_port = httpuv::randomPort(), 
  headless = FALSE
)

test$wait_for_shiny_ready()
ext <- c("css", "js", "json", "html")
for (i in 1:length(ext)){
  # Set the extension value
  test$shiny_set_input("left_ui_1-testingtext", ext[i])
  # Trigger the file to be read
  test$shiny_set_input("left_ui_1-testingtrigger", i)
  # Launch the minification
  test$shiny_set_input("left_ui_1-launch", i)
}

test$stop()
```

It's safer to wrap these tests between `if(interactive())`, as running the checks outside of your current session might not launch the app correctly, and launching external processes (the one running the app with Chrome) might fail when run non-interactively.
And on top of that, running these inside your CI might cause some pain, and of course, it will not work on CRAN checks.

We'll also be building "standard" function checks, which you can find in the `test/` folder.

This step is available at [github.com/ColinFay/minifying, folder step-4-strengthen](https://github.com/ColinFay/minifying/tree/master/step-4-strengthen).

## Step 5: Deploy {.unnumbered}

As an example, we will deploy this app in three media: as a package, on RStudio Connect, and with Docker.

### Before deployment checklist {.unnumbered}

-   [x] *`devtools::check()`, run from the command line, return 0 errors, 0 warnings, 0 notes*

-   [x] *The current version number is valid, i.e* if the current app is an update, the version number has been bumped: it makes sense, before the first deployment, to keep a version number of `0.0.0.9000`, and increment this dev version whenever we implement changes or do test deployments.
    Because this is a "true" deployment, we bumped the version to `0.1.0`.

-   [x] *Everything is fully documented*: we have documented all the functions, even the internal, there is a Vignette that describes the business logic, and the README is filled.

-   [x] *Test coverage is good, i.e. you cover a sufficient amount of the codebase, and these tests cover the core/strategic algorithms*.

-   [x] *The person to call if something goes wrong is clear to everyone involved in the product.*.

-   [x] *The debugging process is clear to everyone involved in the project, including how to communicate bugs to the developer team, and how long it will take to get changes implemented*: this project will be made open source, so the bug will have to be listed on the Github repo.
    To help that, we added a link to the GitHub repository on the application.

-   [x] *(If relevant) The server it is deployed on has all the necessary software installed (Docker, Connect, `Shiny Server`, etc.) to make the application run*.

-   [x] *The server has all the system requirements needed (the system libraries), and if not, they are installed with the application (if it's dockerized)*: NodeJS will need to be installed on Docker and on the server running RStudio Connect.
    A check is also added on top of `run_app()` for the availability of NodeJS on the system, especially for people installing it as a package.
    This check will also check if `node-minify` has been installed, and if not, it will be installed.
    This check might take some time to run, but it will only be performed the first time the app is launched.

-   [x] *The application, if deployed on a server, will be deployed on a port which will be accessible by the users*: when building the Dockerfile using `{golem}`, the correct port is exposed (*i.e* the app will run on port 80, which is also made available).
    For the other medium, the port will be automatically chosen, either by `{shiny}` or by Connect.

-   [x] *(If relevant) The environment variables from the production server are managed inside the application*: not relevant.

-   [x] *(If relevant) The app is launched on the correct port, or at least this port can be configured via an environment variable*: not relevant.

-   [x] *(If relevant) The server where the app is deployed has access to the data sources (database, API...)*: not relevant.

-   [x] *If the app records data and there are backups for these data*: not relevant

### Deploy as a tar.gz {.unnumbered}

To share an application as a tar.gz, you can call `devtools::build()`, which will compile a `tar.gz` file inside the parent folder of your current application.
You can then share this archive, and install it with `remotes::install_local("path/to/tar.gz")`.
Note that this can also be done with base R, but `{remotes}` offers a smarter way when it comes to managing the dependencies of your archived package.

This `tar.gz` can also be sent to a package repository; be it the CRAN or any other package manager you might have in your company.

### Deploy on RStudio Connect {.unnumbered}

Once we are sure that the server running Connect has NodeJS installed, and that we have installed the minify module with `minifyr::minifyr_npm_install()`, we can create the app.R using `golem::add_rstudioconnect_file()`, and then push to the Connect server.

### Deploy with Docker {.unnumbered}

To create the `Dockerfile`, we'll start by launching `golem::add_dockerfile()`.
This function will compute the system requirements,[^appendix-1] and create a generic `Dockerfile` for your application.
Once this is done, we will create/update the `.dockerignore` file at the root of the package, so that unwanted files are not bundled with our Docker image.

[^appendix-1]: Note that at the time of writing these lines, there is also an issue with the dependencies collected by the `sysreq` API, leading to an issue when attempting to compile the Dockerfile.
    Removing the installation of `libgit2-dev` solved the issue.

Inside our `Dockerfile`, we will also change the default repo to use <https://packagemanager.rstudio.com/all/latest>, which proposes precompiled packages for our system, making the installation faster.
We will also add an installation of NodeJS, which is needed by our application..

Then, we can go to our terminal and compile the image!

``` {.bash}
docker build -t minifying . 
```

Now we've got a working image!
We can try it with:

``` {.bash}
docker run -p 2811:80 minifying
```

This step is available at [github.com/ColinFay/minifying, folder step-5-deploy](https://github.com/ColinFay/minifying/tree/master/step-5-deploy).