From dc79ca9bb04631491ce64e3f7f8bda16fe4bedce Mon Sep 17 00:00:00 2001 From: Ronny Bergmann Date: Thu, 1 Jun 2023 20:44:43 +0200 Subject: [PATCH 1/4] Propose a detailed example for Quarto tutorials to markdown to be used in Documenter. --- examples/README.md | 5 +- examples/example-09-render-to-documenter.md | 168 ++++++++++++++++++++ 2 files changed, 171 insertions(+), 2 deletions(-) create mode 100644 examples/example-09-render-to-documenter.md diff --git a/examples/README.md b/examples/README.md index 9b7bf83..84ef734 100644 --- a/examples/README.md +++ b/examples/README.md @@ -8,11 +8,12 @@ * [Publishing a single format, publishing without rendering](./example-06-no-render.md) * [Publishing a single format](./example-07-publish-single-format.md) * [Publishing to other services](./example-08-publish-to-others-services.md) +* [Render with Quarto to Publish with Documenter.jl](./example-09-render-to-documenter.md) ## Repositories using Quarto actions -- [Earthdata Cloud Cookbook](https://nasa-openscapes.github.io/earthdata-cloud-cookbook/) ([source](https://github.com/NASA-Openscapes/earthdata-cloud-cookbook), [workflow file](https://github.com/NASA-Openscapes/earthdata-cloud-cookbook/blob/main/.github/workflows/quarto-publish.yml)) This book contains `.md` and `.ipynb` files, and is built with Quarto and Python in GHA, and deployed to Github Pages. +- [Earthdata Cloud Cookbook](https://nasa-openscapes.github.io/earthdata-cloud-cookbook/) ([source](https://github.com/NASA-Openscapes/earthdata-cloud-cookbook), [workflow file](https://github.com/NASA-Openscapes/earthdata-cloud-cookbook/blob/main/.github/workflows/quarto-publish.yml)) This book contains `.md` and `.ipynb` files, and is built with Quarto and Python in GHA, and deployed to Github Pages. - [R Manuals Quarto website](https://rstudio.github.io/r-manuals/) ([source](https://github.com/rstudio/r-manuals), [workflow file](https://github.com/rstudio/r-manuals/blob/main/.github/workflows/build-website.yaml)) This projects uses a workflow to build several books with R and Quarto and organizes them in a website deployed to Github pages. -- [Pathology Atlas](https://www.patolojiatlasi.com/EN) ([source](https://github.com/patolojiatlasi/patolojiatlasi.github.io), [workflow file](https://github.com/patolojiatlasi/patolojiatlasi.github.io/blob/main/.github/workflows/Quarto-Render-Bilingual-Book-Push-Other-Repos-GitLab.yml)) This multilingual website is rendered in two versions and deployed using Github Actions. +- [Pathology Atlas](https://www.patolojiatlasi.com/EN) ([source](https://github.com/patolojiatlasi/patolojiatlasi.github.io), [workflow file](https://github.com/patolojiatlasi/patolojiatlasi.github.io/blob/main/.github/workflows/Quarto-Render-Bilingual-Book-Push-Other-Repos-GitLab.yml)) This multilingual website is rendered in two versions and deployed using Github Actions. diff --git a/examples/example-09-render-to-documenter.md b/examples/example-09-render-to-documenter.md new file mode 100644 index 0000000..70baa39 --- /dev/null +++ b/examples/example-09-render-to-documenter.md @@ -0,0 +1,168 @@ +# Quarto Actions: Rendering Julia tutorials to be Published with Documenter.jl + +Quarto Markdown files can be used very well to write tutorials that accompany a documentation +of a Julia package. +The advantage is, that the examples in the tutorials can be run (and cached), and the resulting +markdown files can directly be saved and included within the documentation which is +afterwards rendered using [Documenter.jl](https://documenter.juliadocs.org/stable/). + +A challenge here are the dependencies, since the CI not only needs Julia and Quarto, but also +Python, Jupyter – and the Julia package. The dependencies to Python and Jupyter can for example +be handled with [CondaPkg.jl](https://github.com/cjdoris/CondaPkg.jl), since that allows a +versioning similar to the `Project.toml` in Julia packages. + +## Folder structure + +In this example we assume, that + +* the documentation is provided in `docs/src` of the repository +* the tutorials are stored within `tutorials/` as well as for example the corresponding `_quarto_yml` +* the folder the results are stored is in this example `docs/src/tutorials` + +Both these folders get their own `Project.toml` to have dependencies for the documentation +different from the dependencies for the tutorials. [Ijulia.jl](https://github.com/JuliaLang/IJulia.jl) should be added to both +the tutorial environment, CondaPkg to the documentation. + +## Quarto Setup + +the `tutorials/_quarto.yml` should use common matk as rendering, here the output +folder is set to fit the caching below. + +```yml +project: + title: "Manopt.jl Tutorials" + output-dir: ../docs/src/tutorials + render: + - "*.qmd" + +execute: + freeze: auto + +format: + commonmark: + variant: -raw_html+tex_math_dollars + wrap: preserve +``` + +## CondaPkg dependencies + +We first specify the versions for Python and Jupyter in `docs/CondaPkg.toml` + +```toml +[deps] +jupyter = "" +python = "3.11" +``` + +## Documenter + +We install all necessary python dependencies when the `docs/make.jl` file, that should be +executable, gets the command line argument `--quarto` + +A minimal example is for this file is + +```julia +#!/usr/bin/env julia +# + +# (a) Activate the docs/ environment +if Base.active_project() != joinpath(@__DIR__, "Project.toml") + using Pkg + Pkg.activate(@__DIR__) + Pkg.develop(PackageSpec(; path=(@__DIR__) * "/../")) #put the main package in / into development mode + Pkg.resolve() + Pkg.instantiate() + if "--quarto" ∈ ARGS # if we render, + Pkg.build("IJulia") # activate/add the right kernel + end +end + +# (b) Issue the actual rendering +if "--quarto" ∈ ARGS + using CondaPkg #Loads all Python dependencies + CondaPkg.withenv() do # activates the Conda environment + @info "Rendering Quarto" + tutorials_folder = (@__DIR__) * "/../tutorials" + # instantiate the tutorials environment if necessary + Pkg.activate(tutorials_folder) + Pkg.resolve() #resolve / setup the tutorials environment + Pkg.instantiate() + Pkg.activate(@__DIR__) # but return to the docs one before + run(`quarto render $(tutorials_folder)`) # render + return nothing + end +end + +# (c) load necessary packages for the documentation +using Documenter#, and all others you need +# (d) call makedocs( to render the docs... +``` + +For step d. the [makedocs](https://documenter.juliadocs.org/stable/man/guide/#Building-an-Empty-Document) documentation is recommended + +## Documenter CI + +For the CI there are four recommended caches + +The final `.github/workflow/documenter.yml` might look as follows + +```yml +name: Documenter +on: + push: + branches: [master] + tags: [v*] + pull_request: + +jobs: + docs: + name: Documentation + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: quarto-dev/quarto-actions/setup@v2 + with: + version: 1.3.353 + - uses: julia-actions/setup-julia@latest + with: + version: 1.9 + - name: "Documenter rendering (including Quarto)" + run: "docs/make.jl --quarto" + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }} +``` + +It is probably recommended to cache the following + +* Julia – maybe even includeing the precompiled packages +* The CondaPkg installed dependencies +* The Quarto freeze folder +* The resulting Quarto markdown files + +This can be achieved with the following steps added after setting up Julia + +```yml + - uses: julia-actions/cache@v1 + with: + cache-compiled: "true" + - uses: actions/cache@v3 + with: + path: tutorials/_freeze + key: ${{ runner.os }}-quarto-freeze + - uses: actions/cache@v3 + with: + path: docs/src/tutorials + key: ${{ runner.os }}-documenter-tutorials + - uses: actions/cache@v3 + with: + path: docs/.CondaPkg + key: ${{ runner.os }}-condapkg +``` + +## Example + +Two examples using this scheme are + +* [Manopt.jl](https://github.com/JuliaManifolds/Manopt.jl) +* [Manifolds.jl](https://github.com/JuliaManifolds/Manifolds.jl) From 92331bb4b8080691677fa8c26aff597703103166 Mon Sep 17 00:00:00 2001 From: Ronny Bergmann Date: Thu, 1 Jun 2023 20:48:02 +0200 Subject: [PATCH 2/4] fix a few typos. --- examples/example-09-render-to-documenter.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/examples/example-09-render-to-documenter.md b/examples/example-09-render-to-documenter.md index 70baa39..9a35da1 100644 --- a/examples/example-09-render-to-documenter.md +++ b/examples/example-09-render-to-documenter.md @@ -25,7 +25,7 @@ the tutorial environment, CondaPkg to the documentation. ## Quarto Setup -the `tutorials/_quarto.yml` should use common matk as rendering, here the output +the `tutorials/_quarto.yml` should use common mark as rendering, here the output folder is set to fit the caching below. ```yml @@ -98,7 +98,7 @@ using Documenter#, and all others you need # (d) call makedocs( to render the docs... ``` -For step d. the [makedocs](https://documenter.juliadocs.org/stable/man/guide/#Building-an-Empty-Document) documentation is recommended +For step (d) the [makedocs](https://documenter.juliadocs.org/stable/man/guide/#Building-an-Empty-Document) documentation is recommended ## Documenter CI @@ -135,7 +135,7 @@ jobs: It is probably recommended to cache the following -* Julia – maybe even includeing the precompiled packages +* Julia – maybe even including the precompiled packages * The CondaPkg installed dependencies * The Quarto freeze folder * The resulting Quarto markdown files From 6746ea43c5a8eac35cb22bdd45fb4cacf442f822 Mon Sep 17 00:00:00 2001 From: Ronny Bergmann Date: Mon, 5 Jun 2023 09:49:36 +0200 Subject: [PATCH 3/4] In practice the Cache is sometimes not really useful and introduces errors. --- examples/example-09-render-to-documenter.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/examples/example-09-render-to-documenter.md b/examples/example-09-render-to-documenter.md index 9a35da1..683eec4 100644 --- a/examples/example-09-render-to-documenter.md +++ b/examples/example-09-render-to-documenter.md @@ -144,8 +144,6 @@ This can be achieved with the following steps added after setting up Julia ```yml - uses: julia-actions/cache@v1 - with: - cache-compiled: "true" - uses: actions/cache@v3 with: path: tutorials/_freeze From 1eb7e34b066e9ca426d56746c5d401b0df02351d Mon Sep 17 00:00:00 2001 From: Ronny Bergmann Date: Tue, 13 Jun 2023 16:14:43 +0200 Subject: [PATCH 4/4] Improve Caches. --- examples/example-09-render-to-documenter.md | 33 ++++++++++++++++----- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/examples/example-09-render-to-documenter.md b/examples/example-09-render-to-documenter.md index 683eec4..6a1074a 100644 --- a/examples/example-09-render-to-documenter.md +++ b/examples/example-09-render-to-documenter.md @@ -143,21 +143,40 @@ It is probably recommended to cache the following This can be achieved with the following steps added after setting up Julia ```yml - - uses: julia-actions/cache@v1 - - uses: actions/cache@v3 + - name: Julia Cache + uses: julia-actions/cache@v1 + - name: Cache Quarto + env: + cache-name: cache-quarto with: path: tutorials/_freeze - key: ${{ runner.os }}-quarto-freeze - - uses: actions/cache@v3 + key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('tutorials/*.qmd') }} + restore-keys: | + ${{ runner.os }}-${{ env.cache-name }}- + - name: Cache Documenter + id: cache-documenter + uses: actions/cache@v3 + env: + cache-name: cache-documenter with: path: docs/src/tutorials - key: ${{ runner.os }}-documenter-tutorials - - uses: actions/cache@v3 + key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('tutorials/*.qmd') }} + restore-keys: | + ${{ runner.os }}-${{ env.cache-name }}- + - name: Cache CondaPkg + id: cache-condaPkg + uses: actions/cache@v3 + env: + cache-name: cache-condapkg with: path: docs/.CondaPkg - key: ${{ runner.os }}-condapkg + key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('docs/CondaPkg.toml') }} + restore-keys: | + ${{ runner.os }}-${{ env.cache-name }}-``` ``` +This should be added before running the documenter call. Note that the restore keys even restore old files even in the case that the key (hash) changed. This way also only updated quarto notebooks are rerendered, but due to no exact match, the new ones are cached afterwards. + ## Example Two examples using this scheme are