How to use a custom pandoc output writer?

[jdelker]

Description

I'm using quarto for project documentation, basically turning "quarto books" into PDF documents.
To render the PDF output to the desired style, I already use an extension supplying the required templates, etc. (veriy similar to PrettyPDF).

Now, there are still some "flaws" in the generated PDF, which I want to eliminate and which go beyond the regular filter or template modifications. Actually, they originate from pandoc's latex output writer, regarding the rendering of some particular elements (like tables). For example this issue: jgm/pandoc#3929.

My requirements could possibly be resolved with a custom output writer I have to create for pandoc, which would render the "fixed" latex that is then used for the PDF generation process.

So my question is:
Is there any configurable way, to tell quarto to use a custom output writer to create the intermediate latex?

[jdelker]

... or alternatively:
I could also handle this from the pdf-engine side and alter the generated latex prior to the PDF generation.
Is there any documentation on how a custom pdf-engine could be used, to e.g. wrap xelatex and apply some latex modifications prior to rendering?

[mcanouil]

Use format: latex, and a post render script.
I believe this is the simplest method.

[mcanouil]

You can also write whatever Lua filter you want to "live" process the input.

[jdelker]

The whole idea about using Quarto is (at least for my use case) to have a integrated pipeline, which renders documents from QMD to PDF. In fact it shall replace a former pile of scripts and tools, which were all bundled together in a too-complicated-workflow.
Quarto is a very promising candidate to replace all that.

Using format: latex and handling everything after that would bring me (partly) back to the previous pile of scripts and tools - which I want to avoid. Tweaking the LaTeX output as required is ok, but I would also loose Quarto's integrated pdf-rendering - unless you know how to invoke that externally with a given latex file ;)

The Lua filters are great for modifying the data as it flows through the pipeline, but as I understood it, the last change to use them are before Pandoc is called to render the LaTeX. As the problem arises just there in the pandoc rendering, I have no chance to modify it before it is passed to the pdf-engine.

[cderv]

My requirements could possibly be resolved with a custom output writer I have to create for pandoc, which would render the "fixed" latex that is then used for the PDF generation process.

You can use Lua filter to tweak the output. You don't need a full custom writer for Pandoc as with Quarto you can insert Lua filter at different step of the rendering, one being the very last. See https://quarto.org/docs/prerelease/1.4/lua_changes.html#more-precise-targeting-of-ast-processing-phases

Doing something like

filters:
  - at: post-render
    path: filter.lua

would apply your Lua filter after all Quarto processing, before Pandoc writing to output. So if you want to write the LaTeX yourself from there instead of Pandoc this could be possible.

Is there any documentation on how a custom pdf-engine could be used, to e.g. wrap xelatex and apply some latex modifications prior to rendering?

Regarding PDF engine, by default Quarto uses its own with some auto-install logic (https://quarto.org/docs/output-formats/pdf-engine.html#quarto-pdf-engine) but you could also opt out and use another one (https://quarto.org/docs/output-formats/pdf-engine.html#latexmk). In that case Quarto won't render to PDF itself and I suspect that you can pass to pdf-engine a program of yours doing the wrapping you need, so that Pandoc calls this engine.

Hope this helps

[jdelker]

Hi @cderv, thanks for your response.
Despite you good description and references, I'm still figuring out, how that could help me (see below).

Doing something like

filters:
 - at: post-render
   path: filter.lua

would apply your Lua filter after all Quarto processing, before Pandoc writing to output. So if you want to write the LaTeX yourself > from there instead of Pandoc this could be possible.

I don't want to write the LaTeX myself ;) Pandoc is doing this just fine - with just a few tiny exceptions I would like to tweak.
Unfortunately, that is nothing I can do earlier in the pipeline, and if I understood your comment correctly, Quarto's filtering ends with the the "post-renderer" before Pandoc is called to generate the LaTex.
So that would unfortunately not help me. I would require some hook, so I can modify the LaTeX after Pandoc and before the PDF (xelatex) rendering. Hence my approach with a custom pandoc output writer - but that would be challenge on it's own.

Considering the PDF-Engine approach, I'm stuck, too.
Actually, I would just like to provide some wrapper-script, that still uses Quarto's pdf-engine(s) but just performs some tweaks on the input LaTex before it is passed on. All my attempts to provide a custom "pdf-engine" (by referencing a local executable script) only result in the error "pdf-engine <...> not known". Unfortunately, the documentation of the "Latexmk" section is not very helpful how to actually integrate a custom pdf-engine.

Any more thoughts?

[cderv]

I don't want to write the LaTeX myself ;) Pandoc is doing this just fine - with just a few tiny exceptions I would like to tweak.

I thought it was bigger change you wanted to make. If you want to tweak only some LaTeX written by Pandoc writer, then post processing after the file being written is the only option.

So that would unfortunately not help me. I would require some hook, so I can modify the LaTeX after Pandoc and before the PDF (xelatex) rendering.

Quarto does not offer hooks entry point for post processing a .tex file before is PDF generation. You would need to use Quarto to produce LaTeX file and then have your own pipeline for pdf rendering 🤔

Hence my approach with a custom pandoc output writer - but that would be challenge on it's own.

Custom Pandoc Output writer is equivalent to Pandoc Lua filter writing TeX output. This is because you mentioned this that I pointed you to the AST entry point we have.

What you want seems to post process the .tex file after it has been produced by Pandoc, and before a pdf engine is called on it.

Actually, I would just like to provide some wrapper-script, that still uses Quarto's pdf-engine(s) but just performs some tweaks on the input LaTex before it is passed on

Unfortunately we don't allow inserting processing before calling the rendering to PDF. This needs to be done in its own pipeline as mention.

• Render to LaTeX with Quarto
• Post process using your own tool
• Render to PDF using an engine.

All my attempts to provide a custom "pdf-engine" (by referencing a local executable script) only result in the error "pdf-engine <...> not known".

The engine needs to be found by Pandoc itself, and probably needs to be a program it recognized. I don't know if its --pdf-engine option supports a script. Maybe if it is executable one.

Unfortunately, the documentation of the "Latexmk" section is not very helpful how to actually integrate a custom pdf-engine.

This is because it is not a Quarto feature. If the Quarto LaTeX engine is opt out, then the PDF rendering will be the same as what Pandoc does. What Quarto does by default is taking over Pandoc doing exactly the pipeline mentioned

• Render to .tex with Pandoc
• Apply some post processing we have
• Call a PDF engine on this .tex output.

I would just like to provide some wrapper-script, that still uses Quarto's pdf-engine(s) but just performs some tweaks on the input LaTex before it is passed on

Considering the pipeline thing mentioned, there is something not documented anywhere I think. The Quarto Latexmk engine has been exported in its own tool call tinitex. It is experimental and available as part of TinyTeX releases : https://github.com/rstudio/tinytex-releases/

Here is the help of this CLI tool

❯ tinitex.exe --help

Usage:   tinitex <input>
Version: 1.6


Description:

  Quarto Latexmk Engine

Options:

  -h, --help                         - Show this help.
  -V, --version                      - Show the version number for this program.
  --pdf-engine        <engine>       - The PDF engine to use
  --pdf-engine-opt    <optionsfile>  - Options passed to the pdf engine. Can be used multiple times - values will be
                                       passed in the order they appear in the command. These must be specified using
                                       an '='.
  --index-engine      <engine>       - The index engine to use
  --index-engine-opt  <optionsfile>  - Options passed to the index engine. Can be used multiple times - values will be
                                       passed in the order they appear in the command. These must be specified using
                                       an '='.
  --bib-engine        <engine>       - The bibliography engine to use
  --no-auto-install                  - Disable automatic package installation
  --tlmgr-opt         <optionsfile>  - Options passed to the tlmgr engine. Can be used multiple times - values will be
                                       passed in the order they appear in the command. These must be specified using
                                       an '='.
  --no-auto-mk                       - Disable the pdf generation loop
  --min               <min>          - The minimum number of iterations
  --max               <max>          - The maximum number of iterations
  --output-dir        <directory>    - The output directory
  --no-clean                         - Don't clean intermediaries
  --log               <file>         - Path to log file
  --log-level         <level>        - Log level (info, warning, error, critical)
  --log-format        <format>       - Log format (plain, json-stream)
  --quiet                            - Suppress console output.

Commands:

  help  [command]  - Show this help or the help of a sub-command.

It was an experiment made to have something that can be reused outside of Quarto - specifically because Quarto latexmk engine gets inspiration (duplicates) the tinytex R package (https://yihui.org/tinytex/r/) that goes with TinyTeX distribution (https://yihui.org/tinytex/).

You could give a go by having your own pipeline (Makefile or others...) that

• Calls Quarto
• Calls your post processing
• Calls tinitex

Hope it helps

[jdelker]

Thanks for the clarification and insight to tinitex.
I'm not sure yet, if this will work out as intended, but at least I'm confident now, that I haven't overlooked any "post-filtering-option" ;)