RFC: splitting up MANUAL.txt

[silby]

Summary

I propose to split Pandoc's MANUAL.txt into multiple documents, separating the documentation of the pandoc command and its options from the documentation of Pandoc-flavored Markdown and other material related to file formats. This would allow the generation of more concise and topic-specific man pages from the documentation, would establish a pattern for documenting currently-undocumented features of certain readers and writers, and could make documentation maintenance more accessible to contributors.

Current state issues

MANUAL.txt runs to 7500+ lines of Markdown and constitutes a large fraction of Pandoc's documentation all by itself. It covers a great many topics, such as:

• Command-line options and arguments of pandoc CLI
• Reference documents for Docx, ODT, and PowerPoint
• Format of defaults files
• Syntax of Pandoc's templates in general
• Writer-specific template variables
• Extensions relevant to multiple readers and writers
• Extensions specific to single readers or writers
• Pandoc-flavored Markdown syntax and its extensions
• Slideshow output
• EPUB output
• Jupyter output

The manual's focus on Pandoc-flavored Markdown as the first-among-equals input format for Pandoc has come to seem out of place to me. Pandoc now parses a great many formats, and a user might well use it primarily with Djot, rST, AsciiDoc, or Commonmark input. Such users are not served by assumptions in the manual that treat Pandoc's Markdown as the default format. Pandoc's Markdown, in turn, is quite featureful, and the "Pandoc's Markdown" section of MANUAL.txt is about 1/3 of the file.

Conversely, for formats other than Pandoc's Markdown, there are format-specific features and missing features that aren't documented anywhere. For a simple example, the HTML reader preserves some semantics by reading the <var> tag into a Pandoc Code with class variable, and the HTML and Texinfo writers understand this convention. (My wip mdoc reader does this too.) Divs with class "section" get treated specially by the HTML writer, and this has caught users out, see #8757. reStructuredText mavens are a good source of evidence here, see @jgm in #10318:

Documenting all the reST features we don't support might be interesting, but I don't think it goes in the manual because it might be quite long, and because it would be odd to include this without similar sections about every other format (and that would be really long). We could think about including a separate document on the website, the way we have a document about Org-mode support.

As mentioned, Pandoc already has format-specific documentation contributed by @tarleb for org and JATS. The work I propose to do will endorse and extend this pattern.

Proposed work

I propose we start splitting MANUAL.txt into multiple documents, each suitable for conversion to a manual page and for rendering as an HTML document on the website. The organizational principle would be that MANUAL.txt continue to serve as the pandoc(1) man page, documenting the function of each command-line option and argument, but excluding the detailed material on specific input and output formats.

The extracted material would be organized and adapted into new documents under doc/, perhaps initially along these lines (not comprehensive):

Title	manpage	Contents taken from MANUAL.txt
Pandoc templates	pandoc-template(5)	Templates section
Defaults files	pandoc-defaults(5)	Defaults files section
Pandoc's Markdown	pandoc-markdown(7)	Pandoc's Markdown section
LaTeX support	pandoc-latex(7)	LaTeX variables and LaTeX-specific extensions
Slide shows	pandoc-slides(7)	Slide shows section
EPUB support	pandoc-epub(7)	EPUBs section

Beyond simple copy-pasting, work would be required to make the extracted material function well as standalone documents and integrate them with the website and manpage builds.

Possible followups

Plenty of reader and writer formats have no Pandoc-specific documentation. With a better-established pattern for writing and publishing these documents, we can start filling in the gaps.

All the documentation of the Pandoc AST as such is aimed at Haskell developers (through the pandoc-types haddock) and Lua filter writers. The documentation of Pandoc's markdown sort of serves as the documentation of "what's a Pandoc document?" We could write an author-focused document that describes the semantics of Block and Inline AST elements without the coding details. This documentation could also guide developers of new native or custom readers and writers, by providing a format-neutral description of the semantics of each element.

Most extensions only affect a subset of readers and writers. Each extension could be documented in its own individual file, with metadata indicating the supported readers and writers. The man and html versions of the relevant format documentation could then be scripted to include all relevant extensions. This would let each format document all available extensions without duplication and drift in the source files.

[jgm]

See commit a04c15a for some interesting background. Long ago, we used to have a separate pandoc_markdown(5) man page. According to the commit, the main reason we gave that up was that it made cross-references difficult. So that's one thing that would need to be thought through with this proposal. If we were just producing man pages, we could just say (see pandoc_markdown(5) under XXX), but we also want nice links in the HTML versions that go to the right places. Probably this is something that could be solved with some ingenuity, but it needs to be thought through. The advantage of the big omnibus MANUAL.txt is that it's trivial to create cross-references within it.

[silby]

Oh interesting, thanks for pointing out that commit. I think the good news is that, unlike at the time of that commit, we can get pretty far with Lua filters. For example, where the manual today has something like See [Templates], we could write See [Templates](template.5.md). A Link filter for the website output could turn this into a link to /template.html and a filter for the man page output could replace it with See pandoc-template(5). More complicated ideas are possible for dealing with See [Page] under heading but I think that in practice, if the general idea I have for the structure of these things is adopted, it'll be most common to link to another document in general rather than an anchor.

(As a data point, MANUAL.txt is now over twice as long as README was at the time of that commit 9 years ago.)

[jgm]

Yes, I agree that in principle we could solve the problem with filters.

[bpj]

Since links can have attributes you can use e.g.

[templates](templates){.doc data-man=5 data-sec=variables}

which would ease the job of the filters considerably. One filter could just do

local attr = link.attributes
if link.classes:includes('doc') then
  if 'man' == FORMAT and attr.man then
    link.target = link.target .. ('(' .. attr.man .. ')')
  elseif config.single_file then
    link.target = '#' .. link.target
    if attr.sec then
      link.target = link.target .. ('-' .. attr.sec)
    end
  else
    link.target = link.target .. extension_for[FORMAT]
    if attr.sec then
      link.target = link.target .. ('#' .. attr.sec)
    end
  end
end

I already use this technique (minus the man attribute) and other similar ones for complex, multi-file documentation which should be converted to multiple formats. The sec attribute is a bit of a hack but it works well. The config table is populated from a read-in JSON file,¹ so bypasses metadata.

Footnotes

1. Actually a YAML file which is converted to JSON with yq and read in at once via a pipe, then decoded with pandoc.json. ↩

[fiapps]

I've never consulted the manual in manpage form. I regularly use the PDF or the website because I can use the table of contents to jump to the relevant section. I find it convenient to have most of the documentation I regularly consult downloadable as a single PDF and would not like to see it split up. The Lua filters documentation is the only thing I regularly consult that isn't in that PDF.

[silby]

Thanks for sharing your use-case. It’s making me think that my proposal here should address both how the documentation source files are organized and how artifacts are built out of them. In PDF format it makes sense to have all the documents in one file rather than have to download a bunch of PDFs, and a future PDF build of the manual could include auxiliary documents as chapters, including parts that aren’t in the present User Guide PDF.

[wlupton]

I regularly use the PDF or the website because I can use the table of contents to jump to the relevant section.

Agreed, so I think that it would be good to retain a single HTML file as one of the generated artifacts.

[jgm]

We already have the documentation split over multiple files -- in addition to the main manual, we have filters, lua filters, custom readers, custom writers, pandoc-server, pandoc-lua, org-mode features...

I don't think some additional splitting would really change the situation all that much, as long as we have good cross-references.

It's also true that we could fairly easily combine multiple sources to create a single PDF or HTML page if we wanted to. Or, we could create a "master table of contents" with links to everything.

Indeed, the opposite is also possible: we could use filters to create multiple outputs (e.g. man pages) from a single source.

[tarleb]

I would very much welcome a split, the current MANUAL is a bit intimidating and overwhelming.

I've also considered to re-home the Lua API docs to a separate file, which might help to navigate the Lua filter documentation.

[lifeunleaded]

I concur in principle for reasons of authoring and actually reading the thing, but when splitting things it would be good to consider ease of search. I frequently word search the man page when I forget a syntax. An option to output it all in one go might be useful. Maybe a shorthand for pandoc --print-default-data-file MANUAL.txt that aggregates all of them, just to have a pile of it in one place.

[jgm]

Getting down to a more detailed level, the original proposal above was:

Title	manpage	Contents taken from MANUAL.txt
Pandoc templates	pandoc-template(5)	Templates section
Defaults files	pandoc-defaults(5)	Defaults files section
Pandoc's Markdown	pandoc-markdown(7)	Pandoc's Markdown section
LaTeX support	pandoc-latex(7)	LaTeX variables and LaTeX-specific extensions
Slide shows	pandoc-slides(7)	Slide shows section
EPUB support	pandoc-epub(7)	EPUBs section

I think that putting pandoc-templates in a separate document would make sense.

I'm less sure about pandoc-defaults. There is a very close coupling between pandoc-defaults and the command-line options; when one is changed, the other needs to be changed too, so it might be more practical to keep them together. This also makes it clearer to the user how this coupling works. On the other hand, it's the sort of thing that often gets its own man page.

pandoc-markdown, you might think, could go in its own document. One problem, though, is that this is where many of the extensions are documented. Documentation of the extensions, one might think, ought to be included in the main man page. To make things even more confusing, some of these extensions also affect other formats. For example, many pandoc-markdown extensions also work for commonmark/gfm. tex_math_dollars works in HTML. So this speaks against putting them in pandoc-markdown.

pandoc-latex-support: Much of this is in a section documenting what different settings of variables do in various formats. So, would we remove the material about LaTeX from this but keep the other formats? That seems odd.

pandoc-slides, pandoc-epub: These could be separate pages. (Along with pandoc-org, pandoc-jats, pandoc-typst, currently already in a separate files but not made into man pages.)

The documentation for filters, lua-filters, and custom readers and writers could also be made into man pages.

Still unsure on the whole. The biggest gains would come from pulling out pandoc-markdown, but for reasons given above this isn't so straightforward.

Most extensions only affect a subset of readers and writers. Each extension could be documented in its own individual file, with metadata indicating the supported readers and writers. The man and html versions of the relevant format documentation could then be scripted to include all relevant extensions. This would let each format document all available extensions without duplication and drift in the source files.

So, on this conception we'd have man pages for pandoc-FORMAT for every supported format? These would include documentation for the variable settings affecting the format, the extensions affecting it, and anything else (e.g. the current section on EPUBs). Not a bad idea perhaps, but it would mean that pandoc would have a huge number of man pages. I'm not sure that would be appreciated.

[silby]

pandoc-markdown, you might think, could go in its own document. One problem, though, is that this is where many of the extensions are documented. Documentation of the extensions, one might think, ought to be included in the main man page. To make things even more confusing, some of these extensions also affect other formats. For example, many pandoc-markdown extensions also work for commonmark/gfm. tex_math_dollars works in HTML. So this speaks against putting them in pandoc-markdown.

I don't believe extensions necessarily need to be found in the pandoc(1) manual page. To me they belong more to the description of reader/writer formats than to the description of how to invoke pandoc. Most formats that aren't Markdown variants support a small number of extensions, and even if we don't do the fancy thing described below of automatically populating format docs with extension descriptions, I think we'd get away with a modest amount of duplication early in the editorial process.

Even as-is, the documentation of widely-supported extensions lags behind reality. auto_identifiers is supported by many formats and often on by default when it is supported, but the manual only mentions a few formats.

Looking at things today, it seems to me like a pandoc-markdown doc might as well cover all the markdown variants supported by pandoc in one place, rather than laboring over pandoc-gfm, pandoc-commonmark, etc.

pandoc-latex-support: Much of this is in a section documenting what different settings of variables do in various formats. So, would we remove the material about LaTeX from this but keep the other formats? That seems odd.

Nah, I should've put an "etc" in there. I think "Variables for X" will land in a pandoc-X doc for all X.

Most extensions only affect a subset of readers and writers. Each extension could be documented in its own individual file, with metadata indicating the supported readers and writers. The man and html versions of the relevant format documentation could then be scripted to include all relevant extensions. This would let each format document all available extensions without duplication and drift in the source files.

So, on this conception we'd have man pages for pandoc-FORMAT for every supported format? These would include documentation for the variable settings affecting the format, the extensions affecting it, and anything else (e.g. the current section on EPUBs). Not a bad idea perhaps, but it would mean that pandoc would have a huge number of man pages. I'm not sure that would be appreciated.

In the man page setting, my experience is that having more man pages makes navigating easier than having a very long man page. There's a slight discovery challenge, in that to see everything there is to see you have to know to type apropos pandoc instead of just man pandoc and using your pager to search, but apropos pandoc provides a clear overview of high-level topics.

As for the volume, git ships 195 man pages, as an extreme example, and it's not an obstacle to packaging git. The union of --list-input-formats and --list-output-formats has 77 formats so I think that's the outside edge of what we'd end up with. Some docs would probably cover more than one format, i.e. the various profiles of JATS, asciidoc variants, the various HTML slide options, maybe the bibliography formats.

Thanks for your review and feedback. I have proposed a variety of things requiring both editorial and technical effort but I don't really want to suggest that it all has to happen in a big bang to be worth while. Extracting a pandoc-markdown page would make the biggest difference for authoring but I don't think it needs to happen off the bat; we could do stuff like create a Templates doc and write the code to build man pages out of other docs first.