Documentation strategy: rustdoc, man pages, header files

[jsha]

I'd like to figure out a solid plan for documentation. Right now our header file is our documentation. That's got a bunch of disadvantages. It's one huge file, so it's hard to navigate. It can't hyperlink words or sentences, or apply styles. And it can't nicely group related functions (like _read/_write/_read_tls/_write_tls). However, it has one big advantage: The contents are auto-generated by inline doccomments.

For Rust code, the standard tool for generating documentation is rustdoc. This generates nicely organized docs for our project right now, but those docs refer to the Rust signatures for functions, not the C signatures. That makes it a non-starter as our primary documentation. But perhaps we can do something hacky like search-and-replacing the signatures in the generated rustdoc? Or perhaps we could convince rustdoc to add features to support FFI projects, though it would be a very significant change in the rustdoc model.

The more typical option is Unix-style man pages, maintained separately from the doccomments and turned into HTML for display on the web. I think long term this is probably the way to go, particularly as the API stabilizes and the documentation doesn't need updating as often. However, it's a shame to lose out on all the useful tooling and styling that rustdoc gives us.

Thoughts?

[kevinburke]

For me the biggest concerns about documentation reflect the concerns that I had getting this code integrated into existing programs, which are about bigger issues than the reference documentation.

The hard part was actually getting to a point where you could put a Rust function from rustls.h into a C file and invoke make and actually get the Rust code to link in. Once that was working, it was relatively smooth sailing - the reference documentation was good enough.

The goal of the project as I see it is to spread memory safe implementations as widely as possible which may mean casting a wide net and attracting more people like me who haven't done C before but are interested in Rust and making projects more secure. Maybe it doesn't need to live on the Rust website, but I'd hope to see documentation sections for:

• Abstracting OpenSSL calls behind a TLS interface, similar to the one in Curl's "vtls" or Postgres's fe-secure.c, (if one does not already exist)
• How to edit configure.ac or configure to find a Rustls installation
• Lifecycle of a TLS request and where you'd expect each rustls function call
• Example code! Hopefully, lots of example C code explaining what's going on, for most of the common function calls. It was confusing to me for example that opening a connection involved lots of read and write calls since Postgres has explicit interfaces for pgtls_read and pgtls_write, naively I just assumed those would be used.
• Introduction to C for people who haven't written C before. What do the different C types mean, that rustls uses? How do you return more than one thing from a function? How do you make an array of variable length?

[kevinburke]

More to your initial question, are there tools that will take a C header file, and turn it into a HTML file with a function summary at the top, with links to the full function declarations? Like the godoc output does, where all of the functions are listed at the start and then you can click on one to get to the docstring.

[cpu]

I've been giving this some thought recently. I'd like to get some kind of API docs online from the rustls.h content as a first step.

More to your initial question, are there tools that will take a C header file, and turn it into a HTML file with a function summary at the top, with links to the full function declarations?

I've been using the BoringSSL API docs as a point of reference. I think that project strikes a pretty good balance of offering usable API docs without too much additional ceremony. Their API doc pages are generated using a small Go program that takes a config file and stylesheet as inputs.

As a quick experiment I tried running this tool unmodified on our .h and it immediately fails with an error about no C++ guard being found (#if defined(__cplusplus)). We can get cbindgen to include those guards by setting cpp_compat = true, but it still doesn't quite work. The BSSL tool seems to expect one guard, and cbindgen is putting a few in-file.

In sum my conclusion is that the tool is pretty boringssl specific and the simplest path forward might be a small Rust port that fits into our use case a bit better:

• I'd rather not introduce a new language/tooling (Go) for just the docs.
• I'd rather not have to do a lot of work configuring cbindgen to match elements of the human curated boringssl header files.
• We only have one .h, not many, and so some of the complexity of doc.go isn't needed.

I'm not super keen on something heavyweight like doxygen and would rather put a bit of work into a small util that can get us something workable with less overhead.

For CSS we can perhaps lift some of the content @ctz created for https://rustls.dev and the Rustls repo's Zola site.

[cpu]

In sum my conclusion is that the tool is pretty boringssl specific and the simplest path forward might be a small Rust port that fits into our use case a bit better

This is handled in #510

I started out by trying to hand-roll a parser that did basic string operations ala the boringSSL Go tool. It was pretty fragile and cumbersome to maintain and I realized that since we're using Rust we have better tools available to us. Namely, https://tree-sitter.github.io/tree-sitter/

In #510 I used tree-sitter-c to generate an AST for the rustls.h headerfile. That's then traversed by the docgen Rust tool to pull out the elements we want to document, their textual representation, and their block comments. From that we can create a structured .json representation that's friendly for generating different presentations.

For now I've focused on a "web" presentation and used Zola to template the .json into web content. In the future we could implement something similar to generate man pages from the .json.

[cpu]

Here's a preview I made with a test repo: https://cpu.github.io/rustls-ffi-docs/

I haven't done a pass updating comments to make sure they're nice. Also need to give some thought to cross-linking.