[Discussion]: Merge with minetest_docs project, unify all documentation

[GreenXenith]

This is a long-term goal! Cleaning up what exists here and merging the other wiki should come first.

The intention of minetest_docs was to improve the developer experience by cleaning up and reformatting the Luanti documentation. That failed miserably due to lack of practical planning and resulting lack of motivation. Since then we have switched to a simpler markdown page approach, that way contribution is easier. We were also considering using something like Pandoc to preprocess the markdown for some advanced features (macros and such).

However, this site checks multiple boxes that all that aimed for: It uses simple markdown pages, has a preprocessor (with far more features than Pandoc! Including macros and templating!), it is already doing the job of some developer documentation, and it looks nice. So my proposal is to unify all Luanti documentation under this one roof.

The wikis are already being merged so I will just cover the possible structure changes and implications associated with merging the API reference.

Structure

The current site structure is based on the old wiki format, which worked fine for a wiki, but isn't great for documentation as a whole. Higher-level categories should exist (for example player-facing, dev-facing, and server-facing) to navigate better.

The Hugo book theme is fine for now, no need to bikeshed that. Would likely need to make multiple book pages for each section, though.

API Reference

Through a lot of discussion on Discord, a few realizations were made:

• Luanti lacks a lot of inline documentation for engine developers
• Developers should be in charge of documenting their code
• Writers should be in charge of writing/approving user-facing documentation based on developer documentation
• LDoc supports comments in C/C++ code

From that, here is a possible system:

• API method structures, types, and short description should be documented in-code with LDoc format
• Documentation imports data exported using LDoc
• Further explanation and examples are added in documentation
• Docs team can modify in-code docs when needed

This means any new features should be documented by the contributor in-code, and a task would be created on-merge for the Docs Team here. The contributor can also provide documentation "metadata" (include it in the PR post?) if the feature lacks methods for LDoc to work with or if they just want to give the Docs Team something to start from, or they could open a separate PR here.

Existing methods would need to have annotations added, which is a rather large task. But type definitions would need to be written at some point anyway.

A note on LDoc: There are 4 Lua annotation systems so far: LuaDoc, LDoc, EmmyLua, and LuaCATS. Each one is based on the previous. Only LDoc is backwards-compatible (it supersedes LuaDoc). Only LDoc supports reading C++ annotations for Lua. LuaCATS is a hot mess right now. There is no need to support EmmyLua/LuaCATS anyway (I can convert LDoc to LuaCATS for VSCode as needed).

I have a few other thoughts but this is what I've written for now. CC @mark-wiemer

[mark-wiemer]

Looks great! I'm busy for the holidays until the end of the year but will be checking GitHub nearly daily. Very excited about this project as you can hopefully see from my contributions to the Luanti Tools repo :) I think we're on the same page that inline docs should include basic modder guidance and link to external docs, which would serve as a broader wiki-type website hosted here :)

It'll take some time, but iterative improvements are much better than nothing. I'll be focusing on Luanti Tools for a bit because I believe the in-editor experience is critical to any sort of guidance, but I'll be watching changes here as well :)

[mark-wiemer]

My general vote is to merge the projects. dev.luanti.org can host both:

• reference material: short descriptions of each function and their parameters
• guide material: longer tutorials, guidance, and cross-cutting concerns

I see this as very similar to learn.microsoft.com, which hosts basically everything about all Microsoft software.

Looks like the main remaining points, some maybe out of scope of this issue, are:

1. Organize the pages into a hierarchy of some sort so it's less intimidating and more easily navigable
2. Start adding inline documentation to serve as the basis for reference material
3. Use a parser like LDoc to extract inline documentation into user-facing content here

Overall it seems the current work on dev.luanti.org is:

1. Importing the "worthwhile" pages from wiki.minetest.org and minetest_docs (Stuff to move from wiki.luanti.org #15)
2. Filling in the remaining gaps in doc coverage (Match lua_api.txt coverage (mega tracker) #47)

[appgurueu]

Higher-level categories should exist (for example player-facing, dev-facing, and server-facing) to navigate better.

I agree. This is very important if mod & engine dev docs are to live under the same roof. We do not want to bother mod developers with engine internals (and vice versa).

API method structures, types, and short description should be documented in-code with LDoc format

Personally I'm a fan of LDoc, but we'll have to see what other @minetest/engine developers think.

[mark-wiemer]

This issue has other points that aren't completed or agreed upon (sectioning, LDoc) - GreenXenith in IRC

Leaving open

[Desour]

It would be nice to have machine-readable documentation for API functions, definitions, and types. From that one could then generate pretty, organized, and searchable doc.

---

Writers should be in charge of writing/approving user-facing documentation based on developer documentation

Further explanation and examples are added in documentation

There should be only one source of truth, imo. And this truth of course comes from the PRs that add the feature (and possibly after that some doc improvement PRs if necessary). The exact wording often matters. Rewriting all the doc in friendlier language and adding "extra" (???? it must already be complete, what further explanation would you want to add????) information would be a source of errors, due to misinterpretation and it becoming outdated. Small examples would be nice to have I guess, but it also needs maintenance.

This means any new features should be documented by the contributor in-code, and a task would be created on-merge for the Docs Team here. The contributor can also provide documentation "metadata" (include it in the PR post?) if the feature lacks methods for LDoc to work with or if they just want to give the Docs Team something to start from, or they could open a separate PR here.

This sounds like you want to have a docs team that takes away the work of the contributor to properly document their own feature. Am I misunderstanding this? Or do you mean it now always needs a docs person to copy-paste the non-user-facing docs to the user-facing docs?

Having someone from a docs team review the docs part of a PR could probably make sense though.

---

I don't believe having a "unified" documentation can work. A wiki is meant to be edited by everyone, right? As such, it should make editing accessible and easy. References on the other hand should be cut in stone, unambiguous, and always up-to-date. Tutorials are best suited in a form of book, which includes much prose text, and guides the reader in a tour through parts of the API, it is not a knowledge base. A wiki can not be kept up-to-date, unless you disallow users adding more content and have enough editors available to maintain the existing parts. So while a wiki can be a knowledge base of extra context, it can't be a reference.

[GreenXenith]

It would be nice to have machine-readable documentation for API functions, definitions, and types.

That is the goal of the LDoc integration and paginated API reference.

And this truth of course comes from the PRs that add the feature (and possibly after that some doc improvement PRs if necessary). The exact wording often matters.

Yes. That is the intent. The PRs define the documentation. Specifically, they use inline documentation to define structure (types, values, structure, etc) for developers to have documentation in-code, and they also provide documentation to be placed here with collaboration from the Docs Team.

Rewriting all the doc in friendlier language and adding "extra" (???? it must already be complete, what further explanation would you want to add????)

This sounds like you want to have a docs team that takes away the work of the contributor to properly document their own feature. Am I misunderstanding this?

I apologize if that was how it sounded, that was not my intent. Forget the "user-facing" verbiage. I simply meant experienced writers should collaborate in making sure documentation is understandable, concise, relevant, and well-formatted (which it currently is very much not).

Small examples would be nice to have I guess, but it also needs maintenance.

Almost every major engine has examples, they are critical for new users.

I don't believe having a "unified" documentation can work.

Hard disagree. It works for pretty much every major engine.

There should be only one source of truth, imo.

A wiki is meant to be edited by everyone, right? As such, it should make editing accessible and easy. References on the other hand should be cut in stone, unambiguous, and always up-to-date.

A wiki can not be kept up-to-date, unless you disallow users adding more content and have enough editors available to maintain the existing parts. So while a wiki can be a knowledge base of extra context, it can't be a reference.

This project is moving towards being a documentation reference and away from a wiki. One source of truth instead of 2 outdated/broken wikis and a messy monolithic API reference. Most of the information the wikis contained should have been unambiguous and up to date.

Tutorials are best suited in a form of book, which includes much prose text, and guides the reader in a tour through parts of the API, it is not a knowledge base.

Correct. It is still a form of documentation, however. It fits well under that umbrella and makes it easy to find resources when under one roof.

[mark-wiemer]

@Desour, I'll reiterate that one website can host both an API reference (like lua_api.md) and tutorials (like Ruben Wardy's book). My favorite example is learn.microsoft.com (I work for Microsoft but I still love that site, all opinions here are my own). As long as the content is sectioned appropriately, having both a concise API reference and prose-y tutorials on the same site can be immensely valuable as a single-source-of-truth.

[appgurueu]

it must already be complete, what further explanation would you want to add????

A lot? The reference can be terse, it doesn't need to give extensive advice on how to use something.

Consider e.g. the ANSI C99 standard versus "The C Programming Language" by K&R. Or the Lua reference manual versus "Programming in Lua".

Sure, for an experienced programmer the reference suffices. But you don't become an experienced programmer by only reading references. (Well, you sorta can, but then it may take a good while longer.)

If we reach close to 100% supplementary docs API coverage, we could discuss unifying all documentation.

Starting to use LDoc in the engine is probably a good idea either way.