Better organise documentation

[cmacmackin]

I know this would be a big job, but it would be really useful if the documentation could be organised better. Function documentation is organised by source file and it often is not self-evident in which file a given function would be written. Browsing through the sourcefiles exposes the user to some of the backend implementation. Furthermore, there is no clear distinction between low-level and high-level features within each module. Much of this criticism applies to Nim docs as a whole (I don't find the default tool is particularly good).

Relatedly, the tutorials section could probably use expansion and additional documentation could likely be given at the top of each file. Of course, I fully appreciate that this would be a lot of work (and generally less interesting than implementing new features).

[Vindaar]

I appreciate the criticism, because it's warranted to an extent. That's why there is a search function in the docs though.

About the docs in general:

The problem is: How would one better organize the docs?

Examples of docs in particular I don't find better than Nim docs:

• Pandas docs are just a super long list of things on the LHS, which is polluted by random namespacing (of course Python being dynamic makes reading docs a horrible thing in general) https://pandas.pydata.org/docs/reference/frame.html
• numpy's docs are so bad that I always google "numpy " to find the correct page: https://numpy.org/doc/stable/

Example of good docs:

• Clojure docs are amazing imo. The overview page of each part packs lots of information and doesn't give you random garbage: https://clojuredocs.org/clojure.core and in particular each page for a function allows for users to provide examples right there: https://clojuredocs.org/clojure.core/map

Now, personally I don't know how to improve the situation without writing a custom doc generator and even if I wanted to do that I wouldn't know what to change.
In general I think the Nim docs are actually pretty good and diverging too much from Nim docs for arraymancer isn't a good idea imo. It's important to have similar documentation for all modules in the ecosystem so that one finds their way.

And note that there is always the index: https://mratsim.github.io/Arraymancer/theindex.html which you can also use for a broad overview + in browser searching for keywords (if for some reason the JS search box isn't helping).

About high level / low level

Arraymancer's docs are already quite different from the average Nim docs, with the header and all that. In principle the machinery is there to put arbitrary modules into arbitrary header tabs. This is all done here:

https://github.com/mratsim/Arraymancer/blob/master/docs/docs.nim#L176-L287

and using the template from here:

https://github.com/mratsim/Arraymancer/blob/master/docs/nimDocTemplates.nim#L113-L194

It would be pretty easy to have a separate tab for only low level stuff. And indeed that's kinda the idea behind the Other docs tab. The issue is just that the mapping needs to be updated and possibly made a bit smarter.

Browsing through the sourcefiles exposes the user to some of the backend implementation.

Personally, I like that. I hate having to step through all sorts of trouble to find references / documentation to the backend parts. Just because they are not relevant to the average user, doesn't mean there shouldn't be docs for them. Could there be a better distinction as such in the docs? Certainly.

Relatedly, the tutorials section could probably use expansion and additional documentation could likely be given at the top of each file.

The doc generation mentioned above and in particular tutorial sections / module headers are all things that are easy to contribute to. So please feel free to help out if this is important to you. I've invested quite some nerves into the whole doc business already to attempt to make the generation smarter and to an extent to allow other people to contribute better.
Don't be intimidated to contribute please.

[mratsim]

My documentation plan is to follow the outline of https://documentation.divio.com/

	Tutorials	How-to guides	Reference	Explanation
oriented to	learning	a goal	information	understanding
must	allow the newcomer to get started	show how to solve a specific problem	describe the machinery	explain
its form	a lesson	a series of steps	dry description	discursive explanation
analogy	teaching a small child how to cook	a recipe in a cookery book	a reference encyclopaedia article	an article on culinary social history

However, Nim doc is seriously lacking for this.

1. Nim doc is "single-file" oriented, it was written with system.nim and the standard library in mind not with libraries that might export public-for-the-public and public-for-internal-modules functions from the same file. Hence documentation is cluttered with things that are irrelevant for end users. nim doc recursive should allow doc generation only for public proc. nim-lang/Nim#14375
2. Nim doc must be done once per file, I can't create a large doc with all functions from a logical module (say, tensors, machine learning, linear algebra, deep learning, ...)
3. Nim doc doesn't support markdown. RST is just a pain to write and RST grid table not being supported doesn't help. [Doc gen Feature request] Grid Table nim-lang/Nim#6429
4. Nim doc doesn't ease creating navigation menu
5. Nim doc crashes when a documentation includes plain * which is considered by RST as the start of italicizing but autogenerated C wrapper are littered with those due to those being pointers [Documentation] uneven '*' character --> Error '*' expected nim-lang/Nim#6910 Escape '*' in doc comments nim-lang/c2nim#147
6. When I want to generate documentation I lose 4-8 hours trying to workaround Nim limitations. This is very discouraging.

I tried to generate json as well as a workaround so that it could be consumed from other doc generators.

I don't mind spending time to polish docs, see for example what I'm producing for work with auditors https://status-im.github.io/nimbus-eth2/auditors-book but Nim docgen is very unsuitable for large projects.

To be clear, my ideal baseline for a documentation system would be:

• Collect and export all public procs and types (exported from arraymancer.nim) into an API reference file.
• Possibility to add markdown files that would be categorized into Tutorials, How-Tos and Explanations
• Image in markdown support.
• Documentation wide search, for example using Algolia.
• Navigation menu (and no fiddling with viewports to try to fix the display).
• No crash on c2nim generated wrapper for something as basic as *
• Single command to generate the documentation, can be integrated in CI.

Extras include:

• Organize the API reference per module (either via folder organization or by introducing a tag).
  This is necessary to turn Arraymancer in some kind of monorepo.
• Pass defined flags to handle when defined(cuda) or when defined(opencl)
• dark mode / theming
• support versioning (devel and latest release).

[cmacmackin]

I agree that nimdoc is... less than stellar. One thing I find particularly irritating (although perhaps just due to an object-oriented background) is that it lists all the types and then all the procedures working on those types, but they often aren't in any particular order. The times module is particularly guilty of this. It makes it hard to determine quickly which procedures are available for Time objects, which are available for DateTime, etc.

Nim doc must be done once per file, I can't create a large doc with all functions from a logical module (say, tensors, machine learning, linear algebra, deep learning, ...)

Could a workaround be to make a file which includes a number of related files and then run nimdoc on that? Or does nimdoc not work that way? Alternately, I suppose you could write a script to generate your docs which concatenates a bunch of files together before running nimdoc. Less than elegant, but should work.

It sounds to me that what you really want would be nim integration with Sphinx. By default it still uses RST, although I think Markdown is an option (at the very least, it does support RST tables). It was written for Python but has plugins to support other languages. It shouldn't be too difficult to write an extension to support Nim constructs. The more difficult task would then be to write a program which would automatically generate the necessary Markdown or RST code to describe your library. You mention exporting docs into JSON, so if you have that then you're already part way there. I have some expertise with documentation generators (having written one for Fortran) so I could potentially help out with this.

Edit: I see you've already raised that issue (nim-lang/needed-libraries#79).