Documentation syntax

[Cecca]

I'm opening this issue to start the discussion on documentations syntax. I think that there are three kinds of documentation:

• in depth documentation, contained in READMEs, Markdown files etc.
• annotated usage examples
• API documentation, describing the interface of each public function

Of the above list, the first two items are already in place. As per the API documentation, I see two possible ways to embed it into source files:

1. Special purpose comments. This is the style adopted by Java, where you have a block commend annotating an element with the relevant documentation.
2. Docstrings. The Lisp/Python way. Definitions embed a string of documentation.

In my opinion, the first approach is better in our case, since it will allow to annotate registerOption invocations, so to document options defined modules.

Lilypond already has block comments, delimited by %{ %}. I propose to use marked block comments for documentation:

%{
This is a normal block comment
%}

%{!
This is a *documentation* comment.

  - it may be in Markdown
  - so we have nice formatting both in source code and in html
%}
someDefinition = 

Notice the ! in the documentation comment to distinguish it from a normal block comment.

I am working on the implementation of a parser with python-ly to parse such comments. I should have a preliminary version ready for the week-end. However, I'd like to know your ideas about the documentation syntax.

[uliska]

I'm opening this issue to start the discussion on documentations syntax.

Thank you very much, that's great.

I think that there are three kinds of documentation:

• in depth documentation, contained in READMEs, Markdown files etc.
• annotated usage examples
• API documentation, describing the interface of each public function

Of the above list, the first two items are already in place.

Well, more or less. I think there still is room for discussion and
decision. Let me outline a few ideas but then move them to another issue
(which I'll open) because I think this issue thread will concentrate on
the other part.

We already have README files. These are particularly great because they
integrate so nicely with Github's web interface. But currently there is
no notion of a consistent way to author, structure and format them.

Then we have the usage examples. These have the benefit of producing
scores that can be made useful for documentation. But here again, this
isn't codified in any meaningful way. There are usage examples (and I'm
talking about the ones in the new structure as well as those in the old
one) that are just that: examples while others are tutorial style
references. Both is valuable but there should be a clear idea which part
of documentation will be used for which target, and from there it should
be possible to have some consistent way of inputting documentation.

Finally we have the "metadata" - the information stored in the
\declareLibrary and \declareModule commands (in the new structure) or in
the oll- \header fields (in the old structure). This also has to be
discussed.

In a nutshell: We have to identify the places where such documentation
can be entered, the potential targets where the information is later
used, and a consistent scheme of where to enter which information in
order to avoid redundancy.

As per the API documentation, I see two possible ways to embed it into
source files:

1. Special purpose comments. This is the style adopted by Java, where
   you have a block commend annotating an element with the relevant
   documentation.
2. Docstrings. The Lisp/Python way. Definitions embed a string of
   documentation.

Yes.

In my opinion, the first approach is better in our case, since it will
allow to annotate |registerOption| invocations, so to document options
defined modules.

I agree with that because that will allow a consistent interface for
both LilyPond and Scheme definitions.

Lilypond already has block comments, delimited by |%{ %}|. I propose
to use marked block comments for documentation:

%{
This is a normal block comment
%}

%{!
This is a documentation comment.

• it may be in Markdown
• so we have nice formatting both in source code and in html
  %}
  someDefinition =

Notice the |!| in the documentation comment to distinguish it from a
normal block comment.

Yes, the modified block comment should be the way to go.
I'd like to have a few more opinions about the concrete idea of the
exclamation mark. I don't see a problem with that, but I'd like to be
sure that this won't cause any trouble with anything else the LilyPond
parser might have to deal with now or in the future.

And in addition we have to discuss which "fields" these comments support
and in which way they can be entered.

I am working on the implementation of a parser with |python-ly|
https://github.com/wbsoft/python-ly to parse such comments. I should
have a preliminary version ready for the week-end. However, I'd like
to know your ideas about the documentation syntax.

This would of course be terrific!
Just one comment: If you think about using python-ly for that we should
ensure that any effort would be harmonized with Frescobaldi's development.
I am thinking of this API documentation not as an isolated openLilyLib
thing. It should rather become a specification for anybody to "comment"
his LilyPond code. And if such comments are present editors like
Frescobaldi should be able to make use of that information for things
like code completion (with extended information about function
signatures for example) and syntax highlighting.
An aspect that should be considered in that context is if the parsing
should be done strictly in the context of a file or if efforts should be
taken to follow includes and parse the whole document.

[Cecca]

I tried to implement this proposal using python-ly without success, and it took me much longer than the couple of days I initially planned :-)

The approach I was trying to use was to extend the parsers provided by python-ly to recognize the documentation syntax. However I found it difficult to develop a parser that could distinguish between top-level name declarations preceded by documentation comments from other combinations of words with the = sign. This is mainly due to the fact that in python-ly parsers the grammar specification is defined implicitly by how parsers are combined, and it is coupled with the parsing process itself.

Another route to explore may be to use a parser generator the takes as input a grammar, like grako. Unfortunately, this is something that I will not be able to do in the short term, but we can add it to our to-do list for the future :-)

[uliska]

I have now discussed the issue of block comments in
http://lists.gnu.org/archive/html/lilypond-user/2015-07/msg00071.html,
and it seems like the %{! %} approach is a viable approach.

The approach should be to read the file, and when such a comment is encountered read that and "apply" it to the next definition. I think the number of possible cases following such a doc comment is sufficiently small to be handled properly.

Within .scm files the syntax could be #!! !#, corresponding to GUILE's block comment syntax with an extra !. But here one could argue that it is better to stick with the ordinary docstring inside the definition. I'm not sure about that: what is better: being consistent with "our" LilyPond doc-comment syntax or with the usual Scheme docstring style. And would one or the other be significantly more difficult to parse? I mean how would a parser determine if a given Scheme-like docstring can be considered a lilypond-doc comment? OTOH parsing regular docstrings would probably "catch" a great number of already existing docstrings.
Probably the best thing is to specify a new, dedicated block comment format but try to parse docstrings too.

Next thing to discuss would be the selection of supported "fields" and a proper syntax that is at the same time good to write, read, and parse.