Support comment event

[black-desk]

I want to write a formatter for djot. So, I want to keep the original comment.

But jotdown do not produce event for comment.

[hellux]

Yes, comments are mostly inaccessible from the events right now. I agree that
it would be useful for some applications (like your formatter) to have
representation for all parts of the input, including comments.

As comments are tightly coupled with attributes, one way could perhaps be to
add them to the attribute set. For example,

text{a=b %cmt% c=d}

currently yields

Start(Paragraph, {}) "" 0..0
Start(Span, {a="b", c="d"}) "" 0..0
Str("A") "A" 0..1
End(Span) "{a=b %cmt% c=d}" 1..16
End(Paragraph) "" 17..17

but could perhaps yield something like

Start(Paragraph, {}) "" 0..0
Start(Span, {a="b", %="cmt", c="d"}) "" 0..0
Str("A") "A" 0..1
End(Span) "{a=b %cmt% c=d}" 1..16
End(Paragraph) "" 17..17

where comments are a special type of attribute key-values.

Attribute keys are &'s str right now but we might need something like

enum Key {
    Comment,
    Attr(&'s str),
}

to distinguish comments from attributes.

Also, dangling attribute sets that are not attached to anything are currently
not represented by the events at all, e.g.

text {a=b %cmt% c=d} {%cmt only%}

currently yields

Start(Paragraph, {}) "" 0..0
Str("text ") "text " 0..5
Str(" ") " " 20..21
End(Paragraph) "" 33..33

but we actually want to access the comments here. We could perhaps add an empty
dummy object that just hosts the attributes in these cases, something like

Start(Paragraph, {}) "" 0..0
Str("text ") "text " 0..5
Empty {a="b", %="cmt", c="d"}) "" 5..5
Str(" ") " " 20..21
Empty {%="cmt only"}) "" 21..21
End(Paragraph) "" 33..33

Any thoughts/suggestions on how to represent the comments?

[black-desk]

I like the Empty design.

But consider:

A{% comment1 % .a .b % comment2 %}

Currently yields:

Start(Paragraph, {}) "" 0..0
Start(Span, {class="a b"}) "" 0..0
Str("A") "A" 0..1
End(Span) "{% comment1 % .a .b % comment2 %}" 1..34
End(Paragraph) "" 34..34

%= might not be enough here.

[hellux]

Attributes are internally implemented as Vec, so it is possible to have multiple comments and also maintain their order among the attributes.

However, Attributes are currently destructive for classes and key-value pairs as well, e.g.

• {.a .b} becomes {class="a b"} and
• {a=b a=c} becomes {a=c}.

For renderers it is convenient to get concatenated classes and only single attributes, but you lose information about the original source. Might be possible to preserve the original set completely (including comments) while also providing a way to iterate over only concatenated classes, single attributes and no comments.

Might need some experimentation with the API.

[black-desk]

Attributes are internally implemented as Vec, so it is possible to have multiple comments and also maintain their order among the attributes.

However, Attributes are currently destructive for classes and key-value pairs as well, e.g.

• {.a .b} becomes {class="a b"} and
• {a=b a=c} becomes {a=c}.

For renderers it is convenient to get concatenated classes and only single attributes, but you lose information about the original source. Might be possible to preserve the original set completely (including comments) while also providing a way to iterate over only concatenated classes, single attributes and no comments.

As for formatter, I think { a=b a=c } turns into { a=c } is good enough.

Might need some experimentation with the API.

[hellux]

I gave it a try with #56. I couldn't think of a good way to add comments in attributes with the current map-like structure so I tried turning it into a list/vec structure that more resembles the input, that preserves duplicate keys, comments and their order.

Any thoughts on this proposal?

[hellux]

Comments are available in events as of release 0.5.0.