PDML Syntax Thoughts

[tajmone]

@pdml-lang, I wanted to share with you some thoughts on the PDML package, its design and potential features.

Most of the considerations found here apply to any modern editor for which we could build a PDML syntax.

Basic vs Advanced Scoping

In my first Syntax draft, I tried to treat the PDML syntax as JSON, using the meta scopes for key-value mappings. This is how the syntax test looked like in commit cc37bbf:

! SYNTAX TEST "Packages/PDML/PDML.sublime-syntax"

  [config
! ^^^^^^^  meta.mapping  meta.tag  meta.mapping.key
!  ^^^^^^  entity.name.tag.localname
! ^        punctuation.definition.tag.begin
    [color orange]
!   ^^^^^^^^^^^^^^  meta.mapping  meta.mapping
!   ^^^^^^          meta.tag  meta.mapping.key
!                ^  punctuation.definition.tag.end
    [size
      [width 1618]
      [height 1000]
    ]
  ]
! ^ meta.mapping  punctuation.definition.tag.end

The scopes meta.mapping, meta.mapping.key and meta.mapping.value were borrowed from the way the native JSON syntax uses them.

Initially, I started looking at how ST handles JSON, XML and YAML, in order to gain insight on how these different syntax are handled, since they all relate to PDML somehow.

Below are the summary findings on how these three syntaxes get scoped, with some comments on the pros and cons of each approach.

Native JSON Syntax Example

This pseudo-test illustrates how ST's natively scopes JSON:

// <- source.json
{
// <- meta.mapping
// <- punctuation.section.mapping.begin
    "DB_config": {
//  ^^^^^^^^^^^     meta.mapping.key + string.quoted.double
//  ^         ^     punctuation.definition.string.(begin/end)
//             ^    punctuation.separator.mapping.key-value
//               ^  meta.mapping.value + meta.mapping
//               ^  punctuation.section.mapping.begin
        "provider": "MySQL",
//      ^^^^^^^^^^            meta.mapping.value + meta.mapping.key
//                  ^^^^^^^   meta.mapping.value + meta.mapping.value
//                         ^  punctuation.separator.mapping.pair

Note how nesting works: complex values are always inside meta.mapping.value, since they are part of the value of their parent key, so meta.mapping.* scopes stack up.

I'm not entirely sure what the practical benefits of these scopes are, but I guess that they allow advanced manipulation via plug-ins, finer control over auto-completions, snippets, etc., and possibly affect interactions with settings (these being the two reasons and goals behind meta scopes).

In any case, JSON (actually jsonc, JSON with Comments) is one of ST's natively employed settings formats, so I'm pretty sure there are valid reasons behind these scopes — and probably by looking into the ST native plug-ins we'll see them referenced too.

Native XML Syntax Example

XML is treated quite differently from JSON, as you can see from the pseudo-test below:

<!--  text.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<!--
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  meta.tag.preprocessor
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  punctuation.definition.tag.begin
-->
<books>
<!--
^^^^^^^  meta.tag
 ^^^^^   entity.name.tag.localname
^     ^  punctuation.definition.tag.(begin/end)
-->
    <book>
        <isbn>978-0135957059</isbn>
<!--          ^^^^^^^^^^^^^^         text.xml -->

There isn't a distinction between opening and closing tags, and tags contents don't receive any special scoping either (just text.xml).

Also interesting to note that, unlike JSON which receives the source.json base scope, XML is treated as text (text.xml).

Native YAML Syntax Example

Finally, YAML, where keys are scopes as tags, their names as unquoted strings, and values depending on their types (string, numbers, constants, etc.).

---
#<- entity.other.document.begin
- hosts: webservers
#<-                  punctuation.definition.block.sequence.item
# ^^^^^              entity.name.tag
# ^^^^^              string.unquoted.plain.out
#      ^             punctuation.separator.key-value.mapping
#        ^^^^^^^^^^  string.unquoted.plain.out

  vars:
    http_port: 80
#              ^^    meta.number.integer.decimal constant.numeric.value
#              ^^    constant.numeric.value

The interesting part of YAML is how scopes variations are employed to handle different types of strings:

- str1: |
#       ^         keyword.control.flow.block-scalar.literal
    Lorem ipsum.
#   ^^^^^^^^^^^^  string.unquoted.block
- str2: "Lorem."
#       ^^^^^^^^  string.unquoted.double

which I guess has practical applications in ST settings and plug-ins, since YAML is one of the formats natively used by ST.

PDML Scoping...

Ultimately, after the initial tests I opted to keep things simple and just stick to focusing on the actual nodes, scoping them as tags and tracking their opening and closing brackets, ignoring their contents altogether. This still allows ST to correctly match each opening tag with its closing ], which means that users can easily control their selection via the predefined key-shortcuts to expand or shrink a selection to include the parent node, or shrink down the tree, which is pretty useful, along with usual native features like jumping to the beginning or end of a tag, etc.

By keeping it simple, it will be very easy to create PDML package for other editors, e.g. VSCode, etc. The simple syntax boils down to just a few lines:

variables:
  node_id: '[a-zA-Z_][a-zA-Z1-9_\.-]*'

contexts:
  prototype:
    - include: escapes

  main:
    - include: pdml_nodes


  pdml_nodes:
    - match: '(\[)({{node_id}})'
      captures:
        0: meta.tag.pml
        1: punctuation.definition.tag.begin.pml
        2: entity.name.tag.localname.pml
      push: node_inside
  node_inside:
    - include: pdml_nodes
    - match: ']'
      scope: punctuation.definition.tag.end.pml
      pop: true

And if we were to use anonymous contexts and drop the variables, it would probably boil down to 10 lines of YAML in total. ST can even export a syntax to the old XML-based format or JSON, which should be usable in TextMate and VSCode, respectively, with minor adjustments.

But the reason I brought up the PDML vs JSON, XML and YAML comparisons was to focus on the hybrid nature of PDML, which can be used as a key-value mapping syntax, along the lines of JSON and YAML, or a markup syntax like XML.

So, I was wondering whether there could be benefits in adopting the meta.mapping scopes when dealing with key-values, and in that case how to handle valueless keys. But then this would introduce some complications, like the whitespace handling issues mentioned in pdml-lang/basic-specification#2 — and right now we might not want to get entangled with any of that...

Invalid PDML Elements

I have a question regarding elements outside the PDML tree, which is not addressed in the Specs or documentation.

It seems reasonable to expect that any .pdml document should contain only PDML nodes, which begs the question of how to handle any stray contents before and after the root node. E.g.

Leading noise.
[pdml root
  [tag1 blah blah]
]
Trailing noise.

Should the text Leading noise. and Trailing noise. be ignored by the PDML syntax? or should they be scoped as invalid? While scoping them as invalid might seem an obvious solution, bear in mind that the syntax should be tolerant for whitespace noise — e.g. leading and trailing new lines, since these might be found in a .pdml document due to editor/IDE settings, or the presence of an .editroconfig file.

So it might be preferable to ignore whitespace noise, but could capture anything else preceding and following the root node.

Another useful addition would be to intercept stray brackets — e.g. a closing ] before the root node, or an unmatched ] following the PDML tree.

I'm not sure that it would be possible to visually distinguish between invalid characters noise and stray brackets though, since the invalid scope is only guaranteed to cover two different cases in colour schemes: invalid.illegal and invalid.deprecated, where only the former applies to PDML.

Beyond Basic PDML: Extensions via Syntax Inheritance

Another reason to keep the Basic PDML syntax simple and straightforward is that I was thinking that when new officially sanctioned PDML extensions are added to PDML, we could include them in the ST package too.

Although *.pdml files will always be associated with the Basic PDML syntax, the package could also provide additional syntaxes extending the basic one, which users could select manually via the command palette (Set syntax as PDML >some extension name<, etc.).

ST4 introduced a cool new syntax feature: inheritance:

In situations where a syntax is a slight variant of another, with some additions or changes, inheritance is a useful tool.

When inheriting a syntax, the key extends is used with a value containing the packages path to the parent syntax. The packages path will start with Packages/ and will contain the package name and syntax filename. For example:

%YAML 1.2
---
name: C++
file_extensions: [cc, cpp]
scope: source.c++
extends: Packages/C++/C.sublime-syntax

A syntax using inheritance will inherit the variables and contexts values from its parent syntax. All other top-level keys, such as file_extensions and scope will not be inherited.

Thanks to this, it will be very easy to include support for any number of PDML extensions in the package. I was thinking that official PDML extensions should be included to this package, whereas third party extensions could be packaged separately, and users could install them according to need.

Of course, end users can also create a custom extension of the PDML syntax locally, in their User/ subfolder of ST packages directory, yet still leverage inheritance to extend the PDML Basic syntax (or any other official extension being supported).

Extended Features

We briefly mentioned en passant about adding to PDML packages for editors some extra functionality like prettifying or minifying the Tree, serializing to/from JSON, XML, YAML, and other useful stuff which might render these packages more appealing than just providing basic syntax highlighting for PDML documents.

In ST, plug-ins are written in Python. It's been quite a while since I had a brush with Python, and I'm not entirely sure how custom plug-ins can integrate third party Python modules (e.g. to parse/serialize JSON, XML, etc.) into a ST package, but my guess is that you can't rely on pip since ST uses the bundled Python runtime and not a full Python installation on the user machine.

ST does provide its own API, but I have no idea if this include JSON, YAML and XML serializing, although these formats are used in the various settings files natively supported by ST — I'll need to check that, for any serializing built-in functionality could come handy here.

Unfortunately, any work in Python won't be reusable on VSCode, which uses JS/TypeScript instead, which kind of sucks really.

Anyhow, I would like to ear your ideas and thoughts on potential uses of editors' plug-ins for managing PDML documents, since you have a clearer vision of where PDML is heading for in the future.

[pdml-lang]

I opted to keep things simple and just stick to focusing on the actual nodes, scoping them as tags and tracking their opening and closing brackets, ignoring their contents altogether.

Good!

By keeping it simple, it will be very easy to create PDML package for other editors, e.g. VSCode, etc.

Yes!

I was wondering whether there could be benefits in adopting the meta.mapping scopes when dealing with key-values, and in that case how to handle valueless keys.

I would suggest not to do that, unless you develop a plugin for a specific domain with known rules.

question of how to handle any stray contents before and after the root node.

Good question. I completely forgot to include this point in the specification. Will do that soon.
The answer is that any whitespace (i.e. spaces, tabs, carriage returns and line feeds) before and after the root node must be ignored, because, as you said: "the syntax should be tolerant for whitespace noise".
However, any other characters preceding and following the root node are invalid.

Another useful addition would be to intercept stray brackets

Yes, these should be treated as invalid.

the package could also provide additional syntaxes extending the basic one

Yes.

BTW: It would be useful to assign a different scope/color to extension nodes (whose names are prefixed with !, e.g. [!foo is an extension node, whereas [foo is a standard node)

... which users could select manually via the command palette

Great!

ST4 introduced a cool new syntax feature: inheritance:

That's very cool, indeed! Supports the DRY principle.

I was thinking that official PDML extensions should be included to this package, whereas third party extensions could be packaged separately, and users could install them according to need.
Of course, end users can also create a custom extension of the PDML syntax locally, in their User/ subfolder of ST packages directory, yet still leverage inheritance to extend the PDML Basic syntax (or any other official extension being supported).

Awesome.

Unfortunately, any work in Python won't be reusable on VSCode, which uses JS/TypeScript instead, which kind of sucks really.

That's unfortunate, indeed. Maybe it's better to first provide this kind of extensions for VSCode, for the simple reason that more people use VSCode than ST.
User thecrumb puts it like this in an ST(!) forum:

Unfortunately for Sublime the editor world has moved on. If you look at any of the recent ‘developer’ surveys VSCode is the new shiny go-to editor. So it makes sense plugin developers have moved on as well.

[pdml-lang]

question of how to handle any stray contents before and after the root node.

Good question. I completely forgot to include this point in the specification. Will do that soon.

This is now specified.

[tajmone]

This is now specified.

That makes things easier, Still, in the Basic PDML syntax I'm not marking as invalid non-whitespace chars right now, since I want to see first how extensions will be implemented at the editor support level. For now the assumption is that the syntax only cares for the actual PDML Tree, ignoring the rest — although that's not entirely correct, since escapes and stray brackets are still being captures even outside the tree — although that's fairly easy to fix, I'd rather keep the syntax compact for now.

In any case, the package is now v1.0.0 and has been submitted to PackageControl and waiting to be added/rejected. So, no further changes until it's approved.

Multiple Trees in a Doc?

A PDML document is a tree of nodes.

Does this mean that there can only be one root node per document?

How should a second tree be handled? As an error (i.e. as text noise after the node), or should/could a parser consider its job done once it has reached the end of the root node, and quit parsing ignoring anything beyond that? (the latter would also justify tolerance for trailing Tree noise).

[pdml-lang]

the package is now v1.0.0 and has been submitted to PackageControl and waiting to be added/rejected.

Wow! That was fast. Great!
As soon as the package is available (please let me know) I will add a link on PDML's website.

Does this mean that there can only be one root node per document?

Yes. The Basic PDML Specification states:
"Each document has exactly one root node."
Note: I just changed "one root node" to "exactly one root node" to make this more clear.

One of the planned optional extensions will probably be to make the root node optional in the PDML document. In that case the parser creates a virtual root node named root.

In practice this can be useful to define several trees in one document, or for key/value documents (like config files). Instead of:

[config
    [width 200]
    [height 100]
]

... you can then simply write:

[width 200]
[height 100]

How should a second tree be handled? As an error

Yes, in Basic PDML it is an error.

or should/could a parser consider its job done once it has reached the end of the root node, and quit parsing ignoring anything beyond that?

This could be another option in a more tolerant parser.