Document-level rendering?

[agoose77]

One of the issues with our current approach is that we handle each cell very much in isolation. MyST cannot perform forward references with this limitation.

One option is to hack more deeply into the notebook, and allow a full-notebook render to be performed. I'm not really thinking the ramifications of this through too much yet, but I already spot some problems:

• A full re-render would be slow for big notebooks
• A partial re-render (parse whole doc, but only re-render cell) would be faster, but lead to out-of-sync issues.

Maybe there is something we could do at a different level to allow plugins to declare that they need more information / have dependencies. I'm just rambling at this stage.

Or, we take the easy way out and say that this is a limitation.

[agoose77]

The big issue here in the MyST context is a hybrid of JupyterLab constraints and markdown-it / MyST constraints

The big "missing feature" is that references are document-level, but right now MarkdownIt is handled per-cell. If we have a per-document renderer context, then we fix the per-cell problem, but then have state invalidation to deal with.

Ultimately, as a UX, I'd like:

• references to work (at the document level)
• partial renders to work (updating a single cell doesn't cause a total reflow)

In technical terms, this would mean:

• the ability to parse each cell into an AST.
• this AST can then be transformed by extensions
• one extension can declare a dependency upon a part of the AST
• this dependency tree is used to perform partial re-renders.
• rendering is stateless; all contextual information handled by AST transforms.

In practical terms, I think this probably means ultimately moving away from markdown-it in favour of something like the unifiedjs ecosystem. We'd also need to modify how JLab invokes the rendering mechanism, namely that it cannot be entirely stateless from the markdown perspective - the mdast renderer could be, though. This might be as simple as bypassing the markdown rendering mechanism 🤮 and registering an mdast renderer. Then the JLab integration can also speak mdast, but we take charge at the notebook level of getting that mdast from the markdown source.

I wish this were trivial 😆

[bollwyvl]

I think to get document-level features to work, the whole authoring experience needs to be rethought, not (just) the parsing/rendering one. For a few years, I've thought ProseMirror is the still the likeliest candidate, and JupyterLab 4 will probably be the right timeline, for CodeMirror 6 and simplified focus for RTC (just yjs).

For example, in a notebook, Instead of JSON cells that sometimes have markdown, the top-level document would be a prosemirror node (nbformat:notebook) that only allowed nbformat:metadata and nbformat:cells at the top, with subsequent, extensible schema structures underneath, each with a dedicated UI.

This would mean most of the time, editing happens against directly properties on nodes, and round-tripping to nbformat JSON (and markdown source) would happen only at hard save points.

One side effect of this might be a custom collaborative editing schema rather than moving the nbformat json schema, or raw text in the editor case, over CRDT.

The UI/DX of a notebook would play out where, much like a google doc, a cherry-picked metadata (title, author, etc) would optionally appear at the top, and unlike the current notebook ux, would start with an empty markdown cell, where tab would change it to a code cell, starting an embedded CM6 UI.

None of this changes #13, however... so there's still very much a need for a robust way to document the intent of the document owner vs whatever opinions their tool author had today. But taking over the whole document, it will be a lot more natural to own the metadata as well.

Done right, the user input and output result is quite pleasing, though:

• https://github.com/benrbray/prosemirror-math (leaving aside the whole katex/mathjax issue for now)

And composable with "big rocks" already being adopting in JupyterLab 4

• https://github.com/sibiraj-s/prosemirror-codemirror-6