Generate Diagrams in Parallel

[KartikSoneji]

At the moment, it looks like the extension generates diagrams as it comes across them, which can get quite slow.
Is it possible to generate diagrams in parallel with a pool of n workers?
That would greatly speed up generation for documents with many diagrams and a slow processor like mermaid.

Not sure if it would need changes to the asciidoctor extension system, but either way I'll be more than happy to help.

[pepijnve]

The extension system isn't designed for asynchronous operation as far as I know. The callback hooks are all synchronous. It might be possible to hack together a solution that achieves some degree of asynchrony, but I think this would be at the cost of reliability.

@mojavelinux wdyt?

[mojavelinux]

In theory, it would be possible to replace the diagram with a marker, then start working on those markers in the background. The caller would have to force a join when the document is done converting. But this is unlikely to help except for the specific circumstances when the document is extremely large.

I think a better approach would be to separate diagram generation from processing altogether if you want to be able to work in this mode. That would not be the behavior/responsibility of this extension (as I would view it to be out of scope), but rather a different one that makes use of this extension for that purpose/workflow. In other words, you would need to think about separating diagram generation from AsciiDoc conversion entirely.

[KartikSoneji]

Except for inline images, the actual generation is independent of the markup, right?
Can't the extension return the markup in those cases at least?

[pepijnve]

Perhaps we could make a companion post processor extension to handle this. When it's loaded, the diagram extension goes through the usual motions, but instead of actually producing the images it collects all the "production instructions". Then in the post processor you execute those instructions in parallel. The post processor could still be synchronous by waiting for the work queue to be completed entirely.

PRs welcome ;)