Generating Image of Plot Automatically (Use in Reproducible Research with org-babel)

[yuki-tsubaki]

I new to the Lisp-Stat world and have generally loved the capability automatically generate and embed figures using Org-Babel in Emacs with other languages I have used for statistics/data analysis.

Goals

I want to find a method to be able to run a code block in my Org document that runs lisp-stat code that:

1. Plots a graph using the plot system (using vega-lite for example)
2. Automatically generates an image file of that plot (e.g. svg or png), as opposed to opening the plot in the browser as is the default behavior. (I could also see a function returning the HTML file path to be useful for some people who are exporting to a website who want to embed the interactive version.)
3. Returns the file path (which Emacs/Org-mode/Org-Babel can then use to embed in the file in my org document using the :results file header argument for the code block).

My Research So Far

When trying to research how to do this I wasn't able to find much of a clue on how I could achieve this. The mention of reproducible research that I did see was using Jupyter (which can be integrated with Org-Babel, but that sounds pretty roundabout and I prefer to have the SLY/SLIME being the one handling execution rather than Jupyter).

I have also found vl-convert which I am currently thinking of wrapping if there is not any more elegant solution (it pains me to think of calling a rust program with an embedded JS interpreter from CL just to get an image of the graph--still better than having to use those directly though :>).

Question

Does anyone know how do this? Even just some help figuring out how to get automatic rendering of the plot to a file would be much appreciated.

Thanks in advance.

[snunez1]

I believe the vega-lite image compiler does what you want. That page describes passing the specification to the vega-lite image compiler on stdin, which would be my first choice. Otherwise you'll need to write the spec to disk and then invoke the compiler on that file. I'm not familiar enough with emacs lisp to advise on how to execute external applications from slime, but I can see that it's possible.

There are a few options to get the JSON spec of the plot. Probably the easiest is to use the encode function and set the stream to be stdin of the vl2png function. Also see write-spec which will return the spec as a string that you can pass into vl2png. Finally, there's a sort of device abstraction that you might look at, intended mostly for internal use.

In short, there isn't something that does exactly what you want, but the pieces are there.

I'll leave this open. I'd appreciate a report when/if you get it working. Org-babel looks interesting; I might start using it myself.

[yuki-tsubaki]

Thanks for the information.

As an update on what I've been doing, I found vega/vl-convert which provides a program which embeds the vega-lite image compiler functionality (with the added benefit of not depending on node or a browser for the execution environment and embedded schema to facilitate offline usage).

It is good enough that I will probably write a wrapper for it.
The only reason I haven't gotten around to it being that I found an issue with handling of separate data files that that distracted me
Graphing using data in a separate file from a local path (rust executable) · Issue #189 · vega/vl-convert. To be clear it works well enough already just with the caveat that the data needs to be embedded in the spec for now.

It should be trivial to implement. It would just be simple wrapper around the plot-to-device that also takes image output parameters, runs vl-convert to convert/render a plot (would probably also have a keyword to enable async conversion for bulk stuff because vl-convert can be slow), and then return the output file path that vl-convert will be writing/has written to.

I'll keep updating here as I get things figured out.

And yes Org-babel is amazing: like Jupyter notebooks but more generalizeable to practically any language (or even multiple which you can make work together and pass values between in a single document), you get the full power of Emacs and Org mode, and more. In short, it just is so much more flexible for letting you use whatever tool you want/need to instead of constricting you one "official" interface and tool.

[Symbolics]

@yuki-tsubaki, how did you go with this? I'm about to start with org-roam and would like to try out lisp-stat in org-babel.

[yuki-tsubaki]

@Symbolics Sorry I've been a bit busy since I was looking at it. Basically I just made a wrapper around plot-to-device that saves the spec to a file, and then runs the vl-convert executable to generate the graphic from that spec. When that is done it should return the path to the image (which org-babel uses to display/embed the image). vl-convert can be a bit slow sometimes so I also made an option to switch between uiop:run-program and uiop:launch-program (depending on your Emacs setup and if you have it set to automatically render the image result when the code block finishes you might still prefer run-program version so the image is actually written by the time the code block returns).

It is really simple honestly. The only complication (and the reason why I haven't yet made a proper polished version yet) is that vl-convert does not yet support separate data files yet (there is an effort to add that in progress apparently so I was waiting (procrastinating) until that was done).

The basic solution does work now though if you are not too annoyed by the data set being embedded in each spec.

[Symbolics]

Good to know it works. I'd love to see a write-up or 'blog about how exactly to do this once you feel it's ready. This seems like a really useful addition to babel.