Should we continue using custom Lua, or should we package the pallene_core as a library?

[hugomg]

Over the last couple of days I have been working on improving the Pallene installer. The goal is to allow installing the compiler via Luarocks and resolve long-standing issues such as #16, that one where we have to run pallenec from the root of the git repository. I'm planning to open a pull-request for that shortly but before I do, I want to gather some feedback...

The key question is how to install the Pallene standard library (the pallene_core stuff). To refresh your memory: the core library contains some low-level C functions used by the C code that we generate. Some of these functions are Pallene functionality we made, and some of these functions are internal PUC-Lua functions that normally aren't exposed in the C API. I've been exploring two alternatives:

1. Create a custom version of the Lua interpreter, that has this expanded C API (this is what we have been doing so far)
2. Package the Pallene standard library as an extension module, installed to the LUA_CPATH. The functions will go in a struct of function pointers, inside an userdata.

I have written a prototype implementation for both of these, but I can't decide which one is better. Here are some of the differences that I found between them:

• In option 1 users always have to install the custom Lua from source, as well as install Luarocks from source to match the custom Lua. In option 2, it might be possible to use the system version of Lua, provided that it is exactly the one Pallene asks for, including the "patch number". For example, Pallene is compatible with Lua 5.4.4 but not 5.4.0.

• In option 1 the installation process has two steps: compile and install the custom Lua via the makefile, and then install the Pallene compiler using Luarocks. In option 2, the installation process is a single step. The luarocks make installs both the Pallene stdlib and the compiler at the same time.

• Option 2 is slightly heavier (around 100KB extra) because the stdlib needs to copy some of the internal functions from the Lua interpreter. Option 1 doesn't duplicate the functions and by design ensures that the Pallene programs and the Lua interpreter are using the exact same implementation of the internal functions.

• In Option 1 we need sudo to install the custom Lua. In Option 2 we don't.

All things considered, I haven't been able to decide which of these options is the best one. Which one do we go with? Is there any benefit at all in allowing people to run Pallene on top of vanilla Lua? Or is it too impractical to ask for exactly Lua 5.4.4? What do you all think?

[hugomg]

After thinking some more, perhaps the best solution is a middle term:

• Create a custom Lua that exports the private functions
• But package the pallene library as a regular Lua module

Similarly to option 1, it has the advantage that we ship and update the pallenelib together with the pallenec, without having to reinstall the Lua every time. And similarly to option 2, it has the advantage that we do not duplicate the private Lua functions. This reduces the amount of bloat in the system and avoids any chance of having conflicting implementations of the internal functions.

[agladysh]

A thought: Some of "private" Lua functions may be detachable from the Lua to a separate module.

I used to do that here: https://github.com/lua-nucleo/lua-nucleo/blob/master/c/luainternals.c. It is probably not very clean C-wise, but, with certain luck, you might be able to it in a cleaner manner at the cost of some code duplication.

This is probably what you're talking about in the option 2.

Custom Lua interpreter might mess with the (non-Lua-initiated) OS package maintainer heads in the long run... Is the size a constraint at this moment?

[hugomg]

The size constraint is not the end of the world, given that it's just an extra 100 Kb. The thing that makes me a bit more nervous is having two copies of the internal functions running at the same time (one in the interpreter, one in the library) and if they are not exactly 100% the same then bad things can happen.

At the end of the day, I don't know which option is least bad... One one hand, having a custom Lua doesn't play nice with OS package maintainer people. On the other hand, putting the internals in a library results in a library that is extremely picky about which version of Lua that it is running on. (Sure, you can try running on your own Lua, but it better be exactly 5.4.4 and god help you if you change the internal ABI in any way).