mirai development

[shikokuchuo]

Keeping track of some items for the pipeline. Also changes that may affect downstream packages.

To-do

2. Improve memory efficiency in mirai

• High memory usage when dispatching many targets ropensci/targets#1220
• threaded dispatcher

Deferred / not contemplated

• NNG changes requested (ETA uncertain) - try not to fork so can use system libraries

Completed

1. Move dispatcher control (monitor) sockets to "pair" protocol

• lighter and more efficient - prove that previous failures not due to the protocol
• status() failures (e.g. dispatcher OOM) will return to reporting errorValue 5 timeout rather than 11 incorrect state
• all public mirai and crew tests perform fine
• released in mirai 0.12.1
• reverted to req/rep in mirai 0.13.1 (for stability, other changes remain)

3. Improve memory efficiency in nanonext

• serialization - actually current implementation is efficient (taking into account compiler optimisation)
• allocation parameters tweaked to ensure not too much memory is reserved
• released in nanonext 0.13.0

4. Register serialization methods without submitting 'mirai' tasks

• to be released in mirai 1.1.1

[wlandau]

Thanks for logging this, @shikokuchuo.

Move dispatcher control (monitor) sockets to "pair" protocol [done]

What was the previous protocol?

please test as much as possible using mirai 0.12.0.9009 (75ddd5e)

Thanks for the FYI.

[wlandau]

From dcdc759, it looks like the previous protocol was req/rep. Is that right? Last time we talked about this, we thought req/rep would be more robust. Can the "monitor" sockets use this protocol safely? I am remembering old issues about local segfaults and dispatchers that keep running after the main R session exits.

Improve memory efficiency [to do]

For the sake of tracking things, I would like to mention here that it would be nice to be able to register serialization methods without submitting special tasks to do so (e.g. through everywhere() behind the scenes).

[wlandau]

We went through bus -> pair -> req/rep. At the time we were getting the hard to reproduce timeout errors on CI and thought it was to do with the protocol, but actually I believe those were garbage collection / object lifetime related. Hence I'm testing to move back to pair from req/rep.

Ah, you answered my question right as I asked! I will keep an eye on this.

[shikokuchuo]

Thx! Some further simplifications in mirai 0.12.0.9010 (a57cad7) Not big changes in and of themselves, but if they prove stable then potentially paves the way for bolder measures.

[shikokuchuo]

@wlandau mirai 0.12.1 released with those changes incorporated. Some more low level stuff being done in nanonext at the moment (summary above updated).

[shikokuchuo]

I've just broadened the serialization capability of {mirai} to handle Arrow or other similar objects (see apache/arrow#40231 (comment)). I haven't changed the way the functions are registered yet, but it should allow you to experiment.

[shikokuchuo]

@krlmlr inviting you to this topic as you contributed some helpful ideas in the past. Will and I discussed development plan for mirai today (not limited to the above, which are short term points). If you have any further suggestions, would be glad to hear them.

[krlmlr]

Thanks, Charlie, for the vignette. In my use case, I'd like to create multiple daemons (each holding a database connection) from my single R process (think a Shiny app). How does mirai support this?

[shikokuchuo]

@krlmlr I've added to that vignette a further example of a Shiny / mirai / DBI / ADBC stack:
https://shikokuchuo.net/mirai/articles/databases.html#shiny-mirai-dbi-adbc-integrated-example
Let me know if this is what you had in mind.

[krlmlr]

Thanks. This example still has daemons(4), initialized once. Can I use exactly one daemon per instance of the app? The server function would initialize the daemon, which would be destroyed after the user exits the app.

I understand that mirai uses a global pool of daemons. Is there a lower-level interface that allows interacting with a specific daemon? Perhaps it's launch_local(), but with a new argument that sends arbitrary startup code to the new daemon() created there?

Other crazy idea: daemons(Inf) 😱

[shikokuchuo]

Right, I see what you mean now! I've updated the vignette example https://shikokuchuo.net/mirai/articles/databases.html#shiny-mirai-dbi-adbc-integrated-example with the general solution (which uses a new compute profile for each session).

I say this is the 'general' solution as you could further customize each everywhere() call to open the database connection with session-specific credentials etc.)

There are other approaches if you just wanted to scale based on load e.g. by calling daemons(dispatcher = FALSE) outside of server() and then using the launch_ functions for each session. Key here is to not use dispatcher as daemons can then scale up and down without limit. Dispatcher is generally not required as the default round-robin behaviour is meant precisely for these webserver-like, production contexts where each query is expected to take roughly the same time.

[krlmlr]

Thanks, Charlie, for your help and patience. This looks like a viable solution, I'm keen to try it in a production project (albeit without ADBC, we have SQL server on the other end unfortunately).

[shikokuchuo]

@wlandau about moving serialization() away from an everywhere() call. I considered this and implementing it differently is awkward as it would mean specifying via the command line. This would likely entail the user having to specify the functions as a character vector (which is not intuitive), as it is not always able to deparse(substitute(.)).

[shikokuchuo]

There is now an interface for this: #120

I will polish it up and merge. Ended up deparsing but not sure how robust this is. Likely to be ok for the usual types of function we are dealing with, but may have to iterate down the line.

[shikokuchuo]

On second thoughts, the interface becomes rather awkward. You register the configuration per daemons() invocation but they apply globally. I think I'll wait for nanonext to update the implementation before I put in the interface in mirai.