Init logger in a Python API library

[vsbogd]

PR fixes issue with logs absence in debug releases. The root cause is that ctor is not called when C API static library is built in debug configuration. I am calling init_logger manually from C library to fix it.

I opened mmastrac/rust-ctor#280 to ask ctor author for help in investigation.

[luketpeterson]

This change looks reasonable enough. But I would like to hear what @mmastrac (ctor author) has to say about the root cause. It looks like he's very responsive so my guess is we'll hear back soon.

[luketpeterson]

Does it make sense to invent a persistent object for all Hyperon Rust clients? I.e. a runtime object that must be created to use the library? Python already has this, and you're using it to initialize the logger there. But Rust and C clients need to depend on ctor or do their own logger initialization.

I think we might need something like a runtime object to own the thread pool for a multi-threaded interpreter. Also I was considering turning all symbol atoms into hashes for performance, and keeping the strings themselves in a central store. Things like that.

If we had such an object, that would give us a place to initialize the logger without relying on ctor.

[vsbogd]

Does it make sense to invent a persistent object for all Hyperon Rust clients?

Luke, do you have any examples of this used in mature Rust libraries? I am not crazy of having global singletons except some specific cases. Even with logger it definitely has drawbacks, for example one cannot properly configure logger from code from the very beginning. But maybe in Rust there is no other way when one need to initialize some global constant object before program started. For example it is the reason why reserved symbol atoms are const instead of static: because I didn't found an easy way to initialize them beforehand.

Anyway I would not create some global object which keeps all what we need at once. I would instead introduce specific APIs for each case and use it to initialize storages. For example address of the string could probably be used instead of its hash and it would not require a separate store. Thread pool could be a part of interpreter and initialized with its instance. Maybe there are cases where we cannot eliminate the global storage. So it is also interesting to see how this problem solved in mature Rust libraries.

[ngeiswei]

I cannot comment on the design of that fix but I can confirm that it works for me.

[luketpeterson]

TLDR: this change is good to merge, IMO. But longer-term, I believe we should remove the logger init from the ctor hook, and try to put it somewhere else.

More details:

do you have any examples of this used in mature Rust libraries?

Many embedded language crates use a top-level runtime state object (often a singleton):

• JavaScript (V8 engine): https://docs.rs/v8/0.73.0/v8/V8/fn.initialize.html
• Python: https://pyo3.rs/main/features.html#features-for-embedding-python-in-rust
• Lua https://docs.rs/mlua/0.8.9/mlua/#the-lua-object

However, I agree with you that a singleton might not be the best design. All of those above examples are rust wrappers around projects with non-rust implementations, and their decision might have been involuntary, rather than a well-considered design choice.

I think the Lua case is the closest to what I have in mind. There is a top-level object but it’s not a singleton.

Hyperon-experimental has the runner::Metta type, but the big difference is that it isn’t created first. Instead, it is created with a Space (and a Tokenizer), which might involve a lot of work to create them in advance. More generally, the current design lets the caller create Atoms and Spaces by themselves without reference to any runner object. (‘static lifetime)

This might end up limiting the optimization that are available to us - although we could solve it like V8 solves it, by creating wrappers for objects under the management of a runner (ie. something like https://docs.rs/v8/0.73.0/v8/trait.Handle.html)

Anyway, this is a bigger discussion, and I apologize for getting us off-topic with regard to logger initialization. If we ultimately moved in this direction, top-level object creation would be a logical place to init the logger.

My final point about the logger is that running the initialization from ctor means that another app that wants to embed the MeTTa interpreter can’t initialize their own logger from their own main (if they wanted to use fern (https://docs.rs/fern/0.6.2/fern/) for example), because the MeTTa logger would be initialized first. Therefore, my preference would be to give the caller an option to do stuff from main() before MeTTa installs its logger.

[vsbogd]

Thanks Luke, on the first glance Lua example is clear, but I think I need to look at the API of the library closer to understand how it works in details.

I am curious why Rust has a such restricted conception of a static read-only object.As far as I remember static can be initialized by const functions only and const functions mean one cannot allocate memory.

[vsbogd]

I used ctor mainly because I didn't want to initialize logger in each unit test. On Java platform for example it is common to initialize logger outside of code of the application by loading specific logger backend library and that is flexible and very convenient. Do you know the similar manner on a Rust platform? The best way I found is https://crates.io/crates/test-env-log but anyway one need to edit each test in order to enable logger. If you can share logger best practices in Rust I would really appreciate it.

[vsbogd]

Merging it as a quickfix as this functionality is important, but I am ready to discuss a proper fix further.