Tracking issue for config (deserialization)

[SpriteOvO]

I'm considering adding a new feature providing the ability to serialize and deserialize for loggers, that allows users to load logging configs from external inputs (particularly from files) so that they can save the existing configs and/or modify configs dynamically from an external file.

The overall idea is, we depending serde as an optional dependency, and then we derive/implement Serialize & Deserialize around all stuff (e.g. All formatters, sinks, and the Logger)

There are some unsolved issues:

• What is the best way to serialize trait object types? Should we implement it ourselves or just use erased-serde crate directly? (I have no experience with this crate)
• How do we handle user-defined sinks and formatters?
• ...

So this is a discussion issue, anyone could leave your opinion or concern :)

---

Steps / History

• Implement RuntimePattern #45
  • Tracking issue for runtime_pattern! macro #55
• Implement config feature #47

[SpriteOvO]

@Lancern Do you have any experience to share about this?

[Lancern]

What's the use case? It sounds a little strange to serialize a logger / formatter / sink since these concepts are abstractions for behaviors rather than data.

[SpriteOvO]

What's the use case? It sounds a little strange to serialize a logger / formatter / sink since these concepts are abstractions for behaviors rather than data.

Sorry I didn't make that clear enough. Precisely, we are adding a feature that allows users to load logging configs from external inputs (particularly from files) so that they can save the existing configs and/or modify configs dynamically from an external file. I guess this is a common use case for logging libraries.

[Lancern]

This use case is perfectly valid. However IMO implementing Serialize and Deserialize directly for formatters, sinks and loggers is not a proper way to achieve this. As I said earlier, these concepts are abstract models of "runtime behaviors", and how could "runtime behaviors" be serialized and deserialized? It just doesn't make sense.

The key point here is that although formatters cannot be serialized and deserialize, their "internal states" might could. So I think the proper way to achieve this would be:

• Adding functions to get (and set?) the "internal states" of formatters, sinks and loggers;
• Serialize and deserialize those internal states.

I'll sketch a draft interface design for this later.

[Lancern]

Take formatters as an example, I propose the following interface. Note that all names are not well-considered and are subject to change.

First, we add a new trait SerializableFormatter that represents a "serializable" formatter (i.e. a formatter with some "serializable internal states"):

pub trait SerializableFormatter : Formatter {
    type State: Deserialize + Serialize;

    fn state(&self) -> Self::State;
    fn set_state(&self, state: Self::State);
    fn new_with_state(state: Self::State) -> Self;
}

The state associate function gets the internal states of the formatter. The set_state function sets the internal states of the formatter (in a thread-safe way). The new_with_state function creates a new formatter with the given internal states. Type of the internal states can be specified through the State associate type.

With this interface in mind, we can save and load formatter configurations from JSON files with the following functions: (error handling code is omitted for simplicity)

pub fn save_formatter_state<F, P>(formatter: &F, config_file: P) -> std::io::Result<()>
where
    F: SerializableFormatter,
    P: AsRef<Path>,
{
    let state = formatter.state();
    let state_json = serde_json::to_string().unwrap();
    std::fs::write(config_file, state_json)?;
    Ok(())
}

pub fn load_formatter_config<F, P>(config_file: P) -> std::io::Result<F>
where
    F: SerializableFormatter,
    P: AsRef<Path>,
{
    let state_json = std::fs::read_to_string(config_file)?;
    let state = serde_json::from_str(&state_json).unwrap();
    let formatter = F::new_with_state(state);
    Ok(formatter)
}

[SpriteOvO]

However IMO implementing Serialize and Deserialize directly for formatters, sinks and loggers is not a proper way to achieve this. As I said earlier, these concepts are abstract models of "runtime behaviors", and how could "runtime behaviors" be serialized and deserialized? It just doesn't make sense.

Agree with that. Maybe we could reuse {Logger,Sink,Formatter}Builder for serialization? They have all the parameters to build their target.

---

Basically, I want users to use this feature as easily as possible, I am considering this solution:

fn main() -> Result<(), _> {
    spdlog::registry().load_from_file("config.json")?;

    module_network();
    module_gui();

    spdlog::registry().save_to_file("config.json")?;
}

fn module_network() -> Result<(), _> {
    let network = spdlog::registry()
        .logger_or_build(
            "network",
            // The `builder` has name `network` is set.
            |builder| builder.sinks([sink1, sink2]).level_filter(_).build()?
        );

    info!(logger: network, "downloading...");
}

fn module_gui() -> Result<(), _> {
    let gui = spdlog::registry()
        .logger_or_build(
            "gui",
            // The `builder` has name `gui` is set.
            |builder| builder.sinks([sink1, sink2]).level_filter(_).build()?
        );

    info!(logger: gui, "user clicked the button");
}

As the code shows, we also have to introduce a new thing Registry, which potentially requires changing LoggerBuilder::build() to return Arc<Logger> instead of Logger since we need to store loggers inside the registry for future acquire.

The idea referenced from log4rs - Configuration via a YAML file and C++ spdlog - Logger registry, I'm not sure if it's a little overkill, looking forward to better ideas.

[Lancern]

How to support custom sinks and formatters? Registry does not know their types when loading from configuration files.

[SpriteOvO]

How to support custom sinks and formatters? Registry does not know their types when loading from configuration files.

Maybe something like this?

spdlog::registry()
    .register_custom_sink("CustomSink", |args: String| -> Result<Box<dyn Sink>, _> {
        let builder: CustomSinkBuilder = serde::from_str(&args)?;
        Ok(Box::new(builder.build()))
    });

[Lancern]

LGTM, but the interface design needs to be discussed further. Maybe we can open a draft PR and discuss there?

[SpriteOvO]

LGTM, but the interface design needs to be discussed further. Maybe we can open a draft PR and discuss there?

Okay, I'll make a draft PR for the initial later ^^

[SpriteOvO]

This is the initial design of the config file, IMO, TOML is the best format for this, YAML is probably fine, but the indentation sensitivity is a bit verbose to me.

# configure the default logger
[default]
sinks = [
    { name = "std_stream_sink", level_filtter = "all", style_mode = "auto" },
    { name = "rotating_file_sink", base_path = "/var/log/my_app/app.log", rotation_policy = { daily = "12:00" } },
    { name = "win_debug_sink", formatter = { name = "pattern_formatter", pattern = "[{level}] {payload}" } }
]
flush_level_filter = "all"
flush_period = "10s"

# configure named loggers "network"
[network]
sinks = [
    { name = "rotating_file_sink", base_path = "/var/log/my_app/err.log", rotation_policy = { file_size = "10M" }, level_filtter = "error" }
]
flush_level_filter = "warn"

# configure named loggers "gui", unnamed loggers, and the rest of loggers
["gui,unnamed,*"]
sinks = [
    { name = "file_sink", path = "/var/log/my_app/misc.log", level_filtter = "all" },
    # user-defined sink/formatter must start with "$" and are registered in Rust code
    { name = "$custom_sink", some_arg = "xxx", level_filtter = "all", formatter = { name = "$custom_formatter", some_arg = "yyy" } }
]
flush_level_filter = "all"

[Lancern]

LGTM, but a small problem on naming convention: what about lower-letter rather than lower_letter? Cargo.toml uses the former.

[SpriteOvO]

LGTM, but a small problem on naming convention: what about lower-letter rather than lower_letter? Cargo.toml uses the former.

I'm fine with the suggestion for keys, just a #[serde(rename_all = "kebab-case")] will do.

About values (e.g. = "std_stream_sink"), I now prefer that it should be consistent with the struct name. Because this part may involve manual deserialization, naming it consistently with the struct allows us to avoid the burden of renaming.

And the logger options will now be under the "logger" key, so that we can reserve the root for future use for configuring global options.

---

So it looks like this now, what do you think?

# configure the default logger
[logger.default]
sinks = [
    { name = "StdStreamSink", level-filtter = "all", style-mode = "auto" },
    { name = "RotatingFileSink", base-path = "/var/log/my-app/app.log", rotation-policy = { daily = "12:00" } },
    { name = "WinDebugSink", formatter = { name = "PatternFormatter", pattern = "[{level}] {payload}" } }
]
flush-level-filter = "all"
flush-period = "10s"

# configure named loggers "network"
[logger.network]
sinks = [
    { name = "RotatingFileSink", base-path = "/var/log/my-app/err.log", rotation-policy = { file-size = "10M" }, level-filtter = "error" }
]
flush-level-filter = "warn"

# configure named loggers "gui", unnamed loggers, and the rest of loggers
[logger."gui,unnamed,*"]
sinks = [
    { name = "FileSink", path = "/var/log/my-app/misc.log", level-filtter = "all" },
    # user-defined sink/formatter must start with "$" and are registered in Rust code
    { name = "$CustomSink", some-arg = "xxx", level-filtter = "all", formatter = { name = "$CustomFormatter", some-arg = "yyy" } }
]
flush-level-filter = "all"

[SpriteOvO]

Also, I plan to just implement deserialization and drop serialization, based on

• If we implement serialization we need more effort to maintain a logger registry, and if we drop it we can just deserialize to a Config struct and then users can obtain what they want from it.

• Rethinking, it doesn't seem to make sense to save a programmatically built/modified logger to a config file at runtime.

[Lancern]

So it looks like this now, what do you think?

Maybe change logger to plural and renaming logger to loggers?

[loggers.default]
# ...

[loggers.network]
# ...

[loggers."gui,unnamed,*"]
# ...

I plan to just implement deserialization and drop serialization

Agree. Serializing a logging configuration at runtime to a file just doesn't make much sense, I couldn't imagine a scenario in which this is useful.