support for multiple JSON line-formats and switching between them

[tstack]

From a comment on this post -- https://news.ycombinator.com/item?id=40718862

I do wish there was support for switching formats so I could switch between different "views" over the same data, maybe it will be possible someday :)

[vagran]

My case is that I am unable to make my format to be selected. It is actually pino format. I tried to add my custom format, like this:

{
    "$schema": "https://lnav.org/schemas/format-v1.schema.json",
    "pino_log": {
        "title": "Pino log",
        "json": true,
        "line-format": [
            {
                "field": "time"
            },
            " ",
            {
                "field": "__level__",
                "text-transform": "uppercase",
                "auto-width": true
            },
            " ",
            {
                "field": "msg"
            }
        ],
        "level-field": "level",
        "level": {
            "fatal": 60,
            "error": 50,
            "warning": 40,
            "info": 30,
            "debug": 20,
            "trace": 10
        },
        "value": {
            "hostname": {
                "kind": "string",
                "identifier": true,
                "hidden": true
            },
            "time": {
                "kind": "integer",
                "identifier": false
            },
            "pid": {
                "kind": "integer",
                "identifier": false
            },
            "level": {
                "kind": "integer",
                "identifier": true,
                "foreign-key": true
            },
            "msg": {
                "kind": "string"
            }
        },
        "timestamp-field": "time",
        "body-field": "msg"
    }
}

but lnav keeps selecting bunyan_log instead. I do not need all fields from bunyan_log line format. But the worst thing is that time is interpreted incorrectly. Bunyay has string kind specified, I have specified integer, but is still selects bunyan. This, by the way, sounds like a bug. Also if it would select my format, how can I specify that this is UTC milliseconds? And I want to display them in UTC.

Here is log example:

{"level":30,"time":1719324213672,"pid":3191448,"hostname":"RaspberryNode","msg":"message here"}

[aisamu]

but lnav keeps selecting bunyan_log instead.

IIUC lnav picks the one with the largest amount of fields on the line-format (e.g. "more specific"?)

[vagran]

but lnav keeps selecting bunyan_log instead.

IIUC lnav picks the one with the largest amount of fields on the line-format (e.g. "more specific"?)

In my case, I need just a subset of bunyan format fields, so my format cannot be more specific. Having just a simple command line argument (or option in UI) to select a specific format would solve everything. It also looks very natural to have such option.

If someone is interested, I currently worked-around my problem by overriding bunyan_log format in local config. Initially, I was under the impression that it is impossible without recompiling lnav itself, since all built-in formats are compiled into the binary. But later I found this feature in the docs.

[hvrauhal]

Regardless of the comment above, it took me a while to figure out that the quality of a JSON format is determined based on the line-format rather than the value matches, but this is how it indeed is:

lnav/src/log_format.cc

Lines 4109 to 4116 in 3d8be48

	if (std::find_if(this->jlf_line_format.begin(),
	this->jlf_line_format.end(),
	json_field_cmp(json_log_field::VARIABLE, ist))
	!= this->jlf_line_format.end())
	{
	retval.vlcr_line_format_count += 1;
	retval.vlcr_count -= 1;
	}

lnav/src/log_format.cc

Line 853 in 3d8be48

this->jlu_quality += res.vlcr_line_format_count;

lnav/src/log_format.cc

Line 1378 in 3d8be48

return log_format::scan_match{jlu.jlu_quality};

lnav/src/log_format.hh

Lines 423 to 425 in 3d8be48

	struct scan_match {
	uint32_t sm_quality;
	};

lnav/src/logfile.cc

Lines 447 to 448 in 3d8be48

	} else if (!best_match
	|| sm.sm_quality > best_match->second.sm_quality)