Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to use line-format for non-JSON encoded logs #1309

Open
IvanRibakov opened this issue Sep 20, 2024 · 2 comments
Open

Ability to use line-format for non-JSON encoded logs #1309

IvanRibakov opened this issue Sep 20, 2024 · 2 comments
Labels
enhancement Feature Request

Comments

@IvanRibakov
Copy link

Is your feature request related to a problem? Please describe.
I'm dealing with non-JSON log files that contain variable-width source file identifiers.
I would like to align log entries using line-format's min-width, max-width, align and overflow settings.

Describe the solution you'd like
I'd like to be able to use field names from the value object in the line-format's field field.

Describe alternatives you've considered
I saw that values have a rewriter field but I was unable to understand from the documentation how to achieve what I would like to using that field. Also, I'm not looking forward to re-implement abbrev logic when it is already available elsewhere.

Additional context

Example of the input logs:

poc  2024-09-20T07:01:20.111540Z INFO    1:Microsoft.Extensions.Hosting.Internal.ApplicationLifetime:0   Content root path: /foo/bar
poc  2024-09-20T07:01:20.111615Z DEBUG   1:Microsoft.Extensions.Hosting.Internal.Host:0   Hosting started
poc  2024-09-20T07:01:30.083486Z INFO    7:Foo.Bar.BarService:34   Worker running at: 09/20/2024 07:01:30 +00:00

Desired output:

poc  2024-09-20T07:01:20.111540Z INFO    1:    M.E.H.I.ApplicationLifetime:0   Content root path: /foo/bar
poc  2024-09-20T07:01:20.111615Z DEBUG   1:                   M.E.H.I.Host:0   Hosting started
poc  2024-09-20T07:01:30.083486Z INFO    7:                F.B.BarService:34   Worker running at: 09/20/2024 07:01:30 +00:00
Here is the custom format file that I'm working with
{
    "$schema": "https://lnav.org/schemas/format-v1.schema.json",
    "indigital": {
        "description": "Format file generated from regex101 entry",
        "regex": {
            "std": {
                "pattern": "^\\s*(?<prod>\\w+)\\s+(?<timestamp>\\d{4,4}-\\d{2,2}-\\d{2,2}T\\d{2,2}:\\d{2,2}:\\d{2,2}.\\d{0,6}Z)\\s+(?<lvl>\\w+)\\s+(?<pid>\\d+):(?<source>.+?):(?<line>\\d+)\\s+(?<msg>.+)$"
            }
        },
        "value": {
            "line": {
                "kind": "integer"
            },
            "lvl": {
                "kind": "string"
            },
            "msg": {
                "kind": "string"
            },
            "pid": {
                "kind": "integer"
            },
            "prod": {
                "kind": "string"
            },
            "source": {
                "kind": "string"
            },
            "timestamp": {
                "kind": "string"
            }
        },
        "level-field": "lvl",
        "body-field": "msg",
        "sample": [
            {
                "line": " poc  2024-09-19T15:33:52.428593Z INFO  5:Foo.Bar.BarService:34   Worker running at: 09/20/2024 07:01:30 +00:00"
            }
        ]
    }
}
@IvanRibakov IvanRibakov added the enhancement Feature Request label Sep 20, 2024
@tstack
Copy link
Owner

tstack commented Sep 27, 2024

The challenge with doing this is that searches and some other operations need to be done on the rendered text and not on the original file, which would affect performance quite a bit.

@IvanRibakov
Copy link
Author

Personally I'd be fine with search working on the original file/stream and highlighting being limited (for example highlighting entire entry if the match is found but not present in the UI) as I'm not going to remove/edit/rewrite any crucial information that I'd be searching on. The goal is only to make the logs more human-eye-friendly by stripping some excessive info and improving the indentation/alignment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Feature Request
Projects
None yet
Development

No branches or pull requests

2 participants