Log Context

[SpencerTorres]

Overview

This PR implements Log Context for logs queries.

Implementation

This feature works by taking a logs query and removing all filters and ORDER BYs. Filters are then re-added for every log context column that is matched in the log row. These filters will match a column using the = operator with the exact value from the selected row. These context columns can be configured on the datasource config page.

A time range filter is also added based on the selected time column. An ORDER BY and LIMIT are also added. Two queries are generated in each direction of the log row. You can scroll up and down in either direction to load more pages from the last timestamp offset.

Log Context Column Configuration:

Log context columns should be the minimum number of columns required to narrow your log row to a single source. For most datasets this will likely be some combination of a service name, host name, container ID, or pod name. Keep in mind that this should also match your ClickHouse table's primary key for optimal performance.

You must SELECT at least one of these context columns in order for the feature to work. The columns must be selected in order to read the value for filtering. If columns are not configured, or none are matched, the following warning is displayed:

For faster query building, you may enable the Auto-Select Columns setting. This will automatically include the context columns in your SELECT, even if OTel is enabled.

When your log row matches your context columns, they will be shown at the top where they can be easily copied.

A note for Map columns

Use bracket syntax in the config. If your context columns are defined as ResourceAttributes['container.id'] and ResourceAttributes['host.name'], you can simply select ResourceAttributes and the plugin will be able to match multiple keys inside the map. Individual field selection works too. Without an alias ClickHouse will name these arrayElement(ResourceAttributes, 'container.id'), but the plugin will still identify these.

[mshustov]

Keep in mind that this should also match your ClickHouse table's primary key for optimal performance.

Do we enforce the right order for filter fields?

Use bracket syntax in the config. If your context columns are defined..

We'll need to extend the guide to recommend the field aliasing feature

[RoryCrispin]

Brilliant! 👍

[SpencerTorres]

Do we enforce the right order for filter fields?

I don't think they need a specific order, the query analyzer should handle that right?

We'll need to extend the guide to recommend the field aliasing feature

Using ALIAS columns would work well with this. Same as if you extracted a column in a materialized view

[aangelisc]

This LGTM, just one nit in the comments. I believe it'd be worthwhile updating the documentation to include some of the information you've specified in the PR description as it's extremely beneficial.

[SpencerTorres]

Still suspicious of the yarn lockfile, but that last type is fixed