Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configurable access logs #489

Open
3 tasks
a-thaler opened this issue Nov 23, 2023 · 6 comments
Open
3 tasks

Configurable access logs #489

a-thaler opened this issue Nov 23, 2023 · 6 comments
Labels
area/service-mesh Issues or PRs related to service-mesh Epic kind/feature Categorizes issue or PR as related to a new feature.

Comments

@a-thaler
Copy link
Contributor

a-thaler commented Nov 23, 2023

Description
The istio module ships a hardcoded set of istio telemetry extensionProviders. One of these providers configures access logging based on stdout using a fixed JSON format. Using the extensionProvider mechanism users can selectively enable access logging on workloads or a gateway.

Users have custom request and response headers used across solutions and with that like to have them included in the access logs as well. As far as I can see is the existing approach of logging to stdout accepted, still istio already supports logging via OTLP. In combination with the strategy of the telemetry module (kyma-project/telemetry-manager#556), any customization should be also possible following the OTLP approach in future.

So one aspect is the actual format configuration, the other the actual backend configuration (here stdout) and the having all options of istio available. Thinking of traces and metrics, it becomes relevant on what the focus of the configurability will be, should we really support using a different tracer implementation? Of course it will be cool to have all options, but the goal is to provide a tested and manageable setup so that the user don't need to start bothering with the topic so much.

Goal
Turn the access logging into a configurable feature, supporting at least additional request and response headers. In best case you can define your own format. The feature should work with a solution based on OTLP as well. Aspects like traces and metrics could be supported in a similar way.

Criterias
As a devops I want to configure which HTTP req and res headers should appear in our monitoring tool (mainly SAP Cloud Logging, but also Dynatrace comes into mind) in order to use them for troubleshooting

Questions needed to get answered:

  • How to configure the header? What approaches can be supported?
    • Individual entries
    • Patterns, like X-*
    • whitelist, blacklist
  • Can the config be applied global only or also for particular URLs ? How?
  • How to deal with encoded headers?

Actions

  • Propose solutions and outline pros and cons
  • Make PoC and come to a documented decision
  • Implement and document the feature

Reasons
Access logs are often a key element for monitoring. If relevant context is encoded in headers, users require to see that context in the logs to improve the observability.

Attachments

@a-thaler a-thaler added the kind/feature Categorizes issue or PR as related to a new feature. label Nov 23, 2023
@a-thaler
Copy link
Contributor Author

After a first discussion round, it seems most valuable to focus on supporting well-defined and preconfigured extensionProviders which can be configured via the modules API. So taking the existing provider based on json and stdout and make the format extensible, having in mind that additional providers might be added later for further scenarios which again can be configured.
Hereby, we see it more important to support well-defined integration scenarios which are tested very well instead of providing full flexibility leveraging providers like envoyHttpAls which are barely tested by the module.
A more concrete API proposal will follow as a next step.

@videlov
Copy link
Collaborator

videlov commented Feb 13, 2024

We brainstormed different options and came up with this simple proposal. We would like to extend Istio CR's spec with additional configuration for Istio's MeshConfig.

We will have two default Istio extension providers definitions for EnvoyFileAccessLogProvider and EnvoyOpenTelemetryLogProvider to emit access logs either in JSON to stdout or OTLP.

The defined access log format is applied to all extension providers. The user activates an extension provider for specific workloads or the entire service mesh using the telemetry API.

Example

apiVersion: operator.kyma-project.io/v1alpha1
kind: Istio
metadata:
  name: default
  namespace: kyma-system
spec:
  config:
    accessLog:
      strategy: "merge" | "replace"
      format:
        tenant: "%REQ(X-TENANT)%"
  • strategy defines how to apply the key-value map in format. either merge with the existing default configuration or replace it with a completely new custom one
  • format refers to LogFormat's labels

@a-thaler
Copy link
Contributor Author

Additional background to the discussion: Activating multiple providers for access logs at the same time is unlikely as the outputs are usually conflicting (on stdout for sure, on OTLP potentially). So the user mainly wants to influence the format and then select the pre-defined output format. We do not see the need at the moment that other output types need to be configurable.
Leveraging an OTLP integration via the telemetry module in future will even provide manipulation options at a later stage.

@strekm strekm added the Epic label Feb 14, 2024
@strekm strekm mentioned this issue Feb 14, 2024
3 tasks
@a-thaler
Copy link
Contributor Author

In case you cannot wait for the feature, there is a workaround to have an additional access log configuration in the EnvoyFilter. This will be a configuration independent to the one which you can enable by the telemetry resource, so the easiest way to avoid duplicate log entries is to not enable access logging using the istio telemetry resources and to add all relevant fields in the EnvoyFilter configuration taken from here.

The following example will add a new log attribute with the fields tenant_id, method, path, traceparent, tracestate for HTTP requests that are inbound to the sidecar with the label app: httpbin.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: tenant-access-log
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: httpbin
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: MERGE
        value:
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
            access_log:
              - name: envoy.access_loggers.file
                typed_config:
                  "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
                  path: /dev/stdout
                  logFormat:
                    jsonFormat:
                      tenant_id: "%REQ(X-Tenant-ID)%"
                      method: "%REQ(:METHOD)%"
                      path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
                      traceparent: "%REQ(TRACEPARENT)%"
                      tracestate: "%REQ(TRACESTATE)%"

Sending a http request to the workload with the header X-TENANT-ID:3213 set will result in the log for the default log configuration and an additional log containing the tenant ID.

{"method":"GET","traceparent":null,"tenant_id":"3213","tracestate":null,"path":"/anything"}

@strekm strekm added the area/service-mesh Issues or PRs related to service-mesh label Mar 15, 2024
@vweckerle
Copy link

I am not sure if it is technically possible with istio. But if there would be a way to make the access log format specific per namespace or even per deployment, that would be a nice extra.
I.e. application A might use custom headers X, Y and Z and application B custom headers A, B and C. Having only one central log format would mean that it has to contain A, B, C, X, Y and Z. This does not scale very well.
But even with just a single log format, it would already be a very helpful extension over the status quo.

@a-thaler
Copy link
Contributor Author

The current Isio API allows to specify many "extensionProviders" which can be of type "accessLog". Via the Istio Telemetry API is is possible to then activate an extensionProvider per workload or namespace.
So indeed you could specify all your log formats centrally (repeating the base format over and over again) and them apply them via the API to the different workloads. There is no way yet to modify a central format dynamically by only adding a single expression.

The planned concept was considering to adjust the singleton accessLog format only. A question is if we should make that accessLog config more flexible and support more logFormats with new extensionProvider names (using the same base settings as the existing provider), or if we stay with the concept of adjusting the default format only and additionally support generic definition of extensionProviders.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/service-mesh Issues or PRs related to service-mesh Epic kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
Development

No branches or pull requests

4 participants