Skip to content

Commit

Permalink
Add OpenTelemetry Java Agent Extension (#165)
Browse files Browse the repository at this point in the history 
* OpenTelemetry Java Agent Extension

This extension for the OpenTelemetry Java Agent detects bindings to
SAP logging services with active OpenTelemetry support. If such a binding
is detected the OpenTelemetry Java Agent is configured with the provided
credentials and appropriate resource attributes are configured.

The change provides documentation for the OpenTelemetry Java Agent Extension
and a sample with the Spring Boot sample app.

---------

Signed-off-by: Karsten Schnitter <[email protected]>
Co-authored-by: JannikBrand <[email protected]>
  • Loading branch information
@KarstenSchnitter @JannikBrand
KarstenSchnitter and JannikBrand authored Nov 13, 2023
1 parent 936f7a7 commit af9d5c2
Show file tree
Hide file tree
Showing 17 changed files with 860 additions and 6 deletions.
121 changes: 121 additions & 0 deletions cf-java-logging-support-opentelemetry-agent-extension/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# OpenTelemetry Java Agent Extension for SAP Cloud Logging

This module provides an extension for the [OpenTelemetry Java Agent](https://opentelemetry.io/docs/instrumentation/java/automatic/).
The extension scans the service bindings of an application for SAP Cloud Logging.
If such a binding is found, the OpenTelemetry Java Agent is configured to ship observability data to that service.
Thus, this extension provides a convenient auto-instrumentation for Java applications running on SAP BTP.

The extension provides two main features:

* auto-configuration of the OpenTelemetry connection to SAP Cloud Logging
* adding resource attributes to describe the CF application

See the section on [configuration](#configuration) for further details.

## Quickstart Guide

Any Java application can be instrumented with the OpenTelemetry Java Agent and this extension by adding the following arguments to the java command:

```sh
java -javaagent:/path/to/opentelemetry-javaagent-<version>.jar \
-Dotel.javaagent-extensions=/path/to/cf-java-logging-support-opentelemetry-agent-extension-<versions>.jar \
# your Java application command
```

If you are using Spring Boot, you can bundle both dependencies with the application.
See the Maven pom of the [Spring Boot sample application](../sample-spring-boot/pom.xml) for details.
When deployed to a Cloud Foundry runtime environment, the Spring Boot jar is expanded, so that the agent and extension jar are available during application start.
In that case, the following Java arguments are required:

```sh
java -javaagent:BOOT-INF/lib/opentelemetry-javaagent-<version>.jar \
-Dotel.javaagent.extensions=BOOT-INF/lib/cf-java-logging-support-opentelemetry-agent-extension-<version>.jar \
# your Java application command
```

See the [example manifest](../sample-spring-boot/manifest-otel-javaagent.yml), how this translates into a deployment description.

For the instrumentation to send observability data to SAP Cloud Logging, the application needs to be bound to a corresponding service instance.
The service instance can be either managed or [user-provided](#using-user-provided-service-instances).

Note, that the OpenTelemetry Java Agent currently only sends traces and metrics by default.
To enable logs, the additional property `-Dotel.logs.exporter=otlp` is required.

## Configuration

The OpenTelemetry Java Agent supports a wide variety of [configuration options](https://opentelemetry.io/docs/instrumentation/java/automatic/agent-config/).
As the extension provides configuration via SPI, all its configuration takes lower precedence than other configuration options for OpenTelemetry.
Users can easily overwrite any setting using environment variables or system properties.

### Configuring the Extension

The extension itself can be configured by specifying the following system properties:

| Property | Default Value | Comment |
|----------|---------------|---------|
| `com.sap.otel.extension.cloud-logging.label` | `cloud-logging` | The label of the managed service binding to bind to. |
| `com.sap.otel.extension.cloud-logging.tag` | `Cloud Logging` | The tag of any service binding (managed or user-provided) to bind to. |
| `otel.javaagent.extension.sap.cf.resource.enabled` or `env(OTEL_JAVAAGENT_EXTENSION_SAP_CF_RESOURCE_ENABLED)` | `true` | Whether to add CF resource attributes to all events. |

The extension will scan the environment variable `VCAP_SERVICES` for CF service bindings.
User-provided bindings will take precedence over managed bindings of the configured label ("cloud-logging" by default).
All matching bindings are filtered for the configured tag ("Cloud Logging" by default).
The first binding will be taken for configuration for the OpenTelemetry exporter.
Preferring user-provided services over managed service instances allows better control of the binding properties, e.g. syslog drains.

### Recommended Agent Configuration

The OpenTelemetry Java Agent offers a lot of configuration options.
The following set of properties is recommended to be used with the extension:

```sh
java -javaagent:/path/to/opentelemetry-javaagent-<version>.jar \
-Dotel.javaagent-extensions=/path/to/cf-java-logging-support-opentelemetry-agent-extension-<versions>.jar \
# enable logs \
-Dotel.logs.exporter=otlp \
# reroute agent logs to otlp \
-Dotel.javaagent.logging=application
# configure logback context \
-Dotel.instrumentation.logback-appender.experimental.capture-mdc-attributes=* \
-Dotel.instrumentation.logback-appender.experimental.capture-key-value-pair-attributes=true \
-Dotel.instrumentation.logback-appender.experimental.capture-code-attributes=true \
-Dotel.instrumentation.logback-appender.experimental-log-attributes=true \
# Disable large resource attributes
-Dotel.experimental.resource.disabled-keys=process.command_line,process.command_args,process.executable.path
```

The [OpenTelemetry Java Instrumentation project](https://github.com/open-telemetry/opentelemetry-java-instrumentation) provides detailed documentation on the configuration properties for [Logback](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/logback/logback-appender-1.0/javaagent) and [Log4j](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/log4j/log4j-appender-2.17/javaagent).

## Using User-Provided Service Instances

The extension provides support not only for managed service instance of SAP Cloud Logging but also for user-provided service instances.
This helps to fine-tune the configuration, e.g. leave out or reconfigure the syslog drain.
Furthermore, this helps on sharing service instances across CF orgs or landscapes.

The extension requires four fields in the user-provided service credentials and needs to be tagged with the `com.sap.otel.extension.cloud-logging.tag` (default: `Cloud Logging`) documented in section [Configuration](#configuration).

| Field name | Contents |
|------------|---------|
| `ingest-otlp-endpoint` | The OTLP endpoint including port. It will be prefixed with `https://`. |
| `ingest-otlp-key` | The mTLS client key in PCKS#8 format. Line breaks as `\n`. |
| `ingest-otlp-cert`| The mTLS client certificate in PEM format matching the client key. Line breaks as `\n`. |
| `server-ca` | The trusted mTLS server certificate in PEM format. Line breaks as `\n`. |

If you have a SAP Cloud Logging service key, you can generate the required JSON file with jq:

```bash
cf service-key cls test \
| tail -n +2 \
| jq '.credentials | {"ingest-otlp-endpoint":."ingest-otlp-endpoint", "ingest-otlp-cert":."ingest-otlp-cert", "ingest-otlp-key":."ingest-otlp-key", "server-ca":."server-ca"}' \
> ups.json
```

Using this file, you can create the required user-provided service:

```bash
cf cups <your-service-name> -p ups.json -t "Cloud Logging"
```

Note, that you can easily feed arbitrary credentials to the extension.
It does not need to be SAP Cloud Logging.
You can even change the tag using the configuration parameters of the extension.
145 changes: 145 additions & 0 deletions cf-java-logging-support-opentelemetry-agent-extension/dependency-reduced-pom.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<parent>
<artifactId>cf-java-logging-support-parent</artifactId>
<groupId>com.sap.hcp.cf.logging</groupId>
<version>3.7.1</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>cf-java-logging-support-opentelemetry-agent-extension</artifactId>
<name>cf-java-logging-support-opentelemetry-agent-extension</name>
<build>
<plugins>
<plugin>
<artifactId>maven-shade-plugin</artifactId>
<version>3.5.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>io.pivotal.cfenv:java-cfenv</artifact>
<includes>
<include>io/pivotal/cfenv/**</include>
</includes>
</filter>
</filters>
<artifactSet>
<excludes>
<exclude>io.opentelemetry</exclude>
<exclude>com.fasterxml.jackson.core</exclude>
</excludes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>11</source>
<target>11</target>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-common</artifactId>
<version>1.31.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
<version>1.31.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.36</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-library</artifactId>
<version>1.3</version>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>hamcrest-core</artifactId>
<groupId>org.hamcrest</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.2</version>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>hamcrest-core</artifactId>
<groupId>org.hamcrest</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.10.19</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>1.36</version>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>jopt-simple</artifactId>
<groupId>net.sf.jopt-simple</groupId>
</exclusion>
<exclusion>
<artifactId>commons-math3</artifactId>
<groupId>org.apache.commons</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>1.36</version>
<scope>test</scope>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-bom</artifactId>
<version>${opentelemetry.sdk.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-bom-alpha</artifactId>
<version>${opentelemetry.sdk.version}-alpha</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<properties>
<maven.compiler.target>11</maven.compiler.target>
<maven.compiler.source>11</maven.compiler.source>
<opentelemetry.sdk.version>1.31.0</opentelemetry.sdk.version>
</properties>
</project>
102 changes: 102 additions & 0 deletions cf-java-logging-support-opentelemetry-agent-extension/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<artifactId>cf-java-logging-support-opentelemetry-agent-extension</artifactId>
<packaging>jar</packaging>

<name>cf-java-logging-support-opentelemetry-agent-extension</name>

<parent>
<artifactId>cf-java-logging-support-parent</artifactId>
<groupId>com.sap.hcp.cf.logging</groupId>
<version>3.7.1</version>
</parent>

<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<opentelemetry.sdk.version>1.31.0</opentelemetry.sdk.version>
</properties>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-bom</artifactId>
<version>${opentelemetry.sdk.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-bom-alpha</artifactId>
<version>${opentelemetry.sdk.version}-alpha</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

<dependencies>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-common</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.pivotal.cfenv</groupId>
<artifactId>java-cfenv</artifactId>
<version>2.5.0</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.5.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>io.pivotal.cfenv:java-cfenv</artifact>
<includes>
<include>io/pivotal/cfenv/**</include>
</includes>
</filter>
</filters>
<artifactSet>
<excludes>
<exclude>io.opentelemetry</exclude>
<exclude>com.fasterxml.jackson.core</exclude>
</excludes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>11</source>
<target>11</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
25 changes: 25 additions & 0 deletions ...ain/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/CloudFoundryResourceProvider.java
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package com.sap.hcf.cf.logging.opentelemetry.agent.ext;

import com.sap.hcf.cf.logging.opentelemetry.agent.ext.attributes.CloudFoundryResourceCustomizer;
import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
import io.opentelemetry.sdk.autoconfigure.spi.ResourceProvider;
import io.opentelemetry.sdk.resources.Resource;
import io.pivotal.cfenv.core.CfEnv;

public class CloudFoundryResourceProvider implements ResourceProvider {

private final CloudFoundryResourceCustomizer customizer;

public CloudFoundryResourceProvider() {
this(new CfEnv());
}

public CloudFoundryResourceProvider(CfEnv cfEnv) {
this.customizer = new CloudFoundryResourceCustomizer(cfEnv);
}

@Override
public Resource createResource(ConfigProperties configProperties) {
return customizer.apply(null, configProperties);
}
}
Loading

0 comments on commit af9d5c2

Please sign in to comment.