OpenTelemetry Collector Symbolicator Processor

An open telemetry collector processor that will symbolicate JavaScript stack traces using source maps. This is compatible with v0.12.0 and onwards of the Honeycomb OpenTelemetry web SDK.

To install this processor, include it in the build config file of your OpenTelemetry collector distro. You can also use the pre-built Honeycomb OpenTelemetry collector distro that already includes this processor.

Requirements

We require the use of CGO for the processors. If you are building your own collector image it must support glibc. We recommend gcr.io/distroless/cc

JavaScript Source Maps

Basic Configuration

Register the plugin in the processors section of your open telemetry collector configuration.

    processors:
      symbolicator:

JavaScript source and source map files

The symbolicator requires access to both your minimised JavaScript source files and associated JavaScript source map files generated by your build process. Ideally these files are versioned with a file hash in the file name. eg. main.c383b093b0b66825a9c3.js. The minimised source file must also contain the sourceMappingURL tag containing the relative path to the source map file. The symbolicator can access these files through a number of different storage mechanisms documented below.

Exception information format

The processor expects the stacktrace information to be formatted into four separate attributes:

columns
functions
lines
urls

Each of these attributes must be a slice with each being of equal length.

Example:

columns: [6465,3512358,3512661,3514018,758545]
functions: ["?","w.callback","push.Br+g.w.crossDomainError","XMLHttpRequest.<anonymous>","XMLHttpRequest.<anonymous>"]
lines: [3582,2,2,2,2]
urls: ["https://example.com/static/dist/main.c383b093b0b66825a9c3.js","https://example.com/static/dist/vendor.1c285a50f5307be9648d.js","https://example.com/static/dist/vendor.1c285a50f5307be9648d.js","https://example.com/static/dist/vendor.1c285a50f5307be9648d.js","https://example.com/static/dist/vendor.1c285a50f5307be9648d.js"]

Advanced Configuration

Attribute Mapping

The following configuration options can also be provided to change the attributes used to look for stack traces and store them.

Config Key	Description	Example Value
`symbolicator_failure_attribute_key`	Signals if the the symbolicator fails to fully symbolicate the stack trace	`exception.symbolicator.failed`
`symbolicator_failure_message_attribute_key`	Contains the error message if the the symbolicator fails to fully symbolicate the stack trace	`exception.symbolicator.error`
`columns_attribute_key`	Which attribute should the columns of the stack trace be sourced from	`exception.structured_stacktrace.columns`
`functions_attribute_key`	Which attribute should the functions of the stack trace be sourced from	`exception.structured_stacktrace.functions`
`lines_attribute_key`	Which attribute should the lines of the stack trace be sourced from	`exception.structured_stacktrace.lines`
`urls_attribute_key`	Which attribute should the urls of the stack trace be sourced from	`exception.structured_stacktrace.urls`
`output_stack_trace_key`	Which attribute should the symbolicated stack trace be populated into	`exception.stacktrace`
`stack_type_key`	Which attribute contains the exception type	`exception.type`
`stack_message_key`	Which attribute contains the exception message	`exception.message`
`preserve_stack_trace`	After the stack trace has been symbolicated should the original values be preserved as attributes	`true`
`original_stack_trace_key`	If the stack trace is being preserved which key should it be copied to	`exception.stacktrace.original`
`original_columns_attribute_key`	If the stack trace is being preserved which key should the functions be copied to	`exception.structured_stacktrace.functions.original`
`original_functions_attribute_key`	If the stack trace is being preserved which key should the lines be copied to	`exception.structured_stacktrace.lines.original`
`original_lines_attribute_key`	If the stack trace is being preserved which key should the columns be copied to	`exception.structured_stacktrace.columns.original`
`original_urls_attribute_key`	If the stack trace is being preserved which key should the urls be copied to	`exception.structured_stacktrace.urls.original`

Additional Options

Config Key	Description	Example Value
`timeout`	Max duration to wait to symbolicate a stack trace in seconds.	`5`
`source_map_cache_size`	The maximum number of source maps to cache. Reduce this if you are running into memory issues with the collector.	`128`

Internal Telemetry

The collector processor emits custom telemetry metrics that provides insight into its status and performance. The custom processor metrics are generated using mdatagen and include metrics such as:

Number of stack frames processed
Number of stack frames that failed to symbolicate
Total source map fetch failures
The size of the source map cache in bytes

To see the full list of the custom telemetry collected, see documentation.md in the symbolicatorprocessor package.

To actually send your internal telemetry to a backend, make sure to enable the metrics service in your collector's yaml config file. Here is an example of a collector config sending metrics to a Honeycomb endpoint:

# ...

service:
  metrics:
    readers:
      - periodic:
          exporter:
            otlp:
              protocol: http/protobuf
              endpoint: "https://api.honeycomb.io:443" # OR, for EU instance: https://api.eu1.honeycomb.io:443
              headers:
                - name: "x-honeycomb-team"
                  value: "HONEYCOMB_API_KEY"
                - name: "x-honeycomb-dataset"
                  value: "otel-collector-metrics"

#...

Internal telemetry is collected by the collector by default and will be sent alongside our custom telemetry. Please visit Open Telemetry's Collector docs for a full list of the default internal metrics.

dSYM Symbolication

Basic Configuration

Register the plugin in the processors section of your open telemetry collector configuration.

    processors:
      dsymsymbolicator:

dSYM files

The symbolicator requires access to the dSYM file generated by the build process. This file must be versioned with the generated build UUID in the file name. eg. 6A8CB813-45F6-3652-AD33-778FD1EAB196.dSYM. The symbolicator can access this dSYM file through a number of different storage mechanisms documented below.

Getting the Build UUID

Given an .xcarchive file generated by Xcode, you can get this build uuid with the dwarfdump tool. For example, the following script will find the latest build and upload it to the app-archives s3 bucket with the ios prefix:

example script

# Get the App Name
if [[ -z "$1" ]]; then
  echo "❌ Usage: $0 <TargetName>"
  exit 1
fi

TARGET_NAME=$1

export ARCHIVE_PATH=$(ls -dt ~/Library/Developer/Xcode/Archives/*/"$TARGET_NAME"*.xcarchive | head -1)
echo "📦 Using Archive Path: $ARCHIVE_PATH"

if [[ ! -d "$ARCHIVE_PATH" ]]; then
  echo "❌ Archive not found for target: $TARGET_NAME! Please archive the project first in Xcode."
  exit 1
fi

find "$ARCHIVE_PATH/dSYMs" -name "*.dSYM" | while read line ; do
   echo "🔍 Found dsym at: $line"
   dsymuuid=$(dwarfdump -u "$line" | awk '{ print $2 }').dSYM
   echo "⬆️ Uploading dsym to: $dsymuuid"
   aws s3 cp --recursive "$line" s3://app-archives/ios/$dsymuuid
done

Exception information format

The processor processes incoming logs and expects the stacktrace information to be formatted one of two formats: generic stack traces or metrickit reports.

Generic Stacktraces

Generic stacktraces must include the following attributes:

exception.stacktrace
app.debug.build_uuid
app.bundle.executable

The processor will use app.debug.build_uuid to look up a corresponding dSYM, and use app.bundle.executable to find the relevant binary archive within that dSYM. It will then attempt to symbolicate as many lines of exception.stacktrace as it can.

Any lines in exception.stacktrace that refer to unknown binaries will be left as-is.

Additionally, while they are not required or used by the processor, exception.message and exception.type attributes are required by OTel semantic conventions, and most downstream error sinks will expect them to be present.

MetricKit

MetricKit logs must include the metrickit.diagnostic.crash.exception.stacktrace_json attribute, corresponding to the jsonRepresentation of a MetricKit crash.

The processor will use binaryUUID and binaryName contained in that JSON report to look up a corresponding dSYM and find the relevant binary archive within that dSYM. It will then attempt to symbolicate as many frames of the crash report as it can.

Any frames that refer to unknown binaries will be left as-is.

Advanced Configuration

Attribute Mapping

The following configuration options can also be provided to change the attributes used to look for stack traces and store them.

Config Key	Description	Example Value
`symbolicator_failure_attribute_key`	Signals if the the symbolicator fails to fully symbolicate the stack trace	`exception.symbolicator.failed`
`symbolicator_failure_message_attribute_key`	Contains the error message if the the symbolicator fails to fully symbolicate the stack trace	`exception.symbolicator.error`
`stack_trace_attribute_key`	Which attribute should the stack trace of a generic stacktrace log be sourced from	`exception.stacktrace`
`original_stack_trace_key`	If the stack trace is being preserved, which key should it be copied to	`exception.stacktrace.original`
`build_uuid_attribute_key`	Which resource attribute should the binary UUID of a generic stacktrace log be sourced from	`app.debug.build_uuid`
`app_executable_attribute_key`	Which resource attribute should the name of the app executable of a generic stacktrace log be sourced from	`app.bundle.executable`
`metrickit_stack_trace_attribute_key`	Which attribute should the json representation of a metrickit stacktrace log be sourced from	`metrickit.diagnostic.crash.exception.stacktrace_json`
`output_metrickit_stack_trace_attribute_key`	Which attribute should the symbolicated metrickit stack trace be populated into	`exception.stacktrace`
`output_metrickit_exception_type_attribute_key`	Which attribute should the exception type be populated into	`exception.type`.
`output_metrickit_exception_message_attribute_key`	Which attribute should the exception message be populated into	`exception.message`.
`preserve_stack_trace`	After the stack trace has been symbolicated should the original values be preserved as attributes	`true`

Additional Options

Config Key	Description	Example Value
`timeout`	Max duration to wait to symbolicate a stack trace in seconds.	`5`
`dsym_cache_size`	The maximum number of dSYMs to cache. Reduce this if you are running into memory issues with the collector.	`128`

Proguard Symbolication

Basic Configuration

Register the plugin in the processors section of your open telemetry collector configuration.

    processors:
      proguard_symbolicator:

Proguard files

The symbolicator requires access to the Proguard file generated by the build process. This file must be versioned with the generated build UUID in the file name. eg. 6A8CB813-45F6-3652-AD33-778FD1EAB196.txt. The symbolicator can access this proguard file through a number of different storage mechanisms documented below.

Exception information format

The processor expects the stacktrace information to be formatted into three separate attributes:

classes
methods
lines

Each of these attributes must be a slice with each being of equal length.

Example:

classes: ["com.example.Test", "com.example.Test"]
methods: ["methodA", "methodB"]
lines: [3582, 2001]

Advanced Configuration

Attribute Mapping

The following configuration options can also be provided to change the attributes used to look for stack traces and store them.

Config Key	Description	Example Value
`symbolicator_failure_attribute_key`	Signals if the the symbolicator fails to fully symbolicate the stack trace	`exception.symbolicator.failed`
`classes_attribute_key`	Which attribute should the classes of the stack trace be sourced from	`exception.structured_stacktrace.classes`
`methods_attribute_key`	Which attribute should the methods of the stack trace be sourced from	`exception.structured_stacktrace.methods`.
`lines_attribute_key`	Which attribute should the lines of the stack trace be sourced from	`exception.structured_stacktrace.lines`
`output_stack_trace_key`	Which attribute should the symbolicated stack trace be populated into	`exception.stacktrace`
`preserve_stack_trace`	After the stack trace has been symbolicated should the original values be preserved as attributes	`true`
`original_classes_attribute_key`	If the stack trace is being preserved which key should the classes be copied to	`exception.structured_stacktrace.classes.original`
`original_methods_attribute_key`	If the stack trace is being preserved which key should the methods be copied to	`exception.structured_stacktrace.methods.original`
`original_lines_attribute_key`	If the stack trace is being preserved which key should the lines be copied to	`exception.structured_stacktrace.lines.original`
`original_stack_trace_key`	If the stack trace is being preserved which key should it be copied to	`exception.stacktrace.original`
`proguard_uuid_attribute_key`	Which resource attribute should the proguard UUID of a generic stacktrace log be sourced from	`app.debug.proguard_uuid`

Additional Options

Config Key	Description	Example Value
`timeout`	Max duration to wait to symbolicate a stack trace in seconds.	`5`
`proguard_cache_size`	The maximum number of proguard files to cache. Reduce this if you are running into memory issues with the collector.	`128`

Common Storage Mechanisms

Both the Source Map and dSYM processors use the same configuration formats for loading the requires files. The examples below are written for Source Map storage, but may easily be adapted to dSYM storage by swapping the names of the configuration keys:

Source Map Key	dSYM storage equivalent	proguard storage equivalent
`source_map_store`	`dsym_store`	`proguard_store`
`local_source_maps`	`local_dsyms`	`local_store`
`s3_source_maps`	`s3_dsyms`	`s3_store`
`gcs_source_maps`	`gcs_dsyms`	`gcs_store`

File Store

The default configuration will load the source(map) files (this must include both the JavaScript source file and the JavaScript source map file) from a local path on disk. You can set the base path that will be used.

    processors:
      symbolicator:
        # source_map_store is used to configure which store to use, in this case local disk
        source_map_store: file_store
        local_source_maps:
          # (optional) path is used to set the base path of the files, defaults to `.`
          path: /tmp/sourcemaps

How does the file store source the files?

Each line of the stack trace includes the URL of the file it originated from. Taking this as an example https://example.com/static/dist/main.c383b093b0b66825a9c3.js. The base file name is then found main.c383b093b0b66825a9c3.js. This path is joined with the configured path and then read from disk.

S3 Store

You can also load the source(map) files (this must include both the JavaScript source file and the JavaScript source map file) from an S3 bucket.

    processors:
      symbolicator:
        # source_map_store is used to configure which store to use, in this case S3
        source_map_store: s3_store
        # s3_source_maps is used to configure the sourcing of source maps from S3
        s3_source_maps:
          # bucket is the name of the bucket the files are stored in
          bucket: source-maps-bucket
          # (optional) region is used to configure the buckets location
          region: us-east-1
          # (optional) prefix is used to nest the files in a sub key of the bucket
          prefix: source-maps

How does the S3 store source the files?

Each line of the stack trace includes the URL of the file it originated from. Taking this as an example https://example.com/static/dist/main.c383b093b0b66825a9c3.js. The base file name is then found main.c383b093b0b66825a9c3.js. This path is joined with the prefix if provided and then used as the key to source from the bucket.

GCS Store

You can also load the source(map) files from a GCS bucket.

    processors:
      symbolicator:
        # source_map_store is used to configure which store to use, in this case GCS
        source_map_store: gcs_store
        # gcs_source_maps is used to configure the sourcing of source maps from GCS
        gcs_source_maps:
          # bucket is the name of the bucket the files are stored in
          bucket: source-maps-bucket
          # (optional) prefix is used to nest the files in a sub key of the bucket
          prefix: source-maps

How does the GCS store source the files?

Each line of the stack trace includes the URL of the file it originated from. Taking this as an example https://example.com/static/dist/main.c383b093b0b66825a9c3.js. The base file name is then found main.c383b093b0b66825a9c3.js. This path is joined with the prefix if provided and then used as the key to source from the bucket.

Name		Name	Last commit message	Last commit date
Latest commit History 78 Commits
.circleci		.circleci
.github		.github
dsymprocessor		dsymprocessor
proguardprocessor		proguardprocessor
symbolicatorprocessor		symbolicatorprocessor
test_assets		test_assets
.gitignore		.gitignore
.tool-versions		.tool-versions
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
NOTICE		NOTICE
OSSMETADATA		OSSMETADATA
README.md		README.md
RELEASING.md		RELEASING.md
SECURITY.md		SECURITY.md
SUPPORT.md		SUPPORT.md
builder-config.yaml		builder-config.yaml
config.yaml		config.yaml
go.work		go.work
go.work.sum		go.work.sum

License

honeycombio/opentelemetry-collector-symbolicator

Folders and files

Latest commit

History

Repository files navigation

OpenTelemetry Collector Symbolicator Processor

Requirements

JavaScript Source Maps

Basic Configuration

JavaScript source and source map files

Exception information format

Advanced Configuration

Attribute Mapping

Additional Options

Internal Telemetry

dSYM Symbolication

Basic Configuration

dSYM files

Getting the Build UUID

Exception information format

Generic Stacktraces

MetricKit

Advanced Configuration

Attribute Mapping

Additional Options

Proguard Symbolication

Basic Configuration

Proguard files

Exception information format

Advanced Configuration

Attribute Mapping

Additional Options

Common Storage Mechanisms

File Store

How does the file store source the files?

S3 Store

How does the S3 store source the files?

GCS Store

How does the GCS store source the files?

About

Resources

License

Code of conduct

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 18

Packages 0

Uh oh!

Contributors 5

Uh oh!

Languages

Packages