instrumentation-grpc doesn't seem to get initialized with --require @opentelemetry/auto-instrumentations-node/register

[dudo]

What happened?

grpc (and others, but I'm especially concerned with grpc) traces aren't being emitted. Calls are successful, but no trace is emitted, and no context (or baggage) is propagated. Incoming http and dns traces are present, so it seems isolated some instrumentation.

I'm using modules and passing --require as an argument to node.

Here's a repo reproducing the problem:

https://github.com/gitops-ci-cd/acme-node/blob/main/package.json#L7

Simply run

docker compose watch

then visit http://localhost:8080/api/v1/hello.

OpenTelemetry Setup Code

OTEL_LOG_LEVEL=debug
OTEL_TRACES_EXPORTER=console
OTEL_METRICS_EXPORTER=none
OTEL_LOGS_EXPORTER=none
OTEL_SERVICE_NAME=acme-node
OTEL_NODE_RESOURCE_DETECTORS=env,host,os,process,serviceinstance,container
OTEL_NODE_ENABLED_INSTRUMENTATIONS=dns,http,express,grpc

package.json

No response

Relevant log output

Loading instrumentation for @opentelemetry/instrumentation-dns
Loading instrumentation for @opentelemetry/instrumentation-express
Loading instrumentation for @opentelemetry/instrumentation-grpc
Loading instrumentation for @opentelemetry/instrumentation-http
OpenTelemetry automatic instrumentation started successfully
...
@opentelemetry/instrumentation-http Applying instrumentation patch for nodejs core module on require hook { module: 'http' }
@opentelemetry/instrumentation-dns Applying instrumentation patch for nodejs core module on require hook { module: 'dns' }

I see hooks for http and dns, but none for grpc (or even express, which I'm using)

Operating System and Version

OSDetectorSync found resource. Resource {
  _attributes: { 'os.type': 'linux', 'os.version': '6.10.14-linuxkit' },
  asyncAttributesPending: false,
  _syncAttributes: { 'os.type': 'linux', 'os.version': '6.10.14-linuxkit' },
  _asyncAttributesPromise: undefined
}

Runtime and Version

Running via docker.

ProcessDetectorSync found resource. Resource {
  _attributes: {
    'process.pid': 18,
    'process.executable.name': 'node',
    'process.executable.path': '/usr/local/bin/node',
    'process.command_args': [
      '/usr/local/bin/node',
      '--require',
      '@opentelemetry/auto-instrumentations-node/register',
      '/usr/src/app/src/index.js'
    ],
    'process.runtime.version': '22.12.0',
    'process.runtime.name': 'nodejs',
    'process.runtime.description': 'Node.js',
    'process.command': '/usr/src/app/src/index.js',
    'process.owner': 'root'
  },
  asyncAttributesPending: false,
  _syncAttributes: {
    'process.pid': 18,
    'process.executable.name': 'node',
    'process.executable.path': '/usr/local/bin/node',
    'process.command_args': [
      '/usr/local/bin/node',
      '--require',
      '@opentelemetry/auto-instrumentations-node/register',
      '/usr/src/app/src/index.js'
    ],
    'process.runtime.version': '22.12.0',
    'process.runtime.name': 'nodejs',
    'process.runtime.description': 'Node.js',
    'process.command': '/usr/src/app/src/index.js',
    'process.owner': 'root'
  },
  _asyncAttributesPromise: undefined
}

[pichlermarc]

Hi @dudo, thanks for reaching out - I think you also need to add the instrumentation hook for ESM (see "Startup command for ESM"):
https://github.com/open-telemetry/opentelemetry-js/blob/main/doc/esm-support.md#using-the-zero-code-option-with-auto-instrumentations-node

[dudo]

Hello, @pichlermarc! That is super helpful, thank you! I was coming from the auto instrumentation library docs, and missed that in the core doc(s). Might be worth adding a similar blurb there, since many folks trying this out will likely start there? I have no data to back that up, obviously 😅 .

So now grpc telemetry is loading, but I'm still not getting any applicable traces. Here are the logs when it loads, which might be a smoking gun

@opentelemetry/instrumentation-http Applying instrumentation patch for nodejs core module on require hook { module: 'http' }
@opentelemetry/instrumentation-net Applying instrumentation patch for nodejs core module on require hook { module: 'net' }
@opentelemetry/instrumentation-express Applying instrumentation patch for module on require hook {
  module: 'express',
  version: '4.21.2',
  baseDir: '/usr/src/app/node_modules/express'
}
@opentelemetry/instrumentation-dns Applying instrumentation patch for nodejs core module on require hook { module: 'dns' }
@opentelemetry/instrumentation-grpc Applying instrumentation patch for module on require hook {
  module: '@grpc/grpc-js',
  version: '1.12.5',
  baseDir: '/usr/src/app/node_modules/@grpc/grpc-js'
}
@opentelemetry/instrumentation-grpc patched gRPC server
@opentelemetry/instrumentation-grpc patching client
@opentelemetry/instrumentation-grpc patching client
@opentelemetry/instrumentation-grpc patching loadPackageDefinition
@opentelemetry/instrumentation-grpc patched makeClientStreamRequest on grpc client
@opentelemetry/instrumentation-grpc patched makeClientStreamRequest on grpc client
@opentelemetry/instrumentation-grpc patched makeClientStreamRequest on grpc client
@opentelemetry/instrumentation-grpc patched makeClientStreamRequest on grpc client

I'm not streaming anything, and I don't see a patch for a unary request.

My code seems fairly straight forward, but I'm not see any traces created for the grpc requests (which are executing and succeeding)

• async controller function
• promisified client(s)

Sorry to bother, but what else am I missing for grpc trace functionality?