Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(winston-transport): Serialize nested objects passed as additional attributes #2116

Conversation

JacksonWeber
Copy link
Contributor

Which problem is this PR solving?

  • If nested objects are passed as additional attributes to be logged by Winston, they will not be properly serialized and output as [object Object].

Short description of the changes

  • Runs JSON.stringify() on the contents of nested objects so they can be parsed before being emitted. If JSON.stringify() throws an error it will be caught by exception handling in the OpenTelemetryTransportV3` class.

@JacksonWeber JacksonWeber requested a review from a team April 18, 2024 23:49
@github-actions github-actions bot requested a review from hectorhdzg April 18, 2024 23:49
Copy link

codecov bot commented Apr 18, 2024

Codecov Report

Merging #2116 (c087a2e) into main (dfb2dff) will decrease coverage by 0.32%.
Report is 59 commits behind head on main.
The diff coverage is 100.00%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2116      +/-   ##
==========================================
- Coverage   90.97%   90.66%   -0.32%     
==========================================
  Files         146      147       +1     
  Lines        7492     7583      +91     
  Branches     1502     1517      +15     
==========================================
+ Hits         6816     6875      +59     
- Misses        676      708      +32     
Files Coverage Δ
.../winston-transport/src/OpenTelemetryTransportV3.ts 93.75% <ø> (ø)
packages/winston-transport/src/utils.ts 100.00% <100.00%> (ø)

... and 9 files with indirect coverage changes

@JacksonWeber JacksonWeber changed the title feat(winston-transport) Serialize nested objects passed as additional attributes feat(winston-transport): Serialize nested objects passed as additional attributes Apr 19, 2024
attributes[key] = splat[key];
}
for (const [key, value] of Object.entries(splat)) {
attributes[key] = typeof value === 'object' ? JSON.stringify(value) : value;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LogAttributes can be nested, right? Could it be that it's just not printed properly by the Log SDK's console exporter (it's depth is 3 - https://github.com/open-telemetry/opentelemetry-js/blob/31eb60dc99dc066cf2085864f2727eb29ee76e91/experimental/packages/sdk-logs/src/export/ConsoleLogRecordExporter.ts#L76) 🤔

Copy link
Contributor Author

@JacksonWeber JacksonWeber Apr 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my testing I've been using the BatchLogRecordProcessor, and when these logs get to the BatchLogRecordProcessorBase's _export method, the log records still show that nested object attributes have not been serialized and so customers receive an unserialized object on the Azure side.
I'm happy to work on this serialization for nested objects at the LogRecordProcessor level if you believe that's a more appropriate place for it. Original ask for this work was based on customer's inability to receive properly serialized objects from Winston which is why I targeted this transport specifically.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I concur with @pichlermarc. I think serialization is working property for nested objects that are logged through the winston-transport.

I tested with this "use-winston.js" script that logs a nest object after setting up new BatchLogRecordProcessor(new OTLPLogExporter()) for log record processing:

/*
cd .../opentelemetry-js-contrib
npm ci --ignore-scripts
npm run compile
cd plugins/node/opentelemetry-instrumentation-winston
npm install --no-save @opentelemetry/exporter-logs-otlp-proto
node use-winston.js
*/

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const logsAPI = require('@opentelemetry/api-logs');
const {
    LoggerProvider,
    SimpleLogRecordProcessor,
    ConsoleLogRecordExporter,
} = require('@opentelemetry/sdk-logs');
const { WinstonInstrumentation } = require('@opentelemetry/instrumentation-winston');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const {BatchLogRecordProcessor} = require('@opentelemetry/sdk-logs');
const {OTLPLogExporter} = require('@opentelemetry/exporter-logs-otlp-proto');

const tracerProvider = new NodeTracerProvider();
tracerProvider.register();
const loggerProvider = new LoggerProvider();
loggerProvider.addLogRecordProcessor(
    // new SimpleLogRecordProcessor(new ConsoleLogRecordExporter())
    new BatchLogRecordProcessor(new OTLPLogExporter())
);
logsAPI.logs.setGlobalLoggerProvider(loggerProvider);
registerInstrumentations({
    instrumentations: [
        new WinstonInstrumentation({
            // See below for Winston instrumentation options.
        }),
    ],
});

const winston = require('winston');

const log = winston.createLogger({
  transports: [new winston.transports.Console()],
})
log.info('hi', {foo: 'bar', deep: {nested: {obj: {spam: 'eggs'}}}});

loggerProvider.forceFlush() // flush before exiting

I sent that via OTLP (admittedly to a personal mock collector implemented in Node.js) and the protobuf request was this:

ExportLogsServiceRequest {
  resourceLogs: [
    ResourceLogs {
      scopeLogs: [
        ScopeLogs {
          logRecords: [
            LogRecord {
              attributes: [
                KeyValue { key: 'foo', value: AnyValue { stringValue: 'bar' } },
                KeyValue {
                  key: 'deep',
                  value: AnyValue {
                    kvlistValue: KeyValueList {
                      values: [
                        KeyValue {
                          key: 'nested',
                          value: AnyValue {
                            kvlistValue: KeyValueList {
                              values: [
                                KeyValue {
                                  key: 'obj',
                                  value: AnyValue {
                                    kvlistValue: KeyValueList {
                                      values: [ KeyValue { key: 'spam', value: AnyValue { stringValue: 'eggs' } } ]
                                    }
                                  }
                                }
                              ]
                            }
                          }
                        }
                      ]
                    }
                  }
                }
              ],
              timeUnixNano: Long { low: -750941568, high: 399051707, unsigned: true },
              severityNumber: 9,
              severityText: 'info',
              body: AnyValue { stringValue: 'hi' },
              droppedAttributesCount: 0,
              observedTimeUnixNano: Long { low: -750941568, high: 399051707, unsigned: true }
            }
          ],
          scope: InstrumentationScope { attributes: [], name: '@opentelemetry/winston-transport', version: '0.2.0' }
        }
      ],
      resource: Resource {
        attributes: [
          KeyValue { key: 'service.name', value: AnyValue { stringValue: 'unknown_service:node' } },
          KeyValue { key: 'telemetry.sdk.language', value: AnyValue { stringValue: 'nodejs' } },
          KeyValue { key: 'telemetry.sdk.name', value: AnyValue { stringValue: 'opentelemetry' } },
          KeyValue { key: 'telemetry.sdk.version', value: AnyValue { stringValue: '1.23.0' } }
        ],
        droppedAttributesCount: 0
      }
    }
  ]
}

I believe that is correct.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did some further investigation on my part and found that the serialization issue seems to be related to passing/displaying the attribute fields when they're sent to Azure. Making an update on the Azure Monitor side to fix this issue. Thank you both for your review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants