Skip to content

Dev Mode: Better handle fragmented log streams that create an incomplete buffer #4

Closed
@austencollins

Description

@austencollins

In Dev Mode, this error happens:

✖ RangeError [undefined]: The value of "offset" is out of range. It must be >= 0 and <= 2. Received 4
    at boundsError (node:internal/buffer:88:9)
    at Buffer.readUInt32BE (node:internal/buffer:311:5)
    at #demuxOutput (file:///Users/assaf/.serverless/releases/4.6.2/package/dist/sf-core.js:1246:1071)
    at IncomingMessage.onData (file:///Users/assaf/.serverless/releases/4.6.2/package/dist/sf-core.js:1242:29431)
    at IncomingMessage.emit (node:events:519:28)
    at IncomingMessage.emit (node:domain:488:12)
    at addChunk (node:internal/streams/readable:559:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)
    at Readable.push (node:internal/streams/readable:390:5)
    at HTTPParser.parserOnBody (node:_http_common:131:24)

The error occurs when the log stream gets fragmented in a way that creates an incomplete buffer, which then causes the demuxer to fail when trying to read the length bytes.

The current implementation assumes each buffer contains complete frames, but Docker's log stream can split frames across multiple chunks. Causing Dev Mode to return this error.

  • We're flooding the system with large log messages
  • Docker's log stream is fragmenting these messages
  • The demuxer in Dev mode receives a fragment that's only 2-3 bytes long
  • It tries to read a 32-bit integer at offset 4 (for the length field)
  • This fails because the buffer is too small

This can be recreated with this code in the container:

// Write many logs very quickly with varying sizes
  const promises = [];
  for (let i = 0; i < 2000; i++) {
    promises.push(
      new Promise((resolve) => {
        setImmediate(() => {
          const size = Math.floor(Math.random() * 10) + 1; // Random size between 1 and 10
          if (i % 2 === 0) {
            // Very large messages to force fragmentation
            console.log(`Log message ${i} with random padding...`.repeat(size * 1000));
          } else {
            // Mix in some tiny messages to create buffer boundaries
            console.error(`Error ${i}`);
          }
          resolve();
        });
      })
    );
  }

  await Promise.all(promises);

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions