Final newline character missing when using AWS S3 sink

[miquelruiz]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When using the AWS S3 sink to upload local logs, the last line of each uploaded object is missing a newline character at the end. This is not the case when using the File sink.

It seems to be caused by the S3 sink calling encoder.serialize on the last event of the batch (https://github.com/vectordotdev/vector/blob/master/src/sinks/util/encoding.rs#L51) while the file sink calls encoder.encode on all events (https://github.com/vectordotdev/vector/blob/master/src/sinks/file/mod.rs#L427).

I would expect these to behave in the same way using the provided config.

Configuration

[sources.messages]
type = "file"
include = [
  "/var/log/messages",
]

[sinks.s3]
inputs = ["messages"]
type = "aws_s3"
bucket = "<redacted>"
region = "us-east-1"
key_prefix = "<redacted>"
healthcheck.enabled = false
filename_append_uuid = false
filename_extension = ""
compression = "none"
encoding.codec = "text"
auth.access_key_id = "<redacted>"
auth.secret_access_key = "<redacted>"
batch.timeout_secs = 60

[sinks.file]
type = "file"
inputs = [ "messages" ]
path = "/tmp/messages-%Y%m%d-%H%M%S.log"
encoding.codec = "text"

Version

vector 0.40.0 (aarch64-unknown-linux-gnu 1167aa9 2024-07-29 15:08:44.028365803)

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Good spot @miquelruiz

I think

vector/src/sinks/util/encoding.rs

Lines 48 to 59 in f7e4470

	match position {
	Position::Last | Position::Only => {
	encoder
	.serialize(event, &mut bytes)
	.map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))?;
	}
	_ => {
	encoder
	.encode(event, &mut bytes)
	.map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))?;
	}
	}

is attempting to handle proper batch encoding when the batch is being encoded as a JSON array. If encode is used, a , is appended to each element as it is encoded, but when encoding as a JSON array the trailing , isn't wanted which is why serialize is used instead.

I think a possible fix for this would be to update:

vector/src/codecs/encoding/encoder.rs

Lines 96 to 104 in f7e4470

	pub const fn batch_suffix(&self) -> &[u8] {
	match (&self.framer, &self.serializer) {
	(
	Framer::CharacterDelimited(CharacterDelimitedEncoder { delimiter: b',' }),
	Serializer::Json(_) | Serializer::NativeJson(_),
	) => b"]",
	_ => &[],
	}
	}

To have it add a \n if the framer is Framer::NewlineDelimited.

[jszwedko]

Opened a PR: #21097