SQS Parser failed

[cnwobi]

Every attempt to reach logs from an S3 bucket is failing

source{component_kind="source" component_id=archival_logs component_type=aws_s3 component_name=archival_logs}: vector::internal_events::aws_sqs::s3: Failed to process SQS message. message_id=596ee12e-ad8f-442f-add5-fccf41406819 error=Failed to fetch s3://some-bucket/some_filename.log.gz: service error error_code="failed_processing_sqs_message" error_type="parser_failed" stage="processing" internal_log_rate_limit=true

When I download this file I can read and parse each line as valid JSON in Ruby.

[jszwedko]

Hi @cnwobi ,

Are you able to reliably reproduce this? If so, could you try setting the following environment variable to get more details from the AWS SDK:

VECTOR_LOG=aws_smithy_runtime=debug,debug vector ...

This should give more details about why the request failed.

[cnwobi]

Hi @jszwedko, Thanks for your prompt response.

I have set the env var for debug

k8s deployment.yaml snipper

          env:
            - name: VECTOR_LOG
              value: "aws_smithy_runtime=debug,debug"

Container logs

2024-11-07T12:13:45.242170Z  INFO vector::app: Log level is enabled. level="aws_smithy_runtime=debug,debug"
2024-11-07T12:13:45.242361Z DEBUG vector::app: messaged="Building runtime." worker_threads=1
2024-11-07T12:13:45.242731Z  INFO vector::app: Loading configs. paths=["/etc/vector"]

However I don't see more details about the error.

2024-11-07T12:13:56.535135Z ERROR source{component_kind="source" component_id=foo_id component_type=aws_s3 component_name=foo}: vector::internal_events::aws_sqs::s3: Internal log [Failed to process SQS message.] has been suppressed 48 times.
2024-11-07T12:13:56.535145Z ERROR source{component_kind="source" component_id=foo_id component_type=aws_s3 component_name=foo}: vector::internal_events::aws_sqs::s3: Failed to process SQS message. message_id=a14a6bf7-c489-40a0-b3f7-c38a15839b04 error=Failed to fetch s3://foo/logs/foobar.log.gz: service error error_code="failed_processing_sqs_message" error_type="parser_failed" stage="processing" internal_log_rate_limit=true
2024-11-07T12:13:56.535180Z DEBUG source{component_kind="source" component_id=foo_id component_type=aws_s3 component_name=foo}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}: aws_credential_types::cache::lazy_caching: loaded credentials from cache

[jszwedko]

Thanks! That could indicate that the SQS is failing to be parsed correctly as an S3 bucket notification. How have you setup the bucket notifications from S3 to SQS? Any chance you could grab and share one of the SQS messages that fails to be processed?

[cnwobi]

Thank you so much, the error was a permissions issue, object was owned by a different aws principal so even though vector had the right access to the bucket.

Changing the ownership for the objects fixed the error.