Failed to read S3 .gz files using aws_s3

[satscreate]

Hi Team,
i am trying to read .gz file contents from s3 using Sqs with source component https://vector.dev/docs/reference/configuration/sources/aws_s3. But getting below error ,

ERROR source{component_kind="source" component_id=s3-source component_type=aws_s3 component_name=s3-source}: vector::internal_events::aws_s3::source: Failed to process SQS message. message_id=51c80ed6-05e1-4a3a-93e4-0ff207e0ef32 error=Could not parse SQS message with id 51c80ed6-05e1-4a3a-93e4-0ff207e0ef32 as S3 notification: missing field Records at line 1 column 30821 error_type="processing_failed" stage="processing"

Version:
vector 0.20.1 (x86_64-pc-windows-msvc 38b806e 2022-04-07)
vector.yaml:

sources:
  s3-source:
    type: aws_s3
    region: ****
    auth:
      access_key_id: ****
      secret_access_key: *****
    strategy: sqs
    sqs:
      queue_url: https://****/1234/ap-4849384938498

sinks:
  stdout:
    type: console
    inputs:
      - s3-source
    target: stdout
    encoding:
      codec: ndjson

But With the help of type: aws_sqs , i could collect the below message as json format (default its xml).

{
  "bucket": "****",
  "cid": "******",
  "fileCount": 2,
  "files": [
    {
      "checksum": "000000000001111000000000000000000",
      "path": "data/location123/1.gz",
      "size": 9404149
    },
    {
      "checksum": "000000000002222000000000000000000",
      "path": "data/location123/2.gz",
      "size": 9356626
    }
   ],
  "pathPrefix": "data/location123",
  "timestamp": 1653550032975,
  "totalSize": 2894819445
}

Debug verbose logs as follows,

DEBUG source{component_kind="source" component_id=s3-source component_type=aws_s3 component_name=s3-source}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"x-amzn-requestid": "58501423-9128-560e-b799-4f2aa4b79a65", "date": "Thu, 26 May 2022 07:59:42 GMT", "content-type": "text/xml", "content-length": "240"} body=[240 bytes]
DEBUG source{component_kind="source" component_id=s3-source component_type=aws_s3 component_name=s3-source}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://url/ method=POST version=HTTP/1.1 headers={"authorization": Sensitive, "content-length": "222", "content-type": "application/x-www-form-urlencoded", "host": "url", "x-amz-content-sha256": "blababsfsfbasbsbsbsbsbsabsabaababb", "x-amz-date": "20220526T075942Z", "user-agent": "Vector/0.20.1 (x86_64-pc-windows-msvc 38b806e 2022-04-07)", "accept-encoding": "identity"} body=[unknown]

Did i miss any config in type: aws_s3 file. Any idea?

[jszwedko]

Hi @satscreate !

That message you have there doesn't look like an SQS bucket notification. What is publishing messages to SQS? A bucket notification should look like: https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html

[satscreate]

@jszwedko correct. I guess its not the actual s3 message structure. But this gives some information about s3 bucket files. like
"path": "data/location123/1.gz",

Is there a way i can use aws_s3 source directly and ask for a 1.gz file and then read that file using single vector or multiple vector file?

[jszwedko]

Hi @satscreate !

Unfortunately not yet. The aws_s3 sink only supports reading via bucket notifications. Would it be feasible to publish SQS messages that match the bucket notification format mentioned above?

We do have an issue discussing adding more ETL-like use-cases, that I think your idea would fit within: #11095

[JeanMertz]

Closing this discussion, as we don't support this use-case yet, but have an open issue to track a potential implementation for this in the future.