How to read and download S3 Object in Rust?

[MrKhan0747]

I want to read and download object store in S3 bucket ? How can I read and download it programmatically in rust?

[rcoh]

Please refer to this example of putobject / getobject: https://github.com/awslabs/aws-sdk-rust/blob/main/examples/s3/src/bin/s3-helloworld.rs

[ymwjbxxq]

I did in this way

let file          = format!("{}.json", client_id);
  let bucket        = std::env::var("CONFIG_BUCKETNAME").expect("CONFIG_BUCKETNAME must be set");

  let result = s3client.
        .get_object()
        .bucket(bucket)
        .key(file)
        .response_content_type("application/json")
        .send()
        .await?;

  let bytes     = result.body.collect().await?.into_bytes();
  let response  = std::str::from_utf8(&bytes)?;
  let temp: YourStruct    = serde_json::from_str(&response).unwrap();

   Ok(Some(temp.prop1))

[Keats]

Is there a way to download a file with the sdk without first collecting it in memory?

[ymwjbxxq]

Yes using streams

 let stream = aws_client
        .s3_client
        .as_ref()
        .unwrap()
        .get_object()
        .bucket(event.bucket.name.to_string())
        .key(event.object.key.to_string())
        .send()
        .await?
        .body
        .map(|result| result.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e)));

    // Convert the stream into an AsyncRead
    let stream_reader = StreamReader::new(stream);

I did it with a Lambda https://github.com/ymwjbxxq/serverless_csv_processing/blob/main/src/bin/handler.rs

[AleksandrukTad]

@ymwjbxxq just double checking. If i have 1 TB file and want to read from line 30 to line 60, will your solution allow me to do that without downloading the whole file?

[Velfi]

To only get a subset of data, I think you'd want to use the range method of get_object. However, that works on bytes and not lines. If you wanted to do lines, I don't think you could avoid reading everything up until the lines you wanted to get b/c you wouldn't know where each lines begins and ends in the object.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.