MemoryError when downloading a large file from S3 with community.aws 8.0.0

[eacherkan-aternity]

Summary

When downloading a ~650MB file from S3 with the aws_s3 module, a MemoryError is thrown. The same role works well with community.aws 7.2.0.

Issue Type

Bug Report

Component Name

aws_s3

Ansible Version

$ ansible --version
ansible [core 2.15.10]
  config file = /runner/ansible/ansible.cfg
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True

Collection Versions

$ ansible-galaxy collection list
# /runner/.ansible/collections/ansible_collections
Collection              Version
----------------------- -------
amazon.aws              8.0.0  
community.aws           8.0.0  
community.crypto        2.20.0 
community.general       9.0.0  
community.postgresql    3.4.1  
community.windows       2.2.0  

# /usr/share/ansible/collections/ansible_collections
Collection              Version
----------------------- -------
amazon.aws              7.5.0  
ansible.posix           1.5.4  
ansible.windows         2.3.0  
awx.awx                 24.2.0 
azure.azcollection      2.3.0  
community.vmware        4.2.0  
google.cloud            1.3.0  
kubernetes.core         3.0.1  
kubevirt.core           1.2.3  
openstack.cloud         2.2.0  
ovirt.ovirt             3.2.0  
redhatinsights.insights 1.2.2  
theforeman.foreman      4.0.0

AWS SDK versions

$ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: [email protected]
License: MIT
Location: /runner/.local/lib/python3.9/site-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.34.81
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/local/lib/python3.9/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.34.81
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/local/lib/python3.9/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
ANSIBLE_NOCOWS(/runner/ansible/ansible.cfg) = True
CONFIG_FILE() = /runner/ansible/ansible.cfg
DEFAULT_LOAD_CALLBACK_PLUGINS(/runner/ansible/ansible.cfg) = True
DEFAULT_PRIVATE_KEY_FILE(/runner/ansible/ansible.cfg) = /runner/ansible/playbooks/ssh/id_rsa_ansible
DEFAULT_REMOTE_USER(/runner/ansible/ansible.cfg) = ec2-user
DEFAULT_ROLES_PATH(/runner/ansible/ansible.cfg) = ['/runner/ansible/roles']
DEFAULT_STDOUT_CALLBACK(/runner/ansible/ansible.cfg) = yaml
DEFAULT_TIMEOUT(/runner/ansible/ansible.cfg) = 30
DEPRECATION_WARNINGS(/runner/ansible/ansible.cfg) = True
HOST_KEY_CHECKING(/runner/ansible/ansible.cfg) = False

OS / Environment

Docker (quay.io/ansible/awx-ee:latest image) on Amazon Linux 2

Steps to Reproduce

- name: download {{ release_bucket }}/{{ env_type }}/{{ s3_path }} to {{ local_path_to_file }}/{{full_file_name}}
  local_action:
    module: aws_s3
    bucket: "{{ release_bucket }}"
    object: /saas/{{ s3_path }}
    dest: "{{ local_path_to_file }}/{{full_file_name}}"
    mode: get

Expected Results

Successful file download.

Actual Results

 An exception occurred during task execution. To see the full traceback, use -vvv. The error was: MemoryError
 fatal: [localhost]: FAILED! => changed=false 
   module_stderr: |-
     Traceback (most recent call last):
       File "/runner/.ansible/tmp/ansible-tmp-1716344439.335101-90-219842493484346/AnsiballZ_s3_object.py", line 107, in <module>
         _ansiballz_main()
       File "/runner/.ansible/tmp/ansible-tmp-1716344439.335101-90-219842493484346/AnsiballZ_s3_object.py", line 99, in _ansiballz_main
         invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
       File "/runner/.ansible/tmp/ansible-tmp-1716344439.335101-90-219842493484346/AnsiballZ_s3_object.py", line 47, in invoke_module
         runpy.run_module(mod_name='ansible_collections.amazon.aws.plugins.modules.s3_object', init_globals=dict(_module_fqn='ansible_collections.amazon.aws.plugins.modules.s3_object', _modlib_path=modlib_path),
       File "/usr/lib64/python3.9/runpy.py", line 225, in run_module
         return _run_module_code(code, init_globals, run_name, mod_spec)
       File "/usr/lib64/python3.9/runpy.py", line 97, in _run_module_code
         _run_code(code, mod_globals, init_globals,
       File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
         exec(code, run_globals)
       File "/tmp/ansible_aws_s3_payload_9kel_yf0/ansible_aws_s3_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1592, in <module>
       File "/tmp/ansible_aws_s3_payload_9kel_yf0/ansible_aws_s3_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1579, in main
       File "/tmp/ansible_aws_s3_payload_9kel_yf0/ansible_aws_s3_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 1022, in s3_object_do_get
       File "/tmp/ansible_aws_s3_payload_9kel_yf0/ansible_aws_s3_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 788, in download_s3file
       File "/tmp/ansible_aws_s3_payload_9kel_yf0/ansible_aws_s3_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_object.py", line 503, in _get_object_content
       File "/usr/local/lib/python3.9/site-packages/botocore/response.py", line 99, in read
         chunk = self._raw_stream.read(amt)
       File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 567, in read
         data = self._fp_read(amt) if not fp_closed else b""
       File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 533, in _fp_read
         return self._fp.read(amt) if amt is not None else self._fp.read()
       File "/usr/lib64/python3.9/http/client.py", line 476, in read
         s = self._safe_read(self.length)
       File "/usr/lib64/python3.9/http/client.py", line 631, in _safe_read
         return b"".join(s)
     MemoryError
   module_stdout: ''
   msg: |-
     MODULE FAILURE
     See stdout/stderr for the exact error
   rc: 1

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[tremble]

Caused by #1139

Thanks for opening this issues @eacherkan-aternity

The aws_s3 module was renamed to s3_object and lives in the amazon.aws collection, so this shouldn't be related to the community.aws version. I've taken a quick look at the code and I suspect the fix is a simple one, I'm trying to test it now.