requests_toolbelt.MultipartEncoder fails to encode S3File objects due to missing len property

[kformanowicz-dotdata]

Hello,

I'm unable to use a S3File object together with requests_toolbelt.MultipartEncoder (https://toolbelt.readthedocs.io/en/latest/user.html). The MultipartEncoder expects the File object to contain len property to calculate its size and fails with the following:

File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 125, in __init__
    self._prepare_parts()
  File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 246, in _prepare_parts
    self.parts = [Part.from_field(f, enc) for f in self._iter_fields()]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 246, in <listcomp>
    self.parts = [Part.from_field(f, enc) for f in self._iter_fields()]
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 495, in from_field
    return cls(headers, body)
           ^^^^^^^^^^^^^^^^^^
  File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 488, in __init__
    self.len = len(self.headers) + total_len(self.body)
                                   ^^^^^^^^^^^^^^^^^^^^
  File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 432, in total_len
    if hasattr(o, 'len'):
       ^^^^^^^^^^^^^^^^^
  File "/home/krzysztof/venvs/py311/lib/python3.11/site-packages/requests_toolbelt/multipart/encoder.py", line 573, in len
    return total_len(self.fd) - self.fd.tell()
           ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

It works with a custom adapter like this:

class S3FileWithLenAdapter:
    def __init__(self, s3_file):
        self._s3_file = s3_file

    @property
    def len(self):
        return self._s3_file.size

    def __getattr__(self, name):
        return getattr(self._s3_file, name)

Could you consider adding len property to S3File objects?

[martindurant]

Is it really len, or __len__() that you need?

One subtle problem, is that files already can be iterated, but the unit is one line rather than one character. So len(list(file)) will not be the same as then len we are talking about above.

[kformanowicz-dotdata]

@martindurant Thanks for replying. The MultipartEncoder actually checks the following attrs in its total_len() function:

def total_len(o):
    if hasattr(o, '__len__'):
        return len(o)

    if hasattr(o, 'len'):
        return o.len

    if hasattr(o, 'fileno'):
        try:
            fileno = o.fileno()
        except io.UnsupportedOperation:
            pass
        else:
            return os.fstat(fileno).st_size

    if hasattr(o, 'getvalue'):
        # e.g. BytesIO, cStringIO.StringIO
        return len(o.getvalue())

So adding any of those properties would solve my issue.

[martindurant]

OK, then let's add len as a property as you originally suggested. This can be done in the upstream AbstractBufferedFile in fsspec.