Skip to content

Commit 1e4603d

Browse files
cymbalrushfacebook-github-bot
authored andcommitted
FileDataLoader fails to read the file when size > INT32_MAX (#4435)
Summary: On macOS, the `read` function will fail with an `EINVAL` error if the size parameter exceeds `INT32_MAX`. This update addresses the issue by adding a check to ensure that the read size does not surpass `INT32_MAX`. On Linux, the maximum permissible read size is 2,147,479,552 bytes ( < `INT32_MAX`), so attempting to read beyond this limit is inconsequential. Pull Request resolved: #4435 Test Plan: Exporting llama3 with `python -m examples.models.llama2.export_llama --checkpoint examples/models/llama-2-7B/consolidated.00.pth --params examples/models/llama-2-7B/params.json --coreml --disable_dynamic_shape -kv ` Without fix Fails with `invalid argument` error. With fix Succeeds. Reviewed By: kirklandsign Differential Revision: D60321719 Pulled By: shoumikhin fbshipit-source-id: fca265c6c1edc628b38a5044693ec7bbe0c0b43a
1 parent 5a20a49 commit 1e4603d

File tree

1 file changed

+8
-1
lines changed

1 file changed

+8
-1
lines changed

extension/data_loader/file_data_loader.cpp

+8-1
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,11 @@
88

99
#include <executorch/extension/data_loader/file_data_loader.h>
1010

11+
#include <algorithm>
1112
#include <cerrno>
1213
#include <cstddef>
1314
#include <cstring>
15+
#include <limits>
1416

1517
#include <fcntl.h>
1618
#include <sys/stat.h>
@@ -189,7 +191,12 @@ Result<FreeableBuffer> FileDataLoader::load(
189191
size_t needed = size;
190192
uint8_t* buf = reinterpret_cast<uint8_t*>(aligned_buffer);
191193
while (needed > 0) {
192-
ssize_t nread = ::read(fd_, buf, needed);
194+
// Reads on macos will fail with EINVAL if size > INT32_MAX.
195+
ssize_t nread = ::read(
196+
fd_,
197+
buf,
198+
std::min<size_t>(
199+
needed, static_cast<size_t>(std::numeric_limits<int32_t>::max())));
193200
if (nread < 0 && errno == EINTR) {
194201
// Interrupted by a signal; zero bytes read.
195202
continue;

0 commit comments

Comments
 (0)