ftello() issue with big files

[bucanero]

When using ftello() with a big file (over 2GB), the returned value is invalid (.e.g 0xffffffff88776655) as if it suffered overflow from a 32-bit limit.

I'd expect such behavior from ftell(), but since ftello() return a off_t type (64 bits) I'd have expected a valid value.

Btw, using stat() on the same file and getting the size attribute returns the expected file size value.

Example:

    FILE *f = fopen("/dev_hdd0/tmp/bigfile.bin", "rb");
    fseeko(f, 0, SEEK_END);
    off_t TotalSize = ftello(f);
    fclose(f);
    printf("File size: %016llX", TotalSize);

[bucanero]

any idea about this issue/bug? @zeldin @miigotu @shagkur

thanks!

[miigotu]

@bucanero methinks your printf has the wrong format specifier.

printf​(​"​File size: ​%lld​"​, TotalSize);

[zeldin]

Actually, ftello in newlib simply calls ftell and then casts the result to an off_t. So you can expect the same behaviour from ftello as from ftell. Still, long should be 64 bits since the PS3 is a 64-bit system, so over 2GB files should be fine with both ftell and ftello...

[bucanero]

hello, thanks for the answers 👍

@bucanero methinks your printf has the wrong format specifier.

printf​(​"​File size: ​%lld​"​, TotalSize);

oh I fixed my printf (now using %016llX) but I still got a wrong value. :(
For example, from a 4311408640 bytes file (0x100FAE000) I've got:

[2021-03-14 02:00:10] File size: 0000000000FAE000

Actually, ftello in newlib simply calls ftell and then casts the result to an off_t. So you can expect the same behaviour from ftello as from ftell. Still, long should be 64 bits since the PS3 is a 64-bit system, so over 2GB files should be fine with both ftell and ftello...

from my test above (the ~4 GB file) it seems that somewhere there might be a 32-bit conversion, even if everything is 64 bits. Any ideas where I could look? newlib or somewhere else?

[zeldin]

from my test above (the ~4 GB file) it seems that somewhere there might be a 32-bit conversion, even if everything is 64 bits. Any ideas where I could look? newlib or somewhere else?

I'd start by looking at the libsysbase implementation of lseek. Maybe there is an lv2 call that truncates the value.

[miigotu]

Yeah without setting up and doing some tests and debugging I have no clue where to point you to.

[bucanero]

I'd start by looking at the libsysbase implementation of lseek. Maybe there is an lv2 call that truncates the value.

thanks @zeldin , I'll take a look around lseek when I have a chance 👍

[bucanero]

I was looking around, and by chance I found this old psl1ght example code from Estwald, where he is using _lseek64_r for proper 2GB+ file size support:

// NOTE: for files > 2GB you can use 64 bit method

    #ifdef USE_64BIT_SEEK

    struct _reent reent1;

    DPrintf ("Using _lseek64_r for large files\n");

    s64 size = _lseek64_r(&reent1, fp->_file, 0, SEEK_END);
    
    _lseek64_r(&reent1, fp->_file, 0, SEEK_SET);

    DPrintf ("Size of file %i bytes\n", (int) size);

    #else

    fseek(fp, 0, SEEK_END);

    int size = ftell(fp);

    DPrintf ("Size of file %i bytes\n", size);

    fseek(fp, 0, SEEK_SET);

    #endif

could be possible to fix ftello() by using _lseek64_r(), as done in this example?

[zeldin]

@bucanero AFAICT both __librt_lseek64_r and __librt_lseek_r in librt call the same LV2 function sysLv2FsLSeek64. So they should give the same result. The only difference is that one returns the value as an _off64_t and the other as an _off_t. Both these seem to be typedeffed to long long though...