read error: Input/output error

[voidking]

I created a juicefs file system. It is used for persistent storage of pods in k8s.
I encountered a strange problem:
When I directly mounted juicefs in VM (juicefs mount -d "mysql://juicefs@(172.x.x.x:3306)/juicefs" /mnt/k8sjfs), all files can be read normally.

However, when I mounted the same path to pod through pv, most of the files can be read normally, but some files cannot be read. When viewing the file, an error is reported: Input/output error

I tried a few possible solutions:

1. Delete and rebuild the mount pod, the problem still exists.

2. Start the pod on another node, and the problem still exists.

3. juicefs version is the same.
   

4. Change a pod image, the problem still exists.
   

5. In the same path, a new file can be read normally.
   

According to my attempts, I can solve this problem by rebuilding the problematic files. However, this is not a good solution. Because I don't know when this problem will occur again.
Moreover, the problematic file looks no different from other files in the VM.

I haven't found the same problem in the issues and discussions history, and I don't know how to solve this problem completely. I hope to get some troubleshooting ideas. Thank you.

[zhijian-pro]

input/output errror usually occurs when the object storage fails to be read.
You can collect logs of the mount pod when the file fails to be read.

[voidking]

Thanks for your reply.
I checked the mount pod log and indeed found an error.

2024/08/22 09:07:34.738952 juicefs[38] <WARNING>: fail to read sliceId 493781 (off:0, size:3439, clen: 3439, inode: 614496): get chunks/0/493/493781_0_3439: status: 404, message: {"RetCode":-148654, "ErrMsg":"file not exist"} [reader.go:809]
2024/08/22 09:07:38.150505 juicefs[38] <WARNING>: GET chunks/0/493/493781_0_3439: status: 404, message: {"RetCode":-148654, "ErrMsg":"file not exist"}; retrying [cached_store.go:710]
2024/08/22 09:07:38.171880 juicefs[38] <WARNING>: fail to read sliceId 493781 (off:0, size:3439, clen: 3439, inode: 614496): get chunks/0/493/493781_0_3439: status: 404, message: {"RetCode":-148654, "ErrMsg":"file not exist"} [reader.go:809]
2024/08/22 09:07:42.020420 juicefs[38] <WARNING>: GET chunks/0/493/493781_0_3439: status: 404, message: {"RetCode":-148654, "ErrMsg":"file not exist"}; retrying [cached_store.go:710]
2024/08/22 09:07:42.024058 juicefs[38] <WARNING>: fail to read sliceId 493781 (off:0, size:3439, clen: 3439, inode: 614496): get chunks/0/493/493781_0_3439: status: 404, message: {"RetCode":-148654, "ErrMsg":"file not exist"} [reader.go:809]
2024/08/22 09:07:42.024106 juicefs[38] <ERROR>: read file 614496: input/output error [reader.go:123]

I have checked the path chunks/0/493/493781_0_3439 in object storage bucket. This file does not exist.
Why can it be read normally in VM?

[zhijian-pro]

You may have enabled the --writeback parameter.
There is a write cache on the machine on which the write is being written, but no other machine does not have the cache, and the block is not uploaded to the object store, so it cannot be read by other mount points
https://juicefs.com/docs/community/guide/cache/#client-write-cache

[voidking]

You may have enabled the --writeback parameter. There is a write cache on the machine on which the write is being written, but no other machine does not have the cache, and the block is not uploaded to the object store, so it cannot be read by other mount points https://juicefs.com/docs/community/guide/cache/#client-write-cache

You are right. I deleted the cache files in /var/jfsCache and then remounted juicefs. Then I read the file again, the same error as in the pod appeared.

Another question I have is why does this error occur? How can I avoid it as much as possible?

[zhijian-pro]

Write cache data by default is stored in /var/jfsCache//rawstaging/, do not delete files under this directory or data will be lost.

Read the documentation mentioned above for details