Unable to host books with FileDescriptor on Server

[MohitMaliFtechiz]

We introduced the hosting books feature for KiwixCustomApps a few days back, and now we are using FileDescriptor to read the zim files from the asset folder in kiwix/kiwix-android#3516. While refactoring the server functionality to use the FileDescriptor to host the zim file on Server we have seen that the server is unable to host books with FileDescriptor.

I don't know if it is possible to host zim files with FileDescriptor, if possible how much time it will take to implement.
I am creating this ticket for this bug to further conversation on this.

[veloman-yunkan]

Support in libzim for opening ZIM files via a file descriptor boils down to a hack that just converts the fd into a Unix-specific file path that is valid within the same process and as long as the original fd stays open. openzim/libzim#860 didn't change that.

Now I am not sure that API-wise it makes sense to pretend that we support opening ZIM files via file-descriptor. Instead we better support only file-paths and let the clients convert fds to paths on their own.

However, opening ZIMs via fds is different in that it also requires supplying offset and data size values.
So currently (i.e. with openzim/libzim#860 recently merged) a ZIM-reference (the data required to open a ZIM archive) is one of the following:

1. A collection of file paths (a single file being just a special case)
2. A collection of (fd, offset, data-chunk-size) triplets

I think that we should unify those two cases into "a collection of (file-path, offset, data-chunk-size) triplets", introduce in libzim a class say (ArchiveRef) for that data and just extend the rest of openzim/kiwix code so that APIs that previously took an std::string representing the file path start accepting an ArchiveRef too.

@kelson42 @mgautierfr What do you think?

[kelson42]

@veloman-yunkan @mgautierfr To me this ticket is basically a duplicate of openzim/libzim#852 and I would like to close ot at such.

[mgautierfr]

I think this issues around fd are a bit different, as they may have different root cause:

• This issue is about opening embedded (split ?) files and I think it is fixed by Be able to open zim archive from several fds. openzim/libzim#860
• Issue Allow extern getFd() to get file descriptors from Java for Kiwix Android openzim/libzim#852 is about a permission removed in the kiwix-android application which (presumably) prevent us to reopen /proc/<pid>/fd/<fd> file.

However, (if this issue is not already fixed) we should tackle this thinking globally about an api to allow us to read data from source for which we don't have a path or we cannot directly reopen it.

So no need to have several issues about that, I will close this issue in favor of openzim/libzim#852.

[MohitMaliFtechiz]

This issue is about opening embedded (split ?) files and I think it is fixed by openzim/libzim#860

@mgautierfr According to your comment for this issue, we are again introducing the hotspot feature for the custom app in kiwix/kiwix-android#4028. However, we are still experiencing the same issue reported in this issue.

Now the ZIM file is correctly opening in the reader via fd chunks, but when we are trying to host the ZIM file on the server it is not showing the ZIM file there. But the language of the book is showing on the server.

Additionally, this fd can be read by the /dev/fd/xxx, here are the logs:

Before

KiwixServer             org.kiwix.kiwixcustomcustomexample   E   FD is valid = true 
FD can read via the dev/fd/fdNumber = true

After

org.kiwix.kiwixcustomcustomexample   E  After 5 second FD is valid = true 
 FD can read via the dev/fd/fdNumber = true

I am not reopening this issue as you have close this issue in favour of openzim/libzim#852 which is the root cause of this issue. Probably this is not working due to search functionality on the server(We can search inside the ZIM file on server which usages the xapian and Xapian needs the file path) probably because of this, it is not showing the ZIM file there. @mgautierfr Please correct me if i am wrong.

But i have some question:

• The search is working in custom apps where we are creating the Archive object with fd list.
• So it should also search on the server as we are passing the Archive in the book which we are adding in Library(). So when showing the list of ZIM files on the server is it using the same archive object or is it creating a new one using a path from the book(Since the path will be null in this case as we have created the archive with fd)?

These questions making me to reopen the issue. But I do not have permission to reopen the issue, @mgautierfr can you do it?