Jan: Advanced RAG supporting multi-file and multi-format

[eckartal]

Overview

Must-have feature. We should improve Jan's RAG feature to allow users to chat with their files and create their knowledge base like assistants with their files.

Goal

• Providing RAG capable of processing and indexing multiple files simultaneously
• Support a wide range of file formats (e.g., pdf, docx, txt, csv, json, html)
• Allowing people to manage and organize multi-file knowledge bases (maybe like assistants?)

[rvrstfn]

Some random links and keywords

https://github.com/AnswerDotAI/byaldi

https://x.com/mervenoyann/status/1832343855477264530?t=Cl14WHEmIGjxcsxALe-BXw&s=19

https://x.com/mervenoyann/status/1831737088468791711?t=Pt47N799cvMv7DjaO_eaAw&s=19

[nl-jj]

IMO RAG doesn't seem to consider multiple uploaded documents (pdf), while they are in the thread history (on M3, 24 GB (Llama 3.1 8B instruct). Also, it would be great to be able to add them as links to the file system as can be reached by the logged in user.