Cant load the ChromaDB again

[BGroenberg]

Hi

I made an dataloader, that loads all my documents into ChromaDB using embedchain. My code looks like this.

from embedchain import App
import os
import glob
from pydantic import FilePath
import os

app = App.from_config(config_path="./config_advisor.yaml")

folder_path = ".\Testdoc"
file_paths = []
for root, dirs, files in os.walk(folder_path):
    for file in files:
        file_paths.append(os.path.join(root, file))

for file_path in file_paths:
    print(file_path)
    app.add(file_path)

Looking at the console, it works fine and loads the data as expected into ChromaDB.
If i move this code directly into the chat app, everything works and im able to "chat" with my documents.

But if i seperate that code, so i got a loader app - and the chat app, it wont load data from the db.

This is basicly what i do to try and load the ChromaDB, when starting my chat app.

from embedchain import App
app = App.from_config(config_path="./config_advisor.yaml")

print(app.get_data_sources())

How do i get this to work ? I want to be able to reuse the persistet ChromaDB.

[amber-beasley-liatrio]

There seems to be a bug in the get_data_sources() method. It eventually tried to get the "data_type" attribute but that throws a no attribute error.

I believe this is the release that introduced the bug: https://github.com/embedchain/embedchain/releases/tag/0.1.82

The model for the data uses "type" but the method that pulls the data is using "data_type"

PR: https://github.com/embedchain/embedchain/pull/1273/files#diff-d494029527663b5dde22e3dc804d17421db5bdc6cdf0d191d3cb294e18aaaff4R16

Code:

• "type": https://github.com/embedchain/embedchain/blob/6518c0c06bc32a0163c0c7407c400cf18a883df2/embedchain/core/db/models.py#L16
• "date_type": https://github.com/embedchain/embedchain/blob/6518c0c06bc32a0163c0c7407c400cf18a883df2/embedchain/app.py#L298

[amber-beasley-liatrio]

reported bug: #1292

[BGroenberg]

Im running version 0.1.8.1, so it really not app.get_data_sources() that's my problem, as far as i can see.

My problem is, that im not able to reuse data fra my ChromaDB. Everytime i restart my app, it has to run the embeddings again. With +100 documents, that quite the pain.

[amber-beasley-liatrio]

Do you have is_persistent set to true in your config.yaml ?

[BGroenberg]

I just need to set the "id" of the application in the config file.
It's working now :)

[trnq-eu]

I have a similar problem. I'm trying to set up a persistent chroma db in order to query my knownledge base without having to upload my files every time. I started with this example: https://github.com/embedchain/embedchain/tree/main/examples/chat-pdf
In the example the app stores the database into a temporary directory. How can I change this to have a persistent database? Thanks.