PDF File Malformed UTF-8 characters, possibly incorrectly encoded

[MuhammadAslam945]

"message": "Malformed UTF-8 characters, possibly incorrectly encoded",
"exception": "InvalidArgumentException",

$dataReader = new FileDataReader($file);
    //dd($dataReader);
    //$filePath =
    $documents = $dataReader->getDocuments();
    // dd($documents);
    // $documents = new Document($text);
    //($documents);
    $splittedDocuments = DocumentSplitter::splitDocuments($documents, 500);

    return $splittedDocuments;

    $embeddingGenerator = new OpenAI3SmallEmbeddingGenerator($config);
    $embeddedDocuments = $embeddingGenerator->embedText($text);

    return $embeddedDocuments;
    dd($embeddedDocuments);

[MuhammadAslam945]

{
"message": "Malformed UTF-8 characters, possibly incorrectly encoded",
"exception": "InvalidArgumentException",
"file": "D:\RagLara\vendor\laravel\framework\src\Illuminate\Http\JsonResponse.php",
"line": 90,
"trace": [
{
"file": "D:\RagLara\vendor\symfony\http-foundation\JsonResponse.php",
"line": 49,
"function": "setData",
"class": "Illuminate\Http\JsonResponse",
"type": "->"
},
{
"file": "D:\RagLara\vendor\laravel\framework\src\Illuminate\Http\JsonResponse.php",
"line": 32,
"function": "__construct",
"class": "Symfony\Component\HttpFoundation\JsonResponse",
"type": "->"

[MuhammadAslam945]

if ($request->hasFile('file')) {
        $file = $request->file->getRealPath();
    } else {
        return 'file not found';
    }
    $dataReader = new FileDataReader($file);

    $documents = $dataReader->getDocuments();

    $splittedDocuments = DocumentSplitter::splitDocuments($documents, 800);

    return $splittedDocuments;
    $embeddingGenerator = new OpenAI3SmallEmbeddingGenerator($config);
    $embeddedDocuments = $embeddingGenerator->embedText($text);

    $memoryVectorStore = new MemoryVectorStore;
    $memoryVectorStore->addDocuments($embeddedDocuments);

    //Once the vectorStore is ready, you can then use the QuestionAnswering class to answer questions
    $qa = new QuestionAnswering(
        $memoryVectorStore,
        $embeddingGenerator,
        new OpenAIChat
    );

    $answer = $qa->answerQuestion('what is the secret of Alice?');

    // control the behavior of the underlying vector store, see VectorStoreBase::similaritySearch
    $answer = $qa->answerQuestion('Where does Alice live?', 2, ['type' => 'city']);

[f-lombardo]

"message": "Malformed UTF-8 characters, possibly incorrectly encoded", "exception": "InvalidArgumentException",

A fix related to wrong encoding in embedding texts has just been integrated. Could you please check if it solves your problem?

[MuhammadAslam945]

Now I check It still it giving error with pdf and docs file for openAI and also if possible share Doctorine postgres sql example in detail I tried it out with PlaceEntity example but it give always through error

…

On Thu, Aug 15, 2024 at 2:06 PM Franco Lombardo ***@***.***> wrote: "message": "Malformed UTF-8 characters, possibly incorrectly encoded", "exception": "InvalidArgumentException", A fix related to wrong encoding in embedding texts has just been integrated. Could you please check if it solves your problem? — Reply to this email directly, view it on GitHub <#201 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATKIQKQVLLGZEF65XUWLFEDZRRVQBAVCNFSM6AAAAABMRFQV2KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJQHEZTIOJUGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>