Uploading openparsed docs to Vector DB

[GedeonIsezerano]

What would be the best way to upload the parsed data to a vector db such as pinecone? The output from Documentparser provides nodes but not embedding values.

Here is my code

from openparse import processing, DocumentParser, Pdf
import os
import pinecone

OPEN_AI_KEY = os.getenv("OPENAI_API_KEY")
PINECONE_API_KEY = os.getenv("PINECONE_API_KEY")
PINECONE_ENVIRONMENT = os.getenv("PINECONE_ENVIRONMENT")

basic_doc_path = "docs/A. Sabo, A. I. Smolyakov, P. Yushmanov - Unknown - Ion temperature effects on plasma flow in the magnetic nozzle.pdf"
print(PINECONE_API_KEY)
# Initialize Pinecone
pc = pinecone.Pinecone(
    api_key=PINECONE_API_KEY,
)

# Create a Pinecone index (if it doesn't exist)
index_name = "paper-embeddings"

# Semantic Processing Pipeline
semantic_pipeline = processing.SemanticIngestionPipeline(
    openai_api_key=OPEN_AI_KEY,
    model="text-embedding-3-large",
    min_tokens=64,
    max_tokens=1024,
)
parser = DocumentParser(
    processing_pipeline=semantic_pipeline,
)
parsed_content = parser.parse(basic_doc_path)

# Store Embeddings in Pinecone
index = pc.Index(index_name)
print(parsed_content.json())

[Filimoa]

I'm not familiar with the details of pinecone, but each node has a text attribute. You can embed this with your standard workflow.

import openparse

basic_doc_path = "./sample-docs/mobile-home-manual.pdf"
parser = openparse.DocumentParser()
parsed_basic_doc = parser.parse(basic_doc_path)
content_dict = parsed_basic_doc.dict()

for node in content_dict["nodes"]:
    node["vector"] = your_function_to_embed_text(node["text"])

We will likely add the vector attribute directly to the node but currently it get's thrown out after the pipeline is finished running.

I will update this answer when that gets implemented.