-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] Utility Functions for Pinecone 1.0.1 #117
Comments
Thanks for the question. We're definitely interested in making the experience as smooth as possible. My general feeling about the legacy utils is that although they are useful, especially when repeatedly doing those actions in integration testing (which is what they were originally used for), the fact they were/are needed reflects shortcomings in the client itself. I think ideally you should have the client do whatever it is you need without resorting to additional utility wrappers. waitUntilIndexIsReadyThis utility handled polling for status to see if the index is ready to handle data operations. This should be covered with the newly added import { Pinecone } from '@pinecone-database/pinecone'
const pinecone = new Pinecone()
await pinecone.createIndex({
name: 'my-index',
dimension: 1586,
waitUntilReady: true // yay, this exists today
}) createIndexIfNotExistsThe legacy utility implementation makes 2 API calls:
I think a better way to handle this with a single call would be to expose an option in import { Pinecone } from '@pinecone-database/pinecone'
const pinecone = new Pinecone()
await pinecone.createClient({
name: 'index-name',
dimension: 1586,
suppressConflicts: true // proposed option, open to ideas for better names
}) I don't have a specific timeframe in mind for this, but it's something I plan to do at some point. If you wanted to PR this change, I'd be happy to review it. chunkedUpsertWhere the legacy utility falls down is when things go wrong with one of the upserts. If it fails, what do you do? You end up running it again and re-upserting everything, which can be a huge waste of time for large datasets. This wasn't a concern in the narrow integration testing and sample app scenarios this util was made for, but in serious production work we need to have a better answer than "retry everything again". There was a point during the pre-release development when we had a Let's say you have a scenario like this: import { Pinecone } from '@pinecone-database/pinecone'
const pinecone = new Pinecone()
const index = pinecone.index('my-index')
const records = [
// you use an embedding model to make, let's say 100k vectors
]
await index.upsert({
records,
batchSize: 100 // not available in the current release
}) In that case, upsert needs to make 1000 API (100k records / 100 batchSize) calls on your behalf. If a handful of these many calls fail, what should it do? How can the method expose to the caller what the failures were? Should it just automatically retry with exponential backoff? How many retries? Timeouts? I do think implementing it in the |
Thank you for your thorough response and insights. It's clear that you're committed to making the integration smoother and more efficient. I appreciate the explanation, and your approach to addressing their shortcomings makes sense, your insights and input are invaluable. 1. Suppress Conflicts Option: 2. Upsert Error Handling and Retry Strategies (Future Discussion): Thank you once again for your responsiveness and dedication to improving Pinecone. |
Is this your first time submitting a feature request?
Describe the feature
I'd like to address a specific concern regarding our project's compatibility with Pinecone 1.0. In the latest Pinecone release, functions such as chunkedUpsert and createIndexIfNotExists have been deprecated but are retained for retro-compatibility.
My question is simple: Are there plans to implement updated utility functions to replace these deprecated ones within our project, or is it expected that users will need to implement these changes independently to ensure Pinecone 1.0 compatibility?
Your insights on this matter would be greatly appreciated.
Thank you,
Describe alternatives you've considered
These functions have been deprecated in Pinecone 1.0, which prompted me to take the initiative to reimplement them as utilities that are compatible with Pinecone 1.0.
Who will this benefit?
It's worth noting that implementing updated utility functions to replace these deprecated ones could have significant benefits. Not only are these utilities very simple to use, but they also have the potential to expedite the development process for all users, enhancing productivity and ensuring a smooth transition to Pinecone 1.0.
Are you interested in contributing this feature?
I'm more than willing to contribute to the integration process and help with any necessary modifications.
Anything else?
No response
The text was updated successfully, but these errors were encountered: