Implement sparse index support #95

austin-denoble · 2025-01-25T20:45:47Z

Problem

2025-01 includes changes to support sparse indexes. We need to adjust several types and functions to support working with these kinds of indexes.

Solution

Update CreateServerlessIndexRequest to allow creating sparse or dense serverless indexes.
Update Index struct to support VectorType and IndexEmbed. Add new IndexEmbed type.
Add new unit tests / update existing unit & integration tests.

package main

import (
	"context"
	"fmt"
	"github.com/pinecone-io/go-pinecone/v2/pinecone"
	"log"
	"os"
)

func main() {
	ctx := context.Background()

	clientParams := pinecone.NewClientParams{
		ApiKey: os.Getenv("PINECONE_API_KEY"),
	}

	pc, err := pinecone.NewClient(clientParams)
	if err != nil {
		log.Fatalf("Failed to create Client: %v", err)
	} else {
		fmt.Println("Successfully created a new Client object!")
	}

	indexName := "my-serverless-index"
	vectorType := "sparse"

	idx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{
		Name:       indexName,
		Metric:     pinecone.Dotproduct,
		VectorType: &vectorType,
		Cloud:      pinecone.Aws,
		Region:     "us-east-1",
		Tags:       &pinecone.IndexTags{"environment": "development"},
	})

	if err != nil {
		log.Fatalf("Failed to create serverless index: %v", err)
	} else {
		fmt.Printf("Successfully created serverless index: %s", idx.Name)
	}
}

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update
Infrastructure change (CI configs, etc)
Non-code change (docs, etc)
None of the above: (explain here)

Test Plan

CI / CD (unit / integration tests)

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1208897809680488
- https://app.asana.com/0/0/1208897809680503

…nt for new deserialization

… and associated methods, add some additional error validation for the different scenarios, update unit / integration tests

austin-denoble · 2025-01-28T16:54:58Z

pinecone/client.go

@@ -540,7 +540,7 @@ func (req CreatePodIndexRequest) TotalCount() int {
 //			   fmt.Printf("Successfully created pod index: %s", idx.Name)
 //		}
 func (c *Client) CreatePodIndex(ctx context.Context, in *CreatePodIndexRequest) (*Index, error) {
-	if in.Name == "" || in.Dimension == 0 || in.Metric == "" || in.Environment == "" || in.PodType == "" {
+	if in.Name == "" || in.Dimension <= 0 || in.Metric == "" || in.Environment == "" || in.PodType == "" {


Refining the check on Dimension here - if it's negative we don't really need to make a request.

nit: Will Dimension <= 0 throw an error message "fields Name, Dimension, Metric, Environment, and Podtype must be included in CreatePodIndexRequest" even though the Dimension is included as a part of the request?

Good call, I can update to mention "positive Dimension" to make it a bit more clear.

austin-denoble · 2025-01-28T16:55:24Z

pinecone/client.go

@@ -549,6 +549,7 @@ func (c *Client) CreatePodIndex(ctx context.Context, in *CreatePodIndexRequest)
 	pods := in.TotalCount()
 	replicas := in.ReplicaCount()
 	shards := in.ShardCount()
+	vectorType := "dense"


sparse cannot be selected for pod indexes so we default for the pod creation path.

austin-denoble · 2025-01-28T16:56:38Z

pinecone/client_test.go

+	require.ErrorContainsf(t, err, "fields Name, Metric, Cloud, and Region must be included in CreateServerlessIndexRequest", err.Error())
+}
+
+func TestCreateServerlessIndexInvalidSparseDimensionUnit(t *testing.T) {


The following are all unit tests to check the validation in the serverless creation path.

austin-denoble · 2025-01-28T17:01:07Z

pinecone/models_test.go

@@ -7,7 +7,7 @@ import (
 	"google.golang.org/protobuf/types/known/structpb"
 )

-func TestMarshalIndexStatus(t *testing.T) {
+func TestMarshalIndexStatusUnit(t *testing.T) {


Noticed all these tests were missing the "Unit" suffix, which is needed for them to be included when running just test-unit.

rohanshah18 · 2025-01-28T18:16:08Z

README.md

+	}
+
+	indexName := "my-serverless-index"
+	vectorType := "dense"


did you mean to show an example for vectorType = sparse?

rohanshah18

LGTM, left a few comments!!

haruska

Looks good

haruska · 2025-02-04T20:45:07Z

pinecone/client.go

+			if in.Dimension != nil {
+				return nil, fmt.Errorf("dimension should not be specified when VectorType is 'sparse'")
+			} else if in.Metric != Dotproduct {
+				return nil, fmt.Errorf("metric should be 'dotproduct' when VectorType is 'sparse'")


metric should be optional for sparse.

austin-denoble added 3 commits January 25, 2025 15:40

update types to handle new sparse index fields, update tests to accou…

4893bc9

…nt for new deserialization

modify CreatePodIndexRequest and CreateServerlessIndexRequest structs…

6a62213

… and associated methods, add some additional error validation for the different scenarios, update unit / integration tests

add unit tests for sparse index validation

ca64d30

austin-denoble marked this pull request as ready for review January 28, 2025 04:55

austin-denoble commented Jan 28, 2025

View reviewed changes

austin-denoble added 2 commits January 28, 2025 12:41

add doc comment and unit tests for IndexEmbed

e40dbf8

add sparse serverless example to README

767fb70

austin-denoble requested review from haruska, jhamon and rohanshah18 January 28, 2025 17:58

rohanshah18 reviewed Jan 28, 2025

View reviewed changes

pull in latest spec and codegen changes

4665c1a

rohanshah18 approved these changes Jan 28, 2025

View reviewed changes

austin-denoble added 3 commits January 30, 2025 00:11

update Vector type to allow optional Values

1ddb626

review feedback, make sure Values is a pointer on Vector

4bf4e9e

fix assertions in vector update tests

1149ebb

austin-denoble merged commit b688b60 into 2025-01 Feb 4, 2025
2 checks passed

austin-denoble deleted the adenoble/implement-sparse-indexes branch February 4, 2025 04:07

haruska reviewed Feb 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement sparse index support #95

Implement sparse index support #95

austin-denoble commented Jan 25, 2025 •

edited

Loading

austin-denoble Jan 28, 2025

rohanshah18 Jan 28, 2025

austin-denoble Feb 1, 2025

austin-denoble Jan 28, 2025

jhamon Feb 3, 2025

austin-denoble Jan 28, 2025

austin-denoble Jan 28, 2025

rohanshah18 Jan 28, 2025

rohanshah18 left a comment

haruska left a comment

haruska Feb 4, 2025

Implement sparse index support #95

Implement sparse index support #95

Conversation

austin-denoble commented Jan 25, 2025 • edited Loading

Problem

Solution

Type of Change

Test Plan

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rohanshah18 left a comment

Choose a reason for hiding this comment

haruska left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

austin-denoble commented Jan 25, 2025 •

edited

Loading