Move all grpc stuff into pinecone.grpc subpackage (pinecone-io#238)

## Problem

In the past this package attempted to import extra grpc deps in the root
`__init__.py` inside a try/catch that caught `ImportError` when deps
were missing. This is error prone because people may be attempting to
use the `pinecone` module alongside other packages that depend on
incompatible versions of these dependencies, so we shouldn't interpret
successful import as a sign that a user has installed the
`pinecone-client[grpc]` extras (with the versions we expect). When these
incompatible versions are present, the `ImportError` doesn't happen but
other runtime errors can occur even when the person is not attempting to
use `GRPCIndex`.

## Solution

The gist of this PR is to better isolate everything GRPC-related into a
subpackage to minimize the changes of a grpc-related dependency clash
getting in the way of using Pinecone. By doing this, issues with grpc
dependencies should never impact people using the non-grpc client.

Summary of changes:
- Moving files
- Contents of `pinecone/data/grpc` moved up and over to `pinecone/grpc`
- Tests moved from `tests/unit/test_index_grpc.py` into
`tests/unit_grpc/test_index_grpc.py`. This is to make it easier to split
grpc and non-grpc test runs.
- Delete remaining cruft in `pinecone/deprecated/legacy_utils.py`
because they were unused and loading grpc deps.
- Making `from pinecone import Pinecone` independent of GRPC
    - References to all GRPC deps removed from `pinecone/__init__.py`
    - Adjusting `pinecone/pinecone.py` to remove `IndexGRPC()` method.
- Setting up for `from pinecone.grpc import Pinecone`
- Add `pinecone/grpc/pinecone.yaml` that extends the other Pinecone
class, modifying only the `Index()` method to use `GRPCIndex`.
- Add `pinecone/grpc/__init__.py` with entries for grpc-flavored
`Pinecone` and `IndexGRPC` exports.
- CI updates
- `pyproject.toml` adjusted to properly indicate grpc deps as `optional
= true`; I discovered in testing that grpc dependencies were always
being installed in CI.
- Add an input var to the `setup-poetry` action so that we can toggle on
grpc dependencies when needed.
- `.github/workflows/testing.yaml` refactored to break GRPC and non-GRPC
into separate jobs, since they have different install requirements.

#### Usage evolution

##### Legacy implementation

```python
import pinecone 
pinecone.init(api_key='foo', environment='bar')

index = pinecone.Index('index-name')
index_grpc = pinecone.GRPCIndex('index-name')
```

##### Unreleased spruce branch (before this PR)

This was an improvement over the legacy approach because we no longer
relied on global state for configuration. Instead, we wrap configuration
state into a class instance. However, the `Pinecone` class had a
dependency on GRPC stuff to implement the `GRPCIndex()` method. The
imports of these extra deps was intended to noop when they were not
installed, but as I described above there are situations where you could
still have runtime errors if other dependencies in your notebook / app
have installed incompatible GRPC dependencies that prevent you from
using the REST version of the client.

```python
from pinecone import Pinecone

p = Pinecone(api_key='foo')
index = p.Index('index-name')
index_grpc = p.GRPCIndex('index-name')
```

##### After this PR, import from different subpackage when GRPC desired

Now the usage is exactly the same for REST vs GRPC except for the import
coming from `pinecone.grpc` instead of `pinecone` when you want data
operations to use grpc. GRPC deps are not loaded at all unless you
specifically import from `pinecone.grpc`.

To use REST, import from `pinecone`

```python
from pinecone import Pinecone

p = Pinecone(api_key='foo')
index = p.Index('index-name')
```

To use GRPC, import from `pinecone.grpc` subpackage

```python
from pinecone.grpc import Pinecone

p = Pinecone(api_key='foo')
index = p.Index('index-name')
```

If you ever wanted to use these side by side for some reason (no real
reason to ever do this unless you are writing tests for this package)
you would have to alias one of the imports to avoid a name collision.

```python
from pinecone import Pinecone
from pinecone.grpc import Pinecone as PineconeGRPC

...
```

## Type of Change

- [x] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [x] This change requires a documentation update
- [x] Infrastructure change (CI configs, etc)

## Test Plan

Describe specific steps for validating this change.

Author: jhamon

.github/actions/build-docs/action.yml

@@ -15,6 +15,8 @@ runs:
 
     - name: Setup Poetry
       uses: ./.github/actions/setup-poetry
+      with:
+        include_grpc: 'true'
 
     - name: Build html documentation
       shell: bash

.github/actions/setup-poetry/action.yml

@@ -1,12 +1,18 @@
 name: 'Setup Poetry'
 description: 'Installs Poetry and dependencies'
+inputs:
+  include_grpc:
+    description: 'Install gRPC dependencies'
+    required: true
+    default: 'false'
 runs:
   using: 'composite'
   steps:
     - name: Install Poetry
       uses: snok/install-poetry@v1
 
     - name: Install dependencies
+      if: ${{ inputs.include_grpc == 'false' }}
       shell: bash
       run: |
         poetry install
@@ -16,6 +22,7 @@ runs:
     # The dependencies that are specific to gRPC are defined in pyproject.toml
     # under tool.poetry.extras
     - name: Install gRPC dependencies
+      if: ${{ inputs.include_grpc == 'true' }}
       shell: bash
       run: |
         poetry install --extras "grpc"

.github/workflows/testing.yaml

@@ -4,8 +4,8 @@ on:
   workflow_call: {}
 
 jobs:
-  run-tests:
-    name: Run tests
+  units-no-grpc:
+    name: Run tests (REST)
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -22,11 +22,49 @@ jobs:
       - name: Setup Poetry
         uses: ./.github/actions/setup-poetry
 
-      - name: Run tests
-        run: make tests
+      - name: Run unit tests
+        run: poetry run pytest --cov=pinecone --timeout=120 tests/unit
 
-      - name: Build Python client
-        run: make package
+  units-grpc:
+    name: Run tests (GRPC)
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.8, 3.9, '3.10', 3.11]
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Setup Poetry
+        uses: ./.github/actions/setup-poetry
+        with:
+          include_grpc: true
+
+      - name: Run unit tests (GRPC)
+        run: poetry run pytest --cov=pinecone --timeout=120 tests/unit_grpc
+
+  package:
+    name: Check packaging
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.8, 3.9, '3.10', 3.11]
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Setup Poetry
+        uses: ./.github/actions/setup-poetry
 
-      - name: Build docs
-        run: make docs
+      - name: Package
+        run: poetry build

Makefile

@@ -9,12 +9,14 @@ image:
 develop:
 	poetry install -E grpc
 
-tests:
-	@echo "Installing dependencies..."
-	poetry install
+test-unit:
 	@echo "Running tests..."
 	poetry run pytest --cov=pinecone --timeout=120 tests/unit
 
+test-grpc-unit:
+	@echo "Running tests..."
+	poetry run pytest --cov=pinecone --timeout=120 tests/unit_grpc
+
 version:
 	poetry version
 

pinecone/__init__.py

@@ -4,9 +4,4 @@
 from .config import *
 from .exceptions import *
 from .control.pinecone import Pinecone
-from .data.index import *
-
-try:
-    from .data.grpc.index_grpc import *
-except ImportError:
-    pass  # ignore for non-[grpc] installations
+from .data.index import *

pinecone/control/pinecone.py

@@ -14,8 +14,6 @@
 from pinecone.utils import get_user_agent
 
 from pinecone.data import Index
-from pinecone.data import GRPCIndex
-
 
 class Pinecone:
     def __init__(
@@ -270,6 +268,3 @@ def _get_status(self, name: str):
     def Index(self, name: str):
         index_host = self.index_host_store.get_host(self.index_api, self.config, name)
         return Index(api_key=self.config.API_KEY, host=index_host)
-
-    def GRPCIndex(self, name: str):
-        return GRPCIndex(self.config, name)

pinecone/data/__init__.py

@@ -1,2 +1 @@
-from .index import Index
-from .grpc.index_grpc import GRPCIndex
+from .index import Index

pinecone/data/grpc/__init__.py

@@ -0,0 +1,2 @@
+from .index_grpc import GRPCIndex
+from .pinecone import Pinecone

pinecone/deprecated/__init__.py

@@ -1,7 +1,7 @@
 import logging
 from abc import ABC, abstractmethod
 from functools import wraps
-from typing import NamedTuple, Optional, Dict
+from typing import Dict
 
 import certifi
 import grpc

pinecone/deprecated/legacy_utils.py

@@ -34,8 +34,8 @@
     REQUIRED_VECTOR_FIELDS,
     OPTIONAL_VECTOR_FIELDS,
 )
-from pinecone.data.grpc.base import GRPCIndexBase
-from pinecone.data.grpc.future import PineconeGrpcFuture
+from .base import GRPCIndexBase
+from .future import PineconeGrpcFuture
 
 __all__ = ["GRPCIndex", "GRPCVector", "GRPCQueryVector", "GRPCSparseValues"]
 

pinecone/grpc/__init__.py

@@ -0,0 +1,7 @@
+from ..control.pinecone import Pinecone
+from .index_grpc import GRPCIndex
+
+class Pinecone(Pinecone):
+    def Index(self, name: str):
+        index_host = self.index_host_store.get_host(self.index_api, self.config, name)
+        return GRPCIndex(api_key=self.config.API_KEY, host=index_host)

pinecone/data/grpc/base.py renamed to pinecone/grpc/base.py

@@ -63,11 +63,11 @@ dnspython = ">=2.0.0"
 python-dateutil = ">=2.5.3"
 urllib3 = "1.25.3"
 tqdm = ">=4.64.1"
-grpcio = ">=1.44.0"
-grpc-gateway-protoc-gen-openapiv2 = "0.1.0"
-googleapis-common-protos = ">=1.53.0"
-lz4 = ">=3.1.3"
-protobuf = "~=3.20.0"
+grpcio = { version = ">=1.44.0", optional = true }
+grpc-gateway-protoc-gen-openapiv2 = { version = "0.1.0", optional = true }
+googleapis-common-protos = { version = ">=1.53.0", optional = true }
+lz4 = { version = ">=3.1.3", optional = true }
+protobuf = { version = "~=3.20.0", optional = true }
 
 [tool.poetry.group.dev.dependencies]
 numpy = ">=1.22"

pinecone/data/grpc/config.py renamed to pinecone/grpc/config.py

@@ -4,7 +4,8 @@
 import pandas as pd
 import pytest
 
-from pinecone import Config, GRPCIndex
+from pinecone import Config
+from pinecone.grpc import GRPCIndex
 from pinecone import DescribeIndexStatsRequest
 from pinecone.core.grpc.protos.vector_service_pb2 import (
     Vector,
@@ -18,7 +19,7 @@
     UpsertResponse,
     SparseValues,
 )
-from pinecone.data.grpc.utils import dict_to_proto_struct
+from pinecone.grpc.utils import dict_to_proto_struct
 
 class TestGrpcIndex:
     def setup_method(self):