Feature/index

[net-cscience-raphael]

Adds the index for NNS and Fulltext.

The setup is documented on the Wiki page https://github.com/vitrivr/vitrivr-engine/wiki/Documentation#creating-indexes further the documentation is appended below.

The indexes have to be set up in the schema.json file.
The indexes were created during schema init.

Nearest Neighbor Search NNS

Index hnsw

The hierarchical navigable small world index (HNSW) can be set up for a VECTOR field by adding the following configuration to the field in the schema config.

Parameters:

• attributes allowed max. 1. The value vector is the attribute name in database.
• type: "NNS" describes the query type.
• parameters.type: "hnsw" describes the indextype.
• distance: describes the distance metric for this index. The hnsw index provides:
  • "manhatten"
  • "euclidean"
  • "cosine"
  • "hamming"
  • "jaccard"
• m: the max number of connections per layer (16 by default)
• efConstruction: the size of the dynamic candidate list for constructing the graph (64 by default)
• efSearch: Specify the size of the dynamic candidate list for search (100 by default)

  "indexes": [
    {
      "attributes": [
        "vector"
      ],
      "type": "NNS",
      "parameters": {
        "type": "hnsw",
        "distance": "cosine",
        "m": "4",
        "efConstruction": "10"
        "efSearch": "1000"
      }
    }
  ]

FullText Search

Index gin

The generalized inverted index (GIN) can be set up for a FULLTEXT field by adding the following configuration to the field in the schema config.

Parameters:

• attributes The value value is the attribute name in database. All attributes will be concatenated with delimiters " || ' ' || " to create a document.
• type: "FULLTEXT" describes the query type.
• parameters.type: "gin" describes the index type.
• english: the Language for fulltext index. (Default "english")

  "indexes": [
    {
      "attributes": [
        "value"
      ],
      "type": "FULLTEXT",
      "parameters": {
        "type": "gin",
        "language": "english"
      }
    }
  ]

[net-cscience-raphael]

TODO: Add warning if query exceeds hnsw limit.