Faiss Indexing unable to modify the number of centroids

[vrdn-23]

Hi,
I've been tinkering with the Dense Indexer and was trying to index a set of documents using Faiss but I'm unable to change the number of centroids.

This is the command I'm running and as I understand it, setting the argument should allow me to control the number of centroids that we index to.

(my_agent_assist) ➜  my-own-agent-assist git:(main) ✗ python -m pyserini.index.faiss --input embeddings --output faiss --hnsw --pq --efC=128 --M=128
(654, 768)
WARNING clustering 654 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 654 points to 256 centroids: please provide at least 9984 training points
WARNING clustering 654 points to 256 centroids: please provide at least 9984 training points
........
WARNING clustering 654 points to 256 centroids: please provide at least 9984 training points
hnsw_add_vertices: adding 654 elements on top of 0 (preset_levels=0)
  max_level = 1
Adding 5 elements at level 1
Adding 649 elements at level 0
Done in 193.076 ms
654

Could someone please confirm whether this is a bug or whether I may not be handling a setting correctly?

[MXueguang]

Hi @vrdn-23,

You can modify it by the --pq-nbits option, by default it is 8, which gives 2^8 = 256 centroids.

pyserini/pyserini/index/faiss.py

Line 72 in 11124e9

index = faiss.IndexPQ(args.dim, args.pq_m, args.pq_nbits, faiss.METRIC_INNER_PRODUCT)

[vicpara]

If I'm using HSNW and PQ it seems that changing the number of centroids is impossible at the time. Is there any reason for it?

pyserini/pyserini/index/faiss.py

Line 64 in 1f6ba77

if args.hnsw and args.pq:

[lintool]

No reason, just haven't connected the Faiss settings to cmdline params. PRs welcome!