Faiss 1.5.3

Bugfixes:

• slow scanning of inverted lists (#836).

Features:

• add basic support for 6 new metrics in CPU IndexFlat and IndexHNSW (#848);
• add support for IndexIDMap/IndexIDMap2 with binary indexes (#780).

Misc:

• throw python exception for OOM (#758);
• make DistanceComputer available for all random access indexes;
• gradually moving from long to int64_t for portability.

Faiss 1.5.2

The license was changed from BSD+Patents to MIT.

Changelog:

• propagates exceptions raised in sub-indexes of IndexShards and IndexReplicas;
• support for searching several inverted lists in parallel (parallel_mode != 0);
• better support for PQ codes where nbit != 8 or 16;
• IVFSpectralHash implementation: spectral hash codes inside an IVF;
• 6-bit per component scalar quantizer (4 and 8 bit were already supported);
• combinations of inverted lists: HStackInvertedLists and VStackInvertedLists;
• configurable number of threads for OnDiskInvertedLists prefetching (including 0=no prefetch);
• more test and demo code compatible with Python 3 (print with parentheses);
• refactored benchmark code: data loading is now in a single file.

Faiss 1.5.1

Changelog:

• a MatrixStats object, which reports useful statistics about a dataset;
• an option to round coordinates during k-means optimization;
• an alternative option for search in HNSW;
• moved stats() and imbalance_factor() from IndexIVF to InvertedLists object;
• range search is now available for IVFScalarQuantizer;
• support for direct uint_8 codec in ScalarQuantizer;
• renamed IndexProxy to IndexReplicas (now ;
• better support for PQ code assignment with external index;
• support for IMI2x16 (4B virtual centroids!);
• support for k = 2048 search on GPU (instead of 1024);
• most CUDA mem alloc failures now throw exceptions instead of terminating on an assertion;
• support for renaming an ondisk invertedlists;
• interrupt computations with interrupt signal (ctrl-C) in python;
• simplified build system (with --with-cuda/--with-cuda-arch options);
• updated example Dockerfile;
• conda packages now depend on the cudatoolkit packages, which fixes some interferences with pytorch. Consequentially, faiss-gpu should now be installed by conda install -c pytorch faiss-gpu cudatoolkit=10.0.

Faiss 1.5.0

Changelog:

• GpuIndexBinaryFlat
• IndexBinaryHNSW

v1.4.0

Faiss 1.4.0

Features:

• automatic tracking of C++ references in Python
• non-intel platforms supported -- some functions optimized for ARM
• override nprobe for concurrent searches
• support for floating-point quantizers in binary indexes

Bug fixes:

• no more segfaults in python (I know it's the same as the first feature but it's important!)
• fix GpuIndexIVFFlat issues for float32 with 64 / 128 dims
• fix sharding of flat indexes on GPU with index_cpu_to_gpu_multiple

---

The Python interface of Faiss closely mimics the C++ interface. This means that all C++ functions, objects, fields and methods are visible and accessible in Python. This is done thanks to SWIG, that automatically generates Python classes from the C++ headers. The downside is that this low-level access means that there is no automatic tracking of C++ references in Python. For example:

index = IndexIVFFlat(IndexFlatL2(10), 10, 100) 

would crash. Python does not know that the IndexFlatL2 is referenced by the IndexIVFFlat, so the garbage collector deallocates the IndexFlatL2 while IndexIVFFlat still references it. In Faiss 1.4.0, we added code to all such constructors that adds a Python-level reference to the object and prevents deallocation. With this upgrade, there should be no crashes in pure Python any more, you can report them right away as issues.

Faiss was developed on 64-bit x86 platforms, Linux and Mac OS. There were quite a few locations in the code that shamelessly assumed that they were compiled with SSE support. Faiss 1.4.0 is portable to other hardware, it has pure C++ code for all operations, and SSE/AVX is only enabled if the appropriate macro are set. This was tested on an ARM platform and also a few operations were optimized for the ARM SIMD operations (in utils_simd.cpp).

To compile on a non-x86 platform, you will need to provide a BLAS library (OpenBLAS works for aarch64) and remove x86-specific flags from the makefile.inc (manually for now). Faiss is not portable to other compilers than g++/clang though.

The search-time parameters like nprobe for IndexIVF are set in the index object. What if you want to perform concurrent searches from several threads with different search parameters? This was not possible so far. Now there is an IVFSearchParameters object that can override the parameters set at the object level. See tests/test_params_override.cpp

Faiss' support for binary indexes is recent, and not so many index types are supported. To work around this, we added IndexBinaryFromFloat, a binary index that wraps around any floating-point index. This makes it possible, for example, to use an IndexHNSW as a quantizer for an IndexBinaryIVF. See tests/test_index_binary_from_float.py

We also fixed a few bugs that correspond to github issues.

v1.3.0

Features:

• Support for binary indexes (IndexBinaryFlat, IndexBinaryIVF)
• Support fp16 encoding in scalar quantizer
• Support for deduplication in IndexIVFFlat
• Support for index serialization

Bugs:

• Fix MMAP bug for normal indexes
• Fix propagation of io_flags in read func
• Fix k-selection for CUDA 9
• Fix race condition in OnDiskInvertedLists

v1.2.1

Features

• Support for on-disk storage of IndexIVF data.

• C API

• extended tutorial to GPU indexes