Code

Non-uniform vector quantization#374

Code

Open

marianotepper wants to merge 174 commits intomainjbellis/jvector:mainfrom nuveqjbellis/jvector:nuveqCopy head branch name to clipboard

+3,479-149

Commits

Commits on Oct 24, 2024

• Adding initial files for Non-uniform Vector Quantization

  marianotepper

  committed
  20642a5

Commits on Oct 28, 2024

• Adding methods to get the number of rows and columns.

  marianotepper

  committed
  4166400

• Adding method to compute the square of a double number

  marianotepper

  committed
  28764d0

• Implementation of the Exponential Natural Evolution Strategies optimizer.

  marianotepper

  committed
  5e9ecbb

• Minor code cleanup

  marianotepper

  committed
  c83bd2d

• Revert "Adding methods to get the number of rows and columns."

  Show description for eabfad3
  marianotepper

  committed
  eabfad3

• Minor code cleanup

  marianotepper

  committed
  1c0608b

• Making the constructor of NESOptimizer public. Now optimize returns an OptimizationResult that facilitates error checking.

  marianotepper

  committed
  aa6dfdd

• Adding tests.

  marianotepper

  committed
  b16b0d6

• Remove unused import

  marianotepper

  committed
  751634a

• Remove unused variable

  marianotepper

  committed
  3a80c1f

• Increase default number of samples in NESOptimizer

  marianotepper

  committed
  8ccf14f

• Add a few more tests for NESOptimizer

  marianotepper

  committed
  1141303

• Add a few more tests for NESOptimizer

  marianotepper

  committed
  a421f1d

• Add javadocs

  marianotepper

  committed
  98c29c5

• Add one test for the case without box constraints

  marianotepper

  committed
  5f986b6

• Put access modifier first

  marianotepper

  committed
  69c745e

• Put access modifier first

  marianotepper

  committed
  a1f94bc

• Improve javadocs

  marianotepper

  committed
  81445e4

• Remove blank line

  marianotepper

  committed
  cab64ba

• Initial test for the non-uniform quantizer

  marianotepper

  committed
  114d60c

Commits on Oct 29, 2024

• Add an overload for the subInPlace vector function, that subtracts a constant to each element.

  marianotepper

  committed
  836f289

• Completed TestNonUniformQuantization.testGaussian with a passing criteria and averaging over 10 trials.

  marianotepper

  committed
  b3811f5

• Add vectorized operations pow, constantMinusExponentiatedVector, and exponentiateConstantMinusVector

  marianotepper

  committed
  c47d2b3

• Add vectorized operations to TestNonUniformQuantization and increase nTrials to 50

  marianotepper

  committed
  93ef1d3

• Add missing blank space

  marianotepper

  committed
  e4e82c5

• Use VectorUtil.squareL2Distance for computing the loss

  marianotepper

  committed
  4250711

• Remove unused import

  marianotepper

  committed
  fa295a6

• Revert "Remove unused import"

  Show description for 8be166b
  marianotepper

  committed
  8be166b

• Remove unused import

  marianotepper

  committed
  a096142

Commits on Oct 31, 2024

• Commit with a stub of the functionality in place. Still non-functional

  marianotepper

  committed
  09b6a93

Commits on Nov 1, 2024

• Commit with a more complete stub of the functionality in place. Still non-functional

  marianotepper

  committed
  91abcae

• Commit vectorized nvqDotProduct.

  marianotepper

  committed
  7a59b2e

Commits on Nov 4, 2024

• Implemented getSubVectors for NVQ. Still non-functional

  tlwillke

  committed
  e3b1141

• Fixed getSubVectors

  tlwillke

  committed
  88ee4f9

• Merge remote-tracking branch 'origin/nuveq' into nuveq

  marianotepper

  committed
  da86dc0

• Commit vectorized nvqCosine.

  marianotepper

  committed
  2b49c2b

• Delete wrong normalization

  marianotepper

  committed
  c022a5d

• Fix access modifier of the fields in QuantizedSubVector

  marianotepper

  committed
  0fa7259

• Remove unused import

  marianotepper

  committed
  a902916

• Implementation of nvqDotProduct and nvqCosine in NativeVectorUtilSupport

  marianotepper

  committed
  46d8837

• Remove unused import

  marianotepper

  committed
  04f828a

• Remove unnecessary semicolons

  marianotepper

  committed
  a598ed3

• Replace switches with enhanced switches

  marianotepper

  committed
  3d52e74

• Add stub implementations of nvqShuffleQueryInPlace

  marianotepper

  committed
  2d6f27e

• Updates to some NVQ comments

  tlwillke

  committed
  898ff3f

• Merge branch 'nuveq' of https://github.com/jbellis/jvector into nuveq

  Show description for 599b6f9
  tlwillke

  committed
  599b6f9

Commits on Nov 5, 2024

• Add implementations of nvqShuffleQueryInPlace

  marianotepper

  committed
  6f48de0

• First implementation of NVQ Euclidean distance calcs. Includes Default and Panama implementations. Native implementation is a WIP

  tlwillke

  committed
  f6e1943

• Merge branch 'nuveq' of https://github.com/jbellis/jvector into nuveq

  Show description for 62c441e
  tlwillke

  committed
  62c441e

• Add query shuffling similarity computation in NVQVectors

  marianotepper

  committed
  fa0cc17

• Replace traditional for loops with enhanced loops

  marianotepper

  committed
  7b7fe8d

• Completed first native and panama implementations for NVQ square L2 distance computation. Removed the redundant Euclidean computations. Updated the NVQ scoring functions to better match those in PQ…

  Show description for 684a47a
  tlwillke

  committed
  684a47a

• Merge branch 'nuveq' of https://github.com/jbellis/jvector into nuveq

  Show description for 18436aa
  tlwillke

  committed
  18436aa

Commits on Nov 6, 2024

• Fix a bunch of minor errors

  marianotepper

  committed
  30c2d66

• First test for NVQuantization and NVQVectors

  marianotepper

  committed
  9f3485d

• Merge remote-tracking branch 'origin/nuveq' into nuveq

  marianotepper

  committed
  7b43382

• Fix bug when dealing with an odd number of dimensions. Expose nvqDequantizeUnnormalized in VectorUtil

  marianotepper

  committed
  7ac54fe

Commits on Nov 7, 2024

• Add TestUtil.normalRandomVector and TestUtil.createNormalRandomVectors

  marianotepper

  committed
  1dbd769

• Add VectorUtil.sub to subtract a float from a vector not in-place

  marianotepper

  committed
  dc0f333

• WIP test for distances in NVQuantization

  marianotepper

  committed
  3cb1b5d

Commits on Nov 8, 2024

• Test for distances in NVQuantization completed. Passes for non-vectorized similarity functions

  marianotepper

  committed
  b0b0a42

• Apply shuffling to the centroid as well for cosine similarity

  marianotepper

  committed
  95f793d

• Fix vectorized dequantization for 4-bit NVQ shuffling

  marianotepper

  committed
  2387cd3

• Major NVQ refactoring mostly completed.

  marianotepper

  committed
  03b57e4

Commits on Nov 13, 2024

• Fix abstraction-breaking functions in VectorUtil. NESOptimizer now uses floats, uses relative error, and allows to specify a maximum number of iterations. Other performance fixes in NVQuantization …

  Show description for 22a109a
  marianotepper

  committed
  22a109a

• Remove commented code

  marianotepper

  committed
  6bd8e59

Commits on Nov 15, 2024

• Lower the number of samples in NESOptimizer. A few other code improvements and cleanups

  marianotepper

  committed
  411e9af

Commits on Nov 17, 2024

• Added NVQ Feature ID. Initial implementation of NVQ for graph indexing. See NVQ and NVQPackedVectors. Separated scorer into NVQScorer.

  tlwillke

  committed
  21076b8

• Merging changes for NVQ graph indexing.

  tlwillke

  committed
  d5dd527

Commits on Nov 18, 2024

• Implement vectorized quantization. SIMD vectorized instructions cleanup. Add documentation

  marianotepper

  committed
  a2ca4ff

• Merge remote-tracking branch 'origin/nuveq' into nuveq

  Show description for 3e1fcdd
  marianotepper

  committed
  3e1fcdd

• Fix order of imports to prevent coupling between graph/disk and pq

  marianotepper

  committed
  57128ba

• Remove commented code

  marianotepper

  committed
  fc8fe14

• Remove redundant interface

  marianotepper

  committed
  879b027

• Remove blank line

  marianotepper

  committed
  7a9643a

• Put a maximum number of attempts in NES solver, just in case.

  marianotepper

  committed
  590d03d

• Reinstate commented code for further investigation

  marianotepper

  committed
  07fb340

Commits on Nov 19, 2024

• Updates to get NVQ working with Bench and for reranking. Changes to Bench and Grid are WIP.

  tlwillke

  committed
  9fe93c4

• Wire in NVQ feature-driven reranker in ODGI. Fix serialization sizing for NVQ header + inline vectors.

  jkni

  committed
  ba8df54

Commits on Nov 22, 2024

• Reduce allocations for NVQ.

  marianotepper

  committed
  c50c982

• Fix bug in computation of the mean

  marianotepper

  committed
  e0098fc

• Add missing parenthesis in comment

  marianotepper

  committed
  4266b66

Commits on Nov 25, 2024

• Eliminate unneeded memory allocations and improve performance. Add an escape hatch for NESOptimizer where the LosFunction can deem that it has reached a good enough value

  marianotepper

  committed
  f56c899

• Fix type in comment

  marianotepper

  committed
  30adcdf

• Removed temporary dataset truncation for Bench.

  tlwillke

  committed
  6f8addc

Commits on Nov 26, 2024

• Code cleanup. Implement exp/log exponentiationtrick

  marianotepper

  committed
  e2a9704

• Merge remote-tracking branch 'origin/nuveq' into nuveq

  marianotepper

  committed
  09e25af

• Minor code improvements in similarity computations for the default engine

  marianotepper

  committed
  85be405

• Eliminate memory allocations and interactions in similarity computations for the Panama engine

  marianotepper

  committed
  c545926

• Replace manual implementation of the inverse Kumaraswamy with a call to the inverseKumaraswamy function

  marianotepper

  committed
  b667a5b

• Quick experiments with approximations to the Kumaraswamy inverse CDF

  marianotepper

  committed
  64ccdf0

• Accelerated implementation of natural log and exp that lead to accelerated computation of the Kumaraswamy inverse CDF

  marianotepper

  committed
  52dde49

• Accelerated computation of the Kumaraswamy CDF

  marianotepper

  committed
  2cea709

• Vectorized acceleration of forwardKumaraswamy and inverseKumaraswamy using fastLog and fastExp

  marianotepper

  committed
  5860bdb

Commits on Nov 27, 2024

• Minor update in the signature of SIMDOps.fastLog

  marianotepper

  committed
  ccb9d92

• Small bench for assessing speed of distance calculations with NVQ

  marianotepper

  committed
  647dc3f

• Minor fixes to bench for assessing speed of distance calculations with NVQ

  marianotepper

  committed
  db765f2

• Create an additional bench for SIMD investigations of Kumaraswamy

  marianotepper

  committed
  cb6c360

• Moving kumaraswamyApproximationScalar and kumaraswamyApproximationSIMD to twenty

  marianotepper

  committed
  6a3b0a7

Commits on Nov 28, 2024

• Make import list in distancesNVQ tighter

  marianotepper

  committed
  d5c2a82

• Measure number of scalar instead of vector operations

  marianotepper

  committed
  7ef54d2

• Refactor kumaraswamy benchmarks into their own package

  marianotepper

  committed
  8e6e47e

• Remove unneeded parameter from forwardKumaraswamy

  marianotepper

  committed
  244e354

• Remove dummy variable used for debugging

  marianotepper

  committed
  7541448

Commits on Dec 1, 2024

• Temp fixes for NVQ species and inverseKumaraswamy.

  tlwillke

  committed
  b8d5f03

• Merge changes to species selection and inverseKumaraswamy.

  tlwillke

  committed
  55f1f62

• Set species for ByteVector based on FloatVector.SPECIES_PREFERRED.

  tlwillke

  committed
  780c7a7

• Updates in kumaraswamyApproximationScalar to make the experiment more representative and in kumaraswamyApproximationSIMD to make it faster

  marianotepper

  committed
  ba9d50a

• Minor updates in kumaraswamyApproximationSIMD

  marianotepper

  committed
  c11ded7

• Minor refactor the version of nvqDequantize8bit that returns a vector

  marianotepper

  committed
  0fcce64

• Refactoring that allows the code to run on any platform. Implementation of FastLog that uses broadcasting

  marianotepper

  committed
  cae0da5

Commits on Dec 2, 2024

• Refactoring of the NVQ SIMD code to always use SPECIES_PREFERRED. Now exp/log trick is used for the forward and inverse Kumaraswamy functions by default

  marianotepper

  committed
  bbac862

Commits on Dec 12, 2024

• New code with that does the query shuffling for 8-bit NVQ. Optimization using a discrete set of values instead of NES

  marianotepper

  committed
  447ab3d

Commits on Dec 13, 2024

• Working implementation with logistic optimization

  marianotepper

  committed
  cbda16a

• Revert changes in distancesNVQ

  marianotepper

  committed
  7d9936b

• Reduced the number of calls to logistic_function for each distance computation.

  marianotepper

  committed
  a346f66

• Added NVQ logistic and logit methods. Renamed Replaced parameters a, b in calls to alpha, X0.

  tlwillke

  committed
  dfbd475

• Merging changes for NVQ logistic methods.

  tlwillke

  committed
  be85a72

• Add FMAs where possible in default backend.

  marianotepper

  committed
  a05c942

• Add FMAs where possible in MathUtil::fastExp and improve its comment.

  marianotepper

  committed
  6a0d926

• Change function parameter names in PanamaVectorUtilSupport to the new scheme

  marianotepper

  committed
  6907e62

• Add better fastExp in SimdOps

  marianotepper

  committed
  2e39599

Commits on Dec 15, 2024

• Added scaling and bias for logistic functions. Changes impact quantization and dequantization.

  tlwillke

  committed
  721e44a

Commits on Dec 16, 2024

• Add dummy accumulator in distancesNVQ

  marianotepper

  committed
  3c95d10

• Merge remote-tracking branch 'origin/nuveq' into nuveq

  marianotepper

  committed
  84d233f

• Fuse division by 255 and logistic scaling

  marianotepper

  committed
  0d7fe86

• Replace a couple of .mul.add patterns by .fma

  marianotepper

  committed
  c53317a

• Fuse division by 255 and logistic scaling

  marianotepper

  committed
  58b3030

Commits on Dec 17, 2024

• Remove extra scaling step not needed anymore with the logistic nonlinearity

  marianotepper

  committed
  5b92789

• Remove extra scaling step not needed anymore with the logistic nonlinearity

  marianotepper

  committed
  78fa8ef

• Remove unused functions from VectorUtil

  marianotepper

  committed
  dd07de3

• Remove old test

  marianotepper

  committed
  edbc76d

• Set learning to true in SiftSmall

  marianotepper

  committed
  b34345c

• Set number of subsegments to 2 in SiftSmall

  marianotepper

  committed
  184198a

• Fix vectorizedLength in nvqCosine8bit. Fix tail computation in all three distances

  marianotepper

  committed
  b133a69

• Set default number fo subvectors to 2 in distancesNVQNVQ

  marianotepper

  committed
  1f1676a

• Cleanup documentation and comments

  marianotepper

  committed
  cba3336

• Cleanup documentation and comments

  marianotepper

  committed
  8512ce4

• Remove unnecessary code

  marianotepper

  committed
  d38c591

• Remove unnecessary imports

  marianotepper

  committed
  16497d4

• Remove unnecessary variable

  marianotepper

  committed
  2238c3a

• Remove unnecessary pow function from VectorUtil

  marianotepper

  committed
  966d5f3

• Fix hyperlinks in documentation

  marianotepper

  committed
  9bda7cd

• Remove unused variables and old documentation

  marianotepper

  committed
  7495341

• Do the optimization using NonuniformQuantizationLossFunction. Fix comments and other minor fixes

  marianotepper

  committed
  099880f

• Remove NESSolver calls from NVQuantization

  marianotepper

  committed
  c551431

• Add versioning code back to NVQuantization

  marianotepper

  committed
  70b88ce

• Remove 4-bit NVQ code

  marianotepper

  committed
  852e770

• Remove Kumaraswamy code from SimdOps

  marianotepper

  committed
  79f8469

• Remove unused static variable from SimdOps

  marianotepper

  committed
  8188d1e

Commits on Dec 18, 2024

• Add new NVQ code to native backend

  marianotepper

  committed
  832173a

• Remove comments

  marianotepper

  committed
  afbf274

• Rename directory "pq" as "quantization"

  marianotepper

  committed
  5b3c16a

• Do not expose the number of bits in the API of NVQuantization

  marianotepper

  committed
  bcc5c0a

• Remove unused variables and apply fix to hash function in NVQVectors

  marianotepper

  committed
  9cb9d8e

• Remove NVQuantization.BitsPerDimension from CompressorParameters

  marianotepper

  committed
  2287b24

• Adjust expected results in testSaveLoadNVQ to accommodate the new formulation

  marianotepper

  committed
  512c899

• Remove bench files for experimenting with the Kumaraswamy approximation

  marianotepper

  committed
  062834b

• Cleaning up distancesNVQ

  marianotepper

  committed
  5e9694a

• Cleaning up distancesNVQ

  marianotepper

  committed
  f814efb

• Cleaned up a few NVQ comments.

  tlwillke

  committed
  3d07f98

• Merge remote-tracking branch 'origin/nuveq' into nuveq

  Show description for 53ecb05
  marianotepper

  committed
  53ecb05

• Cleaning up distancesNVQ

  marianotepper

  committed
  38e54c2

• Remove unused import

  marianotepper

  committed
  db927c4

• Remove unused import

  marianotepper

  committed
  64d10e8

• Restore the FUSED_ADC feature in Bench

  marianotepper

  committed
  9aa25f1

• Merge remote-tracking branch 'origin/main' into nuveq

  Show description for ed3d6ec
  marianotepper

  committed
  ed3d6ec

Commits on Dec 19, 2024

• Merge changes from main

  marianotepper

  committed
  5599773

• Cosmetic changes to distancesNVQ

  marianotepper

  committed
  46bd177

• Conform to the new interface that uses encodeTo

  marianotepper

  committed
  d9bc9b0

• Remove unused code to declutter. The dequantization path wsa useful for debugging but is not truly exercized now. replace a couple of FMA patterns in the tail computations.

  marianotepper

  committed
  aedda1a

Commits on Dec 20, 2024

• Replaced the logistic/logit apir with their NQT variants

  marianotepper

  committed
  649f59e

• Replaced one occurrence of snake case

  marianotepper

  committed
  ab128f8