Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new Layer "DataCell" to manage data access #40

Closed
LHT129 opened this issue Sep 25, 2024 · 1 comment
Closed

Add a new Layer "DataCell" to manage data access #40

LHT129 opened this issue Sep 25, 2024 · 1 comment
Assignees
Labels
kind/feature New feature or request

Comments

@LHT129
Copy link
Collaborator

LHT129 commented Sep 25, 2024

I think high level index should ignore the specific implement of quantization and io type, So the quantization and io components can be managed by new layer DataCell which provide different data access interface to the index layer. just like the following show

Layer Description Implement
Index Layer ANN algorithm & architecture. Graph (HNSW,NNDescent) + Reorder, IVF(PQ,SQ,fastscan), HNSW_IVF, MultiIndex ......
DataCell Layer data access, use quantization, io and others FlattenDC, GraphNeighborhoodDC, MixtureDC .......
Operator Layer quantizations, IO and other basic components PQ,SQ,MemoryIO,SSDIO,......

this abstract layered approach may facilitate scalability and code reuse
currently we only have index layer, so the issue will provide Datacell Layer and Operator Layer in this feature.
the related pull requests:

@LHT129
Copy link
Collaborator Author

LHT129 commented Nov 6, 2024

Maybe like this

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request
Projects
None yet
Development

No branches or pull requests

5 participants