Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add kv state cache module. #1777

Merged
merged 20 commits into from
Mar 1, 2024
Merged

Add kv state cache module. #1777

merged 20 commits into from
Mar 1, 2024

Conversation

vegetableysm
Copy link
Collaborator

What do these changes do?

Support store kv cache into vineyard.
Share kv cache between different vineyardd.

vegetableysm and others added 14 commits February 28, 2024 14:03
- support kv state cache on local machine
  - implement cache object / cache block object
  - support api of query and update for cache access
- using radix tree to store token list as key
- seal cache object to vineyard
  - serialize / deserialize radix tree
  - build / resolve cache object

Working on #1708 and #1712.

Signed-off-by: vegetableysm <[email protected]>
Signed-off-by: Ye Cao <[email protected]>
Co-authored-by: Ye Cao <[email protected]>
Co-authored-by: Ye Cao <[email protected]>
-  Add timestamp for node
- Support merge two radix tree
- Support merge cache object
- Refactor the framework of cache object, cache block object and radix
tree
- Fix error of setting sub_tree flag, as well as some derived bugs
caused by the error
- Format the code to an unified style

Signed-off-by: vegetableysm <[email protected]>
- Support merge cache object with radix tree.
- Use cache version to control merge process.

Signed-off-by: vegetableysm <[email protected]>
What do these changes do?
-------------------------

As titled.


Related issue number
--------------------

Fixes #1743

Signed-off-by: Ye Cao <[email protected]>
Copy link
Member

@sighingnow sighingnow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename the directory kv-state-cache to llm-cache.

Move tests/test_state_cache*.cc to modules/llm-cache/tests/.

CMakeLists.txt Outdated Show resolved Hide resolved
modules/kv-state-cache/CMakeLists.txt Outdated Show resolved Hide resolved
modules/kv-state-cache/ds/kv_state_cache.cc Outdated Show resolved Hide resolved
modules/kv-state-cache/ds/kv_state_cache.cc Outdated Show resolved Hide resolved
CMakeLists.txt Outdated Show resolved Hide resolved
modules/kv-state-cache/ds/kv_state_cache_block.h Outdated Show resolved Hide resolved
src/server/services/etcd_meta_service.cc Outdated Show resolved Hide resolved
src/server/services/local_meta_service.h Show resolved Hide resolved
src/server/services/redis_meta_service.h Show resolved Hide resolved
@vegetableysm vegetableysm force-pushed the kv-state-cache branch 3 times, most recently from ed95e24 to 51f8c27 Compare February 28, 2024 07:55
Signed-off-by: vegetableysm <[email protected]>
@sighingnow
Copy link
Member

Ping me when it becomes ready for merging.

@sighingnow sighingnow requested a review from mengke-mk February 29, 2024 13:26
@sighingnow sighingnow merged commit 88d474f into main Mar 1, 2024
6 of 7 checks passed
@sighingnow sighingnow deleted the kv-state-cache branch March 1, 2024 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants