fixes fizz commit for now #104
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Run tests on CI Run long tests (navy/bench) every day on CI Run CI on prebuild docker image Run only centos build on CI Update docker file used in CI Centos8 is EOL Disable failing clang-format-check Add extra param to build-package.sh Add scripts for rebuilding/pushing docker images Taken from: pmem/dev-utils-kit@30794c3 Extend CI to rebuild docker automatically Update build-cachelib-docker.yml Do not use shallow clone to make sure Docker rebuild logic works correctly. Added required packages to install Intel ittapi Update CI to use intel/CacheLib repo (pmem#17) Add multi-tier navy benchmark and run it on CI - fix navy multi-tier config for NUMA bindings added code coverage support in CacheLib Adding libdml to CentOS docker image (pmem#53) only exclude allocator-test-NavySetupTestm, shm-test-test_page_size tests added perf and numactl to docker packages --------------------------------------------- one large commit for all CI and code coverage see above for the change history.
to utilize combined locking.
This includes printing: - allocSize - allocated memory size - memory usage fraction
Part 1. ----------------------------------------- This includes the following: - Multi-tier allocator with TierId - allocateInternalTier - creating multi-tier allocator on shared memory Other patches can be combined/merged with this patch (such as multi-tier serialization support and improvements to eviction). We will name those compatible with Part 1 in later patches.
------------------------- There are two parts to this commit and we can split them up. Part 1) This commit contains the additional memory tiers tests for different pool sizes. We also use getPoolSize(pid), to get total size from all pools across allocators. Part 2) This part can be merged with the initial multi-tier part 1. It fixes the tiering sizes (pulls changes from what was issue75 rebased commit that did not make it into upstream commits).
for the compressed ptr changes that were introduced upstream. - Includes later cosmetic changes added by sounak 9cb5c29
Part 1. (single tier) ----------------------------- added per pool class rolling average latency (upstream PR version) fix for rolling stats (on multi-tier to be followed by multi-tier rolling stats implementation in the following commit) it should be noted - an attempt was made to use average alloc latency as a guide to control background mover batch size. While average alloc latency decreased, so did throughput because batch size became too big and put contention on locks.
Part 2. (multi-tier support) -------------------------------------- There is also an introduction to kMaxTiers in Cache.h - this should probably be split from this commit. added per tier pool class rolling average latency (based on upstream PR)
----------------------- Hot queue iterator for 2Q. Will start at Hot queue and move to Warm queue if hot queue is exhausted. Useful for promotion semantics if using 2Q replacement. rebased on to develop and added some tests.
Part 2. ---------------------------- This patch introduces tryEvictToNextMemoryTier and some additional multi-tier tests. We can consider merging tryEvictToNextMemoryTier with the initial implementation and seperating the tests into a seperate patch. Per tier pool stats (multi-tier patch part 3.) -------------------- This introduces per tier stats this can go with multi-tier patch part 2. Fix token creation and stats (pmem#79) (multi-tier patch 4.) --------------------------------- This patch can go after we implement tryEvictToNextMemoryTier (or multi-tier part 2) and should be combined as such. * Fix issue with token creation * Do not increment evictFail* stats if evictFailConcurrentFill were incremented correct handling for expired items in eviction (pmem#86) (multi-tier patch 5.) ----------------------------------------------------- This can be merged with patches that fix token creation and probably squashed into multi-tier patch 2. - we first check if an item is expired under mmContainer lock and if so mark it for eviction so it is recycled back up to allocateInternalTier.
-updated configs for numa bindings
Part 1. -------------------------------------- This adds the following: 1. tryPromoteToNextTier. This could go with multi-tier part 2 2. Promotion iterators. This could go with MM2Q promotion iterators patch. It also enables background workers in the cache config. Future changes to the background workers can be merged with this patch. Background evictors multi-tier Part 2. -------------------------------- This should be rolled into background evictors part 1. improved bg stats structure and cachebench output adds the following: - approx usage stat - evictions / attempts per class Background evictors multi-tier Part 3. -------------------------------- use approximate usage fraction
updated the docker gcc version to 12 --------- Co-authored-by: Matt Rae <[email protected]>
instead of always inserting to topmost tier
* Chained item movement between tiers - currently sync on the parent item for moving. - updated tests accordingly, note that we can no longer swap parent item if chained item is being moved for slab release. * added some debug checks around chained item check * fix slab release behavior if no movecb
---------------------------------------------------------------- this can go with background evictors multi-tier part 1
* Set dependencies to working versions and use dependencies from build context, instead of downloading cachelib:develop during build step. This makes sure that dependencies are always build in proper versions. * Fix CacheStats size
Part 4. ------------------------------- batch eviction / promotion - these changes are pretty significant so we would avoid squashing this commit in any prior background evictor patch
-------------------------------- this is to be used in conjunction with the background evictors to completely disable any hot path evictions - forcing all evictions to occur via background threads
…to check if upstream has fixed this yet)
byrnedj
force-pushed
the
develop
branch
3 times, most recently
from
9ba4e79 to
c1ff397
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
should drop once facebook#310 is closed
This change is