Code

[WIP] Persistent CAGRA kernel#2316

Code

Closed

achirkin wants to merge 46 commits intorapidsai:branch-24.08rapidsai/raft:branch-24.08from achirkin:fea-persistent-cagraachirkin/raft:fea-persistent-cagraCopy head branch name to clipboard

+1,367-131

Commits

Commits on May 14, 2024

• Sync via barriers

  achirkin

  committed
  f8d25c1

  Copy full SHA for f8d25c1

• Waiting on CPU side.

  Show description for cd012c4
  achirkin

  committed
  cd012c4

  Copy full SHA for cd012c4

• Use simple atomics for synchronization.

  Show description for a1a091c
  achirkin

  committed
  a1a091c

  Copy full SHA for a1a091c

• Added launcher_t - a helper state machine struct to interleave work queue submitting with worker releasing

  achirkin

  committed
  e6ad7b6

  Copy full SHA for e6ad7b6

• Initialize the kernel runner in a separate thread and improve thread safety related to the runner.

  achirkin

  committed
  0137dd4

  Copy full SHA for 0137dd4

• Added small memory sync optimizations

  achirkin

  committed
  25dda44

  Copy full SHA for 25dda44

• Change the benchmark neighbors time size_t -> uint32_t and make benchmark loop event sync optional

  Show description for 1d60ab5
  achirkin

  committed
  1d60ab5

  Copy full SHA for 1d60ab5

• Slightly increase occupancy to improve QPS

  achirkin

  committed
  efd7966

  Copy full SHA for efd7966

• Align input and sync variables with cache lines to avoid cache conflicts

  Show description for e7c35df
  achirkin

  committed
  e7c35df

  Copy full SHA for e7c35df

• Optimize the waiting for the input inside the kernel.

  achirkin

  committed
  7089ed8

  Copy full SHA for 7089ed8

• cagra wrapper: avoid constructing rmm uvectors when not needed

  achirkin

  committed
  259e5ec

  Copy full SHA for 259e5ec

• Avoid any calls to RMM in IO threads.

  Show description for 63f996a
  achirkin

  committed
  63f996a

  Copy full SHA for 63f996a

• Use atomics on the persistent runner (shared_ptr) to reduce the number of mutex locks

  achirkin

  committed
  3d1011d

  Copy full SHA for 3d1011d

• Remove the shared state and the mutex lock from NVTX helpers

  achirkin

  committed
  8b19907

  Copy full SHA for 8b19907

• Split the sync queue in two: job descriptors and idle worker handles

  achirkin

  committed
  4d2b8d5

  Copy full SHA for 4d2b8d5

• Add the third-party atomic_queue headers for easier testing

  achirkin

  committed
  83355ab

  Copy full SHA for 83355ab

• Tweak the CPU waiting behavior to avoid busy-spinning

  achirkin

  committed
  f177a81

  Copy full SHA for f177a81

• Add a single-threaded deque for pending_reads to reduce the cpu/cache load

  achirkin

  committed
  db5b002

  Copy full SHA for db5b002

• ann_bench: minimize chances for GPU sync between benchmark cases

  achirkin

  committed
  c748160

  Copy full SHA for c748160

• Fix OOB bugs revealed on GH

  achirkin

  committed
  d51729c

  Copy full SHA for d51729c

• Add a thread-local weak_ptr for the runner to further reduce possible contention

  achirkin

  committed
  9dd3d32

  Copy full SHA for 9dd3d32

• Keep result buffers between runs to avoid blocking the persistent kernel.

  achirkin

  committed
  4361a5e

  Copy full SHA for 4361a5e

• Avoid an extra layer of atomics on the persistent runner (shared_ptr)

  achirkin

  committed
  e96cc0f

  Copy full SHA for e96cc0f

• Reducing congestions: avoid too many writes to the last_touch/heartbeat atomic

  achirkin

  committed
  c86dfcf

  Copy full SHA for c86dfcf

• Make a custom implementation of the shared resource queue to optimize the throughput when .pop is the bottleneck

  achirkin

  committed
  aaba912

  Copy full SHA for aaba912

• Add expectation-based sleep to the waiting loop

  achirkin

  committed
  8a4ff2e

  Copy full SHA for 8a4ff2e

• Make the gpu worker report reading the handle is done earlier.

  Show description for 732072d
  achirkin

  committed
  732072d

  Copy full SHA for 732072d

• Move the last_touch initialization into the constructor of the containing type

  achirkin

  committed
  7450f6f

  Copy full SHA for 7450f6f

• Modify the resource queue to never loop on head/tail counters

  achirkin

  committed
  8920dfc

  Copy full SHA for 8920dfc

• Replace yield() with a smarter, work-aware pause() to ease the CPU usage by threads

  achirkin

  committed
  ba78957

  Copy full SHA for ba78957

• Expose thread_block_size parameter

  achirkin

  committed
  304a864

  Copy full SHA for 304a864

• Make the 'persistent' parameter in the search_params

  achirkin

  committed
  0879955

  Copy full SHA for 0879955

• Update the parameter parser to use the 'persistent' flag in the search_params struct

  achirkin

  committed
  affdcb2

  Copy full SHA for affdcb2

Commits on May 15, 2024

• Merge remote-tracking branch 'rapidsai/branch-24.06' into fea-persistent-cagra

  achirkin

  committed
  56195f5

  Copy full SHA for 56195f5

• Fix the uses_stream() not adapted to the previous change introducing the 'persistent' search parameter

  achirkin

  committed
  a48d8f8

  Copy full SHA for a48d8f8

• Merge branch 'branch-24.06' into fea-persistent-cagra

  achirkin

  authored
  6a1e5f1

  Copy full SHA for 6a1e5f1

• Merge remote-tracking branch 'rapidsai/branch-24.06' into fea-persistent-cagra

  achirkin

  committed
  c408dae

  Copy full SHA for c408dae

Commits on May 16, 2024

• Merge branch 'branch-24.06' into fea-persistent-cagra

  achirkin

  authored
  cf26a2b

  Copy full SHA for cf26a2b

• Recover the uses_stream() function in the cagra_wrapper after the code merge

  achirkin

  committed
  6079cc9

  Copy full SHA for 6079cc9

Commits on May 17, 2024

• Merge branch 'branch-24.06' into fea-persistent-cagra

  achirkin

  authored
  8dd4714

  Copy full SHA for 8dd4714

Commits on Jun 13, 2024

• Merge branch 'branch-24.08' into fea-persistent-cagra

  achirkin

  authored
  fc2ac99

  Copy full SHA for fc2ac99

• Merge branch 'branch-24.08' into fea-persistent-cagra

  achirkin

  authored
  9580ac6

  Copy full SHA for 9580ac6

Commits on Jun 24, 2024

• Restart the persistent kernel if launch parameters changes

  achirkin

  committed
  74d47c2

  Copy full SHA for 74d47c2

• Merge remote-tracking branch 'rapidsai/branch-24.08' into fea-persistent-cagra

  achirkin

  committed
  608e61d

  Copy full SHA for 608e61d

• Allow 'pinned' value for the benchmark queries and adjust host refinement helper to avoid unnecessary allocations and stream syncs in that case

  achirkin

  committed
  22a77c2

  Copy full SHA for 22a77c2

Commits on Jul 8, 2024

• Merge branch 'branch-24.08' into fea-persistent-cagra

  achirkin

  authored
  c62644d

  Copy full SHA for c62644d