Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Async flash #781

Open
wants to merge 72 commits into
base: main
Choose a base branch
from
Open

Async flash #781

wants to merge 72 commits into from

Commits on Sep 18, 2023

  1. async flash start

    JohnSully committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    1e67982 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2023

  1. Initial async impl

    JohnSully committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    e95c8b4 View commit details
    Browse the repository at this point in the history
  2. Use MultiGet

    JohnSully committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    f0e8c6f View commit details
    Browse the repository at this point in the history
  3. Remove a global lock

    JohnSully committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    cdc5e8f View commit details
    Browse the repository at this point in the history
  4. Avoid 0 size batches

    JohnSully committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    71b92e7 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. More fixes

    JohnSully committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    b0935de View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2023

  1. Configuration menu
    Copy the full SHA
    7e745ef View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a4ca527 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2023

  1. want to correctly cache the system total memory always, as force-evic…

    …tion-percent is a mutable config (#236)
    msotheeswaran-sc authored and GitHub Enterprise committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    0b5b0c0 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2023

  1. commitChanges async api

    a00817524 authored and JohnSully committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    f53dbd3 View commit details
    Browse the repository at this point in the history
  2. missed changes

    a00817524 authored and JohnSully committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    eea48c5 View commit details
    Browse the repository at this point in the history
  3. some fixes

    a00817524 authored and JohnSully committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    e874752 View commit details
    Browse the repository at this point in the history
  4. build fixes

    a00817524 authored and JohnSully committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    4c014ee View commit details
    Browse the repository at this point in the history
  5. correcting classname

    a00817524 authored and JohnSully committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    1129f67 View commit details
    Browse the repository at this point in the history
  6. missed changes

    a00817524 authored and JohnSully committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    de73b2c View commit details
    Browse the repository at this point in the history
  7. fixed some errors

    a00817524 authored and JohnSully committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    4818ddf View commit details
    Browse the repository at this point in the history
  8. reverting changes of async io rocksdb option as this is version depen…

    …dent
    a00817524 authored and JohnSully committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    0cf263c View commit details
    Browse the repository at this point in the history
  9. removing unwanted loc

    a00817524 authored and JohnSully committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    567ef8d View commit details
    Browse the repository at this point in the history
  10. whitespace removed

    a00817524 authored and JohnSully committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    b6b7b1b View commit details
    Browse the repository at this point in the history
  11. review rework, added serverassert default switch case,removed debug p…

    …rints, removed unwanted reset of pointers
    a00817524 authored and JohnSully committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    1d59efb View commit details
    Browse the repository at this point in the history
  12. add support for overload-ignorelist (#237)

    msotheeswaran-sc authored and GitHub Enterprise committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    c889505 View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2023

  1. Configuration menu
    Copy the full SHA
    20ae39d View commit details
    Browse the repository at this point in the history
  2. Overload CPU reading metric (#240)

    * Overload CPU reading metric
    
    * fix
    
    * fix
    jbohac-sc authored and GitHub Enterprise committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    8a5857d View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2023

  1. Configuration menu
    Copy the full SHA
    8fec9ba View commit details
    Browse the repository at this point in the history
  2. add cool

    zliang-sc committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    443e1af View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    18d6241 View commit details
    Browse the repository at this point in the history
  4. Merge pull request #241 from Snapchat/improve-insufficient-replica-alarm

    update insufficient replicas metrics to separate severity
    zliang-sc authored and GitHub Enterprise committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    deb139e View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2023

  1. update overload-ignore-list to accept subnets as well (#239)

    * update overload-ignore-list to be use subnets as well
    
    * get ip directly from conn
    
    * add getIPV4 impl
    
    * ntohl
    
    * don't shift at the end of the loop
    
    * update for more efficient impl and add ipv6 support
    
    * array of arrays
    
    * one combined array
    
    * have to reassign output of sdscat
    
    * include mask in operator<
    
    * replace raw numbers with constants
    
    * add log when we ignore client for loadshedding
    msotheeswaran-sc authored and GitHub Enterprise committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    49afd8f View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2023

  1. add overload-protect-tenacity to configure how many clients to evict (#…

    …243)
    
    * add overload-protect-tenacity to configure how many clients to evict
    
    * need to divide tenacity config by 100 to get percentage
    msotheeswaran-sc authored and GitHub Enterprise committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    bb115ca View commit details
    Browse the repository at this point in the history
  2. TLS cname loadshedding ignore list (#242)

    * TLS cname loadshedding ignore list
    jbohac-sc authored and GitHub Enterprise committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    bb7c5a4 View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2024

  1. refactor asyncwrite and asyncread work queue

    Yu Zhao (Yuri) authored and JohnSully committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    e1935a7 View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2024

  1. Fix compile warning

    JohnSully committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    3d4c5a0 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2024

  1. change overload protection to sending -OVERLOAD errors instead of dis…

    …connecting clients (#245)
    msotheeswaran-sc authored and GitHub Enterprise committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    b74652a View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2024

  1. handle nodes in handshake state in statsd module cluster nodes metrics (

    #246)
    
    * handle nodes in handshake state in statsd module cluster nodes metrics
    
    * handle all possible outputs of cluster nodes
    
    * Update src/modules/keydb_modstatsd/modmain.cpp
    
    Co-authored-by: Sergey Kolosov <[email protected]>
    
    * make error plural
    
    Co-authored-by: Sergey Kolosov <[email protected]>
    2 people authored and GitHub Enterprise committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    2b01c6c View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2024

  1. reset replica offsets after resetting replication buffer (#247)

    msotheeswaran-sc authored and GitHub Enterprise committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    7a32cf9 View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2024

  1. Add redis bloom module (#252)

    * build redisbloom in dockerfile
    
    * add separate step for redisbloom
    
    * end RUN properly
    
    * can't verify with github?
    
    * just needed ca-cert
    msotheeswaran-sc authored and GitHub Enterprise committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    d1484aa View commit details
    Browse the repository at this point in the history
  2. Fix replication rdb load timeout (#251)

    * don't timeout replication when loading rdb
    
    * fix bracket placement
    msotheeswaran-sc authored and GitHub Enterprise committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    5681d06 View commit details
    Browse the repository at this point in the history
  3. skip memefficiency tests in CI as they are inconsistent (#250)

    msotheeswaran-sc authored and GitHub Enterprise committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    77eaaa2 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2024

  1. Configuration menu
    Copy the full SHA
    65b9165 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    989421e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7b50232 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2024

  1. Log hitrate metric in modstatsd (#255)

    https://jira.sc-corp.net/browse/CACHE-1543
    
    Log hit rate metric separately because M3DB caps values at 360 million so dividing monotonically increasing `keyspace_hit/miss` values is meaningless with any significant amount of traffic. Multiplying by one million since M3DB only uses integer values.
    acope-sc authored and GitHub Enterprise committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    195828e View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2024

  1. Limit repl buffer writes per event loop (#256)

    We discovered that full sync loops were happening because after the replica loads the RDB file, the repl backlog which the primary sends over can sometimes exceed int32 max, causing an SSL error due to integer overflow:
    
    ```
    Error writing to client: error:140D010F:SSL routines:SSL_write:bad length
    ```
    
    After fixing that error by capping the  and testing in staging, we still observed errors because the primary attempts to send a massive (multi-GB) payload in one event loop, which on the staging instances takes over one minute causing the primary to fail health checks and restart. To address this we cap the writes sent per event loop to a configurable amount, similar to `NET_MAX_WRITES_PER_EVENT` in the normal case.
    acope-sc authored and GitHub Enterprise committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    cba3998 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. Fix for hit rate metric (#257)

    Tweak hit rate metric to get delta instead of aggregate
    acope-sc authored and GitHub Enterprise committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    752f204 View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. Configuration menu
    Copy the full SHA
    490ee84 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2024

  1. Configuration menu
    Copy the full SHA
    c08683e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    260f47b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e5f8e77 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. Configuration menu
    Copy the full SHA
    e7ab275 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. Fix crash

    JohnSully committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    8f2dbc3 View commit details
    Browse the repository at this point in the history
  2. Fix hung clients

    JohnSully committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    c5c4779 View commit details
    Browse the repository at this point in the history
  3. Fix typo in test path

    JohnSully committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    8c4fa5d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    7a488a8 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8a2016a View commit details
    Browse the repository at this point in the history
  6. Fix ARM CI

    JohnSully committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    bff8b73 View commit details
    Browse the repository at this point in the history
  7. Build with flash

    JohnSully committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    6c15c43 View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2024

  1. Emit maxstorage metric

    JohnSully committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    fe5503a View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. Configuration menu
    Copy the full SHA
    c83c55f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ad2a282 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    088a1db View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    d84fa52 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    559588b View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2024

  1. Configuration menu
    Copy the full SHA
    5fbe991 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2024

  1. Configuration menu
    Copy the full SHA
    a430473 View commit details
    Browse the repository at this point in the history
  2. Add FLASH tests to CI

    JohnSully committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    0267bc5 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f5e367d View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. Configuration menu
    Copy the full SHA
    41733d7 View commit details
    Browse the repository at this point in the history
  2. Fix warning

    JohnSully committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    bb1780a View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2024

  1. Handle pending IO fastsync

    JohnSully committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    e580a8e View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2024

  1. Configuration menu
    Copy the full SHA
    c50e2d4 View commit details
    Browse the repository at this point in the history
  2. Naming fix

    JohnSully committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    66be482 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f886fce View commit details
    Browse the repository at this point in the history