CMake #421

…st 2

Change min tested version to 18.04

…e are still issues with the sentinel tests but attempting to bind only to localhost causes failures

…ync (Snapchat#371) * save replid for all masters in rdb * expanded rdbSaveInfo to hold multiple master structs * parse repl-masters from rdb * recover replid info from rdb in active replica mode, attempt partial sync * save offset from rdb into correct variable * don't change replid based on master in active rep * save and load psync info from correct fields

…ge non TLS behavior

…/keydb-internal into merge_os_apr_13

setup Machamp ci

…internal into keydbpro

Merge in changes from Open Source. For the most part, this encompasses two changes: - Adding partial sync capabilities to multi-master - Adding ReadWrite locks to prevent global lock issues when forking threads, specifically with regards to the time thread.

…internal into keydbpro

…Snapchat/keydb-internal into psync_multimaster_fixes

Psync multimaster fixes

Quick fix to make psync tests work

Fix PSYNC test crashes

…Pack, CTest; [vcpkg.json] Use open-source cross-platform dependency manager

Merge OSS back into pro

…y disconnect preventing disruption during shutdown

Implements the soft shutdown feature

Update build-and-publish.sh

…internal into keydbpro

Fix certificate leak during connection when tls-allowlists are used

Check tracking on every db in rdbLoadRio

… the connection is not yet closed and the event handler runs

…g the lock when trackChanges is set

Fix a race in processChanges/trackChanges with rdbLoadRio by acquirin…

Fix a race when undoConnectWithMaster changes mi->repl_transfer_s but…

Repl asan fixes

…loop tlsProcessPendingData() needs to be called before we execute queued commands because it may enqueue more commands

Atlas fixes

Improve reliability of machamp tests

Verify that partial syncs occur in psync2-reg-multimaster

* need to include stdint for uintptr_t * need to include stdint for uintptr_t * use atomic_load for g_pserver->mstime * use atomic_load for g_pserver->mstime * Integrate readwritelock with Pro Code * Integrate readwritelock with Pro Code * Defensive asserts for RWLock * Defensive asserts for RWLock * Save and restore master info in rdb to allow active replica partial sync (Snapchat#371) * save replid for all masters in rdb * expanded rdbSaveInfo to hold multiple master structs * parse repl-masters from rdb * recover replid info from rdb in active replica mode, attempt partial sync * save offset from rdb into correct variable * don't change replid based on master in active rep * save and load psync info from correct fields * Save and restore master info in rdb to allow active replica partial sync (Snapchat#371) * save replid for all masters in rdb * expanded rdbSaveInfo to hold multiple master structs * parse repl-masters from rdb * recover replid info from rdb in active replica mode, attempt partial sync * save offset from rdb into correct variable * don't change replid based on master in active rep * save and load psync info from correct fields * placement new instead of memcpy * placement new instead of memcpy * Remove asserts, RW lock can go below zero in cases of aeAcquireLock * Remove asserts, RW lock can go below zero in cases of aeAcquireLock * Inclusive language * Inclusive language * update packaging for OS merge * update packaging for OS merge * modify dockerfile to build within image * modify dockerfile to build within image * Make active client balancing a configurable option * Make active client balancing a configurable option * With TLS throttle accepts if server is under heavy load - do not change non TLS behavior * With TLS throttle accepts if server is under heavy load - do not change non TLS behavior * Only run the tls-name-validation test if --tls is passed into runtest * Only run the tls-name-validation test if --tls is passed into runtest * Fix KeyDB not building with TLS < 1.1.1 * Fix KeyDB not building with TLS < 1.1.1 * update changelog to use replica as terminology * update changelog to use replica as terminology * update copyright * update copyright * update deb copyright * update deb copyright * call aeThreadOnline() earlier * call aeThreadOnline() earlier * Removed mergeReplicationId * Removed mergeReplicationId * acceptTLS is threadsafe like the non TLS version * acceptTLS is threadsafe like the non TLS version * setup Machamp ci * setup Machamp ci * make build_test.sh executable * make build_test.sh executable * PSYNC production fixes * PSYNC production fixes * fix the Machamp build * fix the Machamp build * break into tests into steps * break into tests into steps * Added multimaster test * Added multimaster test * Update ci.yml Change min tested version to 18.04 * Update ci.yml Change min tested version to 18.04 * fork lock for all threads, use fastlock for readwritelock * fork lock for all threads, use fastlock for readwritelock * hide forklock object in ae * hide forklock object in ae * only need to include readwritelock in ae * only need to include readwritelock in ae * time thread lock uses fastlock instead of std::mutex * time thread lock uses fastlock instead of std::mutex * set thread as offline when waiting for time thread lock * set thread as offline when waiting for time thread lock * update README resource links * update README resource links * Fix MALLOC=memkind build issues * Fix MALLOC=memkind build issues * Fix module test break * Fix module test break * Eliminate firewall dialogs on mac for regular and cluster tests. There are still issues with the sentinel tests but attempting to bind only to localhost causes failures * Eliminate firewall dialogs on mac for regular and cluster tests. There are still issues with the sentinel tests but attempting to bind only to localhost causes failures * remove unused var in networking.cpp * remove unused var in networking.cpp * check ziplist len to avoid crash on empty ziplist convert * check ziplist len to avoid crash on empty ziplist convert * remove nullptr subtraction * remove nullptr subtraction * cannot mod a pointer * cannot mod a pointer * need to include stdint for uintptr_t * need to include stdint for uintptr_t * use atomic_load for g_pserver->mstime * use atomic_load for g_pserver->mstime * Integrate readwritelock with Pro Code * Integrate readwritelock with Pro Code * Defensive asserts for RWLock * Defensive asserts for RWLock * Save and restore master info in rdb to allow active replica partial sync (Snapchat#371) * save replid for all masters in rdb * expanded rdbSaveInfo to hold multiple master structs * parse repl-masters from rdb * recover replid info from rdb in active replica mode, attempt partial sync * save offset from rdb into correct variable * don't change replid based on master in active rep * save and load psync info from correct fields * Save and restore master info in rdb to allow active replica partial sync (Snapchat#371) * save replid for all masters in rdb * expanded rdbSaveInfo to hold multiple master structs * parse repl-masters from rdb * recover replid info from rdb in active replica mode, attempt partial sync * save offset from rdb into correct variable * don't change replid based on master in active rep * save and load psync info from correct fields * placement new instead of memcpy * placement new instead of memcpy * Remove asserts, RW lock can go below zero in cases of aeAcquireLock * Remove asserts, RW lock can go below zero in cases of aeAcquireLock * Inclusive language * Inclusive language * call aeThreadOnline() earlier * call aeThreadOnline() earlier * Removed mergeReplicationId * Removed mergeReplicationId * Make active client balancing a configurable option * Make active client balancing a configurable option * With TLS throttle accepts if server is under heavy load - do not change non TLS behavior * With TLS throttle accepts if server is under heavy load - do not change non TLS behavior * acceptTLS is threadsafe like the non TLS version * acceptTLS is threadsafe like the non TLS version * PSYNC production fixes * PSYNC production fixes * Ensure we are responsive during storagecache clears * Ensure we are responsive during storagecache clears * Ensure recreated tables use the same settings as ones made at boot * Ensure recreated tables use the same settings as ones made at boot * Converted some existing PSYNC tests for multimaster * Converted some existing PSYNC tests for multimaster * Inclusive language fix * Inclusive language fix * Cleanup test suite * Cleanup test suite * Updated test replica configs so tests make sense * Updated test replica configs so tests make sense * active-rep test reliability * active-rep test reliability * Quick fix to make psync tests work * Quick fix to make psync tests work * Fix PSYNC test crashes * Fix PSYNC test crashes * Ensure we force moves not copies when ingesting bulk insert files * Ensure we force moves not copies when ingesting bulk insert files * Disable async for hget commands as it is not ready * Disable FLASH * Fix crash in save of masterinfo * Fix musl/Alpine build failures * Remove unnecessary libs * update readme * update readme * remove Enterprise references * Limit max overage to 20% during RDB save * Delete COPYING to replace with BSD license * update deb master changelog * Update license * Fix Readme typo from github org transition Replace mention of scratch-file-path with db-s3-object * Fix reference counting failure in the dict. This is caused by std::swap also swapping refcounts * Fix assertion in async rehash * Prevent crash on shutdown by avoiding dtors (they are unnecessary anyways) * Initialize noshrink, it was dangling * Prevent us from starting a rehash when one wasn't already in progress. This can cause severe issues for snapshots * Avoid unnecessary rehashing when a rehash is abandoned * Dictionary use correct acquire/release semantics * Add fence barriers for the repl backlog (important for AARCH64 and other weak memory models) * Silence TSAN errors on ustime and mstime. Every CPU we support is atomic on aligned ints, but correctness matters * Disable async commands by default * Fix TSAN warnings on the repl backlog * Merge OSS back into pro * Fix unmerged files * Fix O(n^2) algorithm in the GC cleanup logic * Fix crash in expire when a snapshot is in flight. Caused by a perf optimization getting the expire map out of sync with the val * On Alpine we must have a reasonable stack size * Revert ci.yml to unstable branch version * Implements the soft shutdown feature to allow clients to cooperatively disconnect preventing disruption during shutdown * Ensure clean shutdown with multiple threads * update dockerfiles * update deb pkg references and changelog * update gem reference * lpGetInteger returns int64_t, avoid overflow (#10068) Fix #9410 Crucial for the ms and sequence deltas, but I changed all calls, just in case (e.g. "flags") Before this commit: `ms_delta` and `seq_delta` could have overflown, causing `currid` to be wrong, which in turn would cause `streamTrim` to trim the entire rax node (see new test) * Fix issue Snapchat#454 (BSD build break) * Do not allow commands to run in background when in eval, Issue Snapchat#452 * Fix certificate leak during connection when tls-allowlists are used * Fix issue Snapchat#480 * Fix crash running INFO command while a disk based backlog is set * check tracking per db * fix warnings * Fix a race when undoConnectWithMaster changes mi->repl_transfer_s but the connection is not yet closed and the event handler runs * Fix a race in processChanges/trackChanges with rdbLoadRio by acquiring the lock when trackChanges is set * Fix ASAN use after free * Additional fixes * Fix integer overflow of the track changes counter * Fix P99 latency issue for TLS where we leave work for the next event loop tlsProcessPendingData() needs to be called before we execute queued commands because it may enqueue more commands * Fix race removing key cache * Prevent crash on load in long running KeyDB instances * Fixes a crash where the server assertion failed when the key exists in DB during RDB load * Remove old assertion which is commented out. * avoid from instatiating EpochHolder multiple times to improve performance and cpu utilization * avoid from instatiating EpochHolder multiple times to improve performance and cpu utilization * src\redis-cli.c: fix potential null pointer dereference found by cppcheck src\redis-cli.c:5488:35: warning: Either the condition '!table' is redundant or there is possible null pointer dereference: table. [nullPointerRedundantCheck] * Fix Issue Snapchat#486 * Workaround bug in snapshot sync - abort don't crash * Improve reliability of async parts of the soft shutdown tests * Improve reliability of fragmentation tests * Verify that partial syncs do indeed occur * Fix O(n) algorithm in INFO command * Remove incorrect assert that fires when the repl backlog is used fully * Make building flash optional * Remove unneeded gitlab CI file * [BUG] Moves key to another DB, the source key was removed if the move failed due to the key exists in the destination db Snapchat#497 (Snapchat#498) Co-authored-by: Paul Chen <[email protected]> * trigger repl_curr_off!= master_repl_offset assert failure when having pending write case * use debug for logging the message instead * rocksdb log using up the diskspace on flash (Snapchat#519) * Fix OpenSSL 3.0.x related issues. (#10291) * Drop obsolete initialization calls. * Use decoder API for DH parameters. * Enable auto DH parameters if not explicitly used, which should be the preferred configuration going forward. * remove unnecessary forward declaration * remove internal ci stuff * remove more internal ci/publishing * submodule update step * use with syntax instead * bump ci ubuntu old ver as latest is now 22.04 * include submodules on all ci jobs * install all deps for all ci jobs Co-authored-by: Vivek Saini <[email protected]> Co-authored-by: Christian Legge <[email protected]> Co-authored-by: benschermel <[email protected]> Co-authored-by: John Sully <[email protected]> Co-authored-by: zliang <[email protected]> Co-authored-by: malavan <[email protected]> Co-authored-by: John Sully <[email protected]> Co-authored-by: jfinity <[email protected]> Co-authored-by: benschermel <[email protected]> Co-authored-by: guybe7 <[email protected]> Co-authored-by: Karthick Ariyaratnam (A) <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Ilya Shipitsin <[email protected]> Co-authored-by: Paul Chen <[email protected]> Co-authored-by: Paul Chen <[email protected]> Co-authored-by: Yossi Gottlieb <[email protected]>

* fix macos build warnings * remove 32 bit ci run as we no longer support it

…t#523)

* Fix TLS tests on newer tcl-tls/OpenSSL. (#10910) Before this commit, TLS tests on Ubuntu 22.04 would fail as dropped connections result with an ECONNABORTED error thrown instead of an empty read. * multithread for ci tests * multithread cluster tests * clients to 1 in ci Co-authored-by: Yossi Gottlieb <[email protected]>

* ship packages with flash option * deb12 fix Co-authored-by: benschermel <[email protected]>

…RT, Fix range issues in ZRANDMEMBER and HRANDFIELD (Snapchat#547) * Avoid integer overflows in SETRANGE and SORT (CVE-2022-35977) (#11720) Authenticated users issuing specially crafted SETRANGE and SORT(_RO) commands can trigger an integer overflow, resulting with Redis attempting to allocate impossible amounts of memory and abort with an OOM panic. * Fix range issues in ZRANDMEMBER and HRANDFIELD (CVE-2023-22458) (#11674) missing range check in ZRANDMEMBER and HRANDIFLD leading to panic due to protocol limitations * use std::min/max * add assert_not_equal Co-authored-by: Oran Agra <[email protected]>

… write batch is active (Snapchat#550)

Currently flash is always built, with this change it is actually optional.

…rwise we'll miss it

…ual child process (Snapchat#558) change hasActiveChildProcess to return true only when there is an actual child process, add hasActiveChildProcessOrBGSave to catch case of forkless bgsave

…pchat#536)

…child

Fixes Snapchat#562

* technically possible for child_type == CHILD_TYPE_AOF without active child * don't release lock on child as it can hang * need child specific release that doesn't trigger cv * refactor aeReleaseForkLockChild to capture releaseRead case * rdb_child_pid isn't the correct value

Fix snprintf uses to avoid hardcoded sizes

Close child info pipes when RDB save thread is done.

…#578) * remove short circuit as it is unsafe

* upgrade to rocksdb v7.9.2 * need c++17 for rocksdb * don't prefix jemalloc * different exception specifier

* close tempfile when done bgsave * declare getTempFileName * open close child pipes in correct place * assert no existing child pipe when opening new child pipe

* add s3 config * missing include * ignore s3 config in config test * Update introspection.tcl * Update config.cpp

…or module notify (Snapchat#604)

) * add expire_lock to previously async access to expireset * add flash option to create-cluster

* restore fork impl of save to slave sockets * default to fork bgsave

…55) (#11857) Issue happens when passing a negative long value that greater than the max positive value that the long can store.

…149) Check the validity of the value before performing the create operation, prevents new data from being generated even if the request fails to execute. Co-authored-by: Oran Agra <[email protected]> Co-authored-by: chendianqiang <[email protected]> Co-authored-by: Binbin <[email protected]>

…arly if client is being closed

* add hash prefixing to rocksdb * don't need to offset internal keys * remove hashslot prefix when loading from rocksdb * don't prefix internal keys * implement slot to keys with rocksdb * fix compile * add hashslot enum to test storage provider * add --flash option for tests * forgot return statement in getKeysInSlot * check for count mismatch * forgot ; * add assert to ensure correctness * fix warnings * add option to show logs of 1 server at a time * add fixed length prefix extractor * add machamp * switch machamp to main branch * add machamp script

…t#662)

* Update build.yaml

…lti-master configs (Snapchat#666) * load error when replicaof is set before multi-master or active-replica * only an issue if enabled

…in FLASH mode. (Snapchat#668) Co-authored-by: k00809413 <[email protected]>

* ignore propagate_in_transaction in processCommand if command is async * make propogate_in_transaction thread local * forgot module.cpp * move client_pause_in_transaction to thread local * don't do async commands while in eval or exec * don't do keys or scan async if in transaction

…use a dict that has already been released (Snapchat#199)

* remove keyproxy test from machamp * Update build.yaml * make insertsinflight shared

* ignore propagate_in_transaction in processCommand if command is async * make propogate_in_transaction thread local * forgot module.cpp * move client_pause_in_transaction to thread local * don't do async commands while in eval or exec * don't do keys or scan async if in transaction

…use a dict that has already been released (Snapchat#199)

* remove keyproxy test from machamp * Update build.yaml * make insertsinflight shared

…in FLASH mode. (Snapchat#668) Co-authored-by: k00809413 <[email protected]>

* Audit Logging for KeyProxy and KeyDB (Snapchat#144) * Audit Log: log cert fingerprint (Snapchat#151) * Add more flash storage stats to info command. * Remove unneeded libs when not building FLASH * Fix mem leak * Allow the reservation of localhost connections to ensure health checks always succeed even at maxclients (Snapchat#181) * Enable a force option for commands (Snapchat#183) * Fix missing newline and excessive logging in the CLI * Support NO ONE for "CLUSTER REPLICATE" command. Co-authored-by: Jacob Bohac <[email protected]> Co-authored-by: Sergey Kolosov <[email protected]> Co-authored-by: John Sully <[email protected]> Co-authored-by: John Sully <[email protected]>

comparision -> comparison

Force evictions when system available memory is low

… mode. (Snapchat#696) Co-authored-by: Karthick Ariyaratnam (A) <[email protected]>

…th the runtime value during config rewrite.

…ents (Snapchat#206)

comparision -> comparison

… mode. (Snapchat#696) Co-authored-by: Karthick Ariyaratnam (A) <[email protected]>

…th the runtime value during config rewrite.

…files ; [src/modules/CMakeLists.txt] Initial add for modules ; [CMakeLists.txt] Bump version

Commits on Jun 5, 2022

Merge branch 'main' into keydbpro

JohnSully committed Jun 5, 2022

Configuration menu

View commit details

Copy full SHA for d6cf39c

Browse repository at this point

Copy the full SHA

d6cf39c View commit details

Browse the repository at this point in the history

Commits on Jan 25, 2023

fix typo & discrepancy

benschermel committed Jan 25, 2023

Configuration menu

View commit details

Copy full SHA for 1113488

Browse repository at this point

Copy the full SHA

1113488 View commit details

Browse the repository at this point in the history

Commits on May 8, 2023

Make it compile on FreeBSD (Snapchat#650 )

bra-fsn authored May 8, 2023

Configuration menu

View commit details

Copy full SHA for f629a76

Browse repository at this point

Copy the full SHA

f629a76 View commit details

Browse the repository at this point in the history

Commits on Jul 6, 2023

ok

acope-sc committed Jul 6, 2023

Configuration menu

View commit details

Copy full SHA for e6bc1c5

Browse repository at this point

Copy the full SHA

e6bc1c5 View commit details

Browse the repository at this point in the history

CMake #421

Are you sure you want to change the base?

CMake #421

Commits on Apr 21, 2022

Commits on Apr 22, 2022

Commits on Apr 23, 2022

Commits on Apr 25, 2022

Commits on Apr 26, 2022

Commits on Apr 27, 2022

Commits on Apr 28, 2022

Commits on May 2, 2022

Commits on May 3, 2022

Commits on May 4, 2022

Commits on May 5, 2022

Commits on May 13, 2022

Commits on May 18, 2022

Commits on May 24, 2022

Commits on May 25, 2022

Commits on Jun 5, 2022

Commits on Jun 14, 2022

Commits on Jun 29, 2022

Commits on Jul 14, 2022

Commits on Aug 17, 2022

Commits on Aug 21, 2022

Commits on Aug 22, 2022

Commits on Aug 23, 2022

Commits on Aug 26, 2022

Commits on Aug 30, 2022

Commits on Sep 28, 2022

Commits on Sep 29, 2022

Commits on Nov 24, 2022

Commits on Dec 14, 2022

Commits on Dec 15, 2022

Commits on Dec 29, 2022

Commits on Jan 17, 2023

Commits on Jan 20, 2023

Commits on Jan 23, 2023

Commits on Jan 25, 2023

Commits on Jan 30, 2023

Commits on Jan 31, 2023

Commits on Feb 3, 2023

Commits on Feb 6, 2023

Commits on Feb 7, 2023

Commits on Feb 9, 2023

Commits on Feb 10, 2023

Commits on Feb 13, 2023

Commits on Feb 15, 2023

Commits on Feb 20, 2023

Commits on Feb 21, 2023

Commits on Feb 23, 2023

Commits on Mar 2, 2023

Commits on Mar 8, 2023

Commits on Mar 13, 2023

Commits on Mar 14, 2023

Commits on Mar 31, 2023

Commits on Apr 6, 2023

Commits on Apr 7, 2023

Commits on Apr 12, 2023

Commits on Apr 19, 2023

Commits on Apr 25, 2023

Commits on May 8, 2023

Commits on May 25, 2023

Commits on May 30, 2023

Commits on Jun 1, 2023

Commits on Jun 12, 2023

Commits on Jun 15, 2023

Commits on Jun 21, 2023

Commits on Jun 27, 2023

Commits on Jun 29, 2023

Commits on Jun 30, 2023

Commits on Jul 5, 2023

Commits on Jul 6, 2023

Commits on Jul 7, 2023

Commits on Jul 10, 2023

Commits on Jul 13, 2023

Commits on Jul 14, 2023

Commits on Jul 26, 2023

Commits on Aug 22, 2023