[ENH]: soft delete databases, add FinishDatabaseDeletion gRPC method to hard delete databases

[codetheweb]

Description of changes

Databases are now always soft deleted instead of being hard deleted. Databases are hard deleted when the new FinishDatabaseDeletion method is called (the garbage collector calls this in the next PR).

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Updated existing test to cover soft delete behavior.

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

n/a

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[codetheweb]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [ENH]: soft delete databases, add FinishDatabaseDeletion gRPC method to hard delete databases #4627 👈 (View in Graphite)
• [ENH]: perform collection hard deletes from garbage collector #4605
• [ENH]: sysdb changes to support moving collection hard deletes to garbage collector #4607
• [ENH]: add validation during garbage collection for empty file paths #4586
• [ENH]: garbage collector v2 orchestrator (supports forking) #4468
• [ENH]: allow deleting v0 from version file & correctly propagate errors on DeleteCollectionVersion #4579
• [ENH]: add operator for garbage collection to list all files used by collection version #4466
• [ENH]: add operator to compute versions to garbage collect from version graph #4464
• [ENH]: add orchestrator to construct version graph for garbage collection #4463
• [ENH]: add batch get version file paths method to Sysdb #4432 : 1 other dependent PR (#4447 )
• [ENH]: SysDb should return lineage, version file paths and root collection ID on collections #4557
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codetheweb]

this method was never used

[propel-code-bot]

Implement Soft Delete for Databases and Add FinishDatabaseDeletion gRPC

This PR reworks database deletion semantics in the system: database deletions now perform a soft delete by marking the database as deleted, rather than hard-removing it. A new FinishDatabaseDeletion gRPC method is introduced, enabling hard deletion of databases that were soft deleted and now meet eligibility criteria (e.g., after a cutoff time and only if they have no collections). All relevant code paths, protobuf definitions, Rust and Go interfaces, implementations, and tests are modified to support this two-phase deletion strategy.

Key Changes:
• All database deletions are now soft deletes (is_deleted flag set).
• New FinishDatabaseDeletion gRPC and API flows for hard deletion, including proto definitions, Go, and Rust implementations.
• Coordinator and DAO layers updated to support new deletion logic and eligibility checks (e.g., only hard delete databases with no collections after cutoff).
• Test suite refactored to validate soft deletion, hard deletion, and related behaviors (including cutoff time logic).
• All code paths, APIs, and proto interfaces for hard delete are now unified under the new method; prior never-used methods removed.

Affected Areas:
• Database deletion logic (Go: dao, dbmodel, coordinator, gRPC service, test)
• Proto API (idl/chromadb/proto/coordinator.proto)
• Rust bindings for SysDB/client API and error types
• Test suites for database/collection lifecycle

Potential Impact:

Functionality: Changes how databases are deleted throughout the system; existing delete flows perform only soft delete, requiring explicit follow-up for hard deletion via the new FinishDatabaseDeletion endpoint.

Performance: Batch hard deletion is implemented efficiently (in chunks of 1000); no major perf regressions expected but GC timing may be affected.

Security: Soft delete reduces risk of accidental data loss, but it's critical that the hard delete only acts on eligible databases.

Scalability: Improved deletion scalability via batch hard delete and separation of soft/hard deletion, facilitating large-scale cleanup without blocking key operations.

Review Focus:
• Correctness of soft vs. hard deletion logic and eligibility checks.
• Correct propagation of is_deleted/soft-deleted databases through all query layers.
• API compatibility (clients must adapt to the two-phase delete).
• Backward compatibility: ensure no regressions in existing flows lacking hard-deletion.

Testing Needed

• Verify soft-deleted databases are excluded from normal queries immediately after mark/delete.
• Test FinishDatabaseDeletion: multiple scenarios (recently deleted, already deleted, with/without collections, cutoff time in past/future).
• Full integration test passes (pytest/yarn/cargo as reported).

Code Quality Assessment

go/pkg/sysdb/metastore/db/dao/database.go: Major refactor, clear transaction usage for safe updates, all major code paths adapted.

go/pkg/sysdb/grpc/tenant_database_service_test.go: Robust and expanded test coverage for soft/hard deletion and edge cases.

go/pkg/sysdb/coordinator/table_catalog.go: All API logic correctly updated; unused methods removed.

idl/chromadb/proto/coordinator.proto: Well-structured proto additions, backward compatible.

rust/sysdb/src/sysdb.rs: Fully adapted client interface, with structured error handling.

Best Practices

API Design:
• Two-phase deletion aligns with robust data retention policies
• Proto contracts kept forward and backward compatible in most cases

Testing:
• Tests for both soft and hard deletion cases, cutoff logic

Transactionality:
• Database changes occur in transactions
• Eligibility requirements enforced before physical deletion

Potential Issues

• If a collection reference is not cleaned up properly, hard deletion may not proceed (check for lingering collections).
• External clients may need awareness of new API for ultimate cleanup.
• Potential for time skew between systems when using cutoff.

This summary was automatically generated by @propel-code-bot