Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add build metadata to db format version and move indexer column families behind a feature flag #8922

Open
arya2 opened this issue Oct 10, 2024 · 0 comments · May be fixed by #8895
Open

Add build metadata to db format version and move indexer column families behind a feature flag #8922

arya2 opened this issue Oct 10, 2024 · 0 comments · May be fixed by #8895
Assignees
Labels
A-state Area: State / database changes C-cleanup Category: This is a cleanup C-enhancement Category: This is an improvement lightwalletd any work associated with lightwalletd S-needs-design Status: Needs a design decision

Comments

@arya2
Copy link
Contributor

arya2 commented Oct 10, 2024

Motivation

The indexes added in #8895 use more storage than expected and are only required for Zebrad nodes that are supporting a light client server.

We want to avoid caching them when they're not needed and ideally allow for pruning them.

Keeping these indexes behind a feature flag or only adding them when enabled in the configuration would require Zebra to concurrently support multiple db formats if/when another db format version bump is required.

If Zebra starts with the indexer feature, then shuts down and restarts without the feature, the db format version on disk will still match the one in code, but the indexes won't be available in the state cache.

Possible Design

Zebra could add build metadata to the db format version to indicate whether the state cache has an index of spending transaction ids.

It could then remove the build metadata from the on-disk db format version when it starts without the feature or indexing enabled in the configuration, and always run the db format upgrade for adding the spending transaction id indexes when it's missing from the on-disk db format version's build metadata.

Zebra could also track which blocks should have the indexes in a new column family so it can start where it left off instead of traversing the entire chain up to the initial tip height if necessary.

@arya2 arya2 added C-enhancement Category: This is an improvement C-cleanup Category: This is a cleanup A-state Area: State / database changes lightwalletd any work associated with lightwalletd labels Oct 10, 2024
@mpguerra mpguerra added the S-needs-design Status: Needs a design decision label Oct 18, 2024
@arya2 arya2 self-assigned this Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-state Area: State / database changes C-cleanup Category: This is a cleanup C-enhancement Category: This is an improvement lightwalletd any work associated with lightwalletd S-needs-design Status: Needs a design decision
Projects
Status: New
2 participants