Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage controller: don't hold detached projects in memory #9712

Open
jcsp opened this issue Nov 11, 2024 · 0 comments · May be fixed by #10264
Open

storage controller: don't hold detached projects in memory #9712

jcsp opened this issue Nov 11, 2024 · 0 comments · May be fixed by #10264
Assignees
Labels
c/storage/controller Component: Storage Controller t/feature Issue type: feature, for new features or requests

Comments

@jcsp
Copy link
Collaborator

jcsp commented Nov 11, 2024

Some use cases for Neon involve storing large numbers of long-term-idle tenants. We already safe resources by detaching them from pageservers, but these tenants remain resident in memory on the storage controller.

Our motivation for storing tenants in-memory (rather than always going to the database) is to enable fast scheduling decisions, but detached tenants are not involved in scheduling.

We should skip these tenants when loading from the database on startup, drop them from the in-memory map of shards when setting them to detached, and then on-demand re-instantiate them when we receive a configuration request that can move them from detached to another state (i.e. a request for /location_conf or to set the tenant's placement policy directly).

@jcsp jcsp added c/storage/controller Component: Storage Controller t/feature Issue type: feature, for new features or requests labels Nov 11, 2024
@jcsp jcsp self-assigned this Jan 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/storage/controller Component: Storage Controller t/feature Issue type: feature, for new features or requests
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant