Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add autoscaling to Helm charts #90

Open
d47853 opened this issue Jun 29, 2020 · 1 comment
Open

Add autoscaling to Helm charts #90

d47853 opened this issue Jun 29, 2020 · 1 comment
Labels
enhancement Improvement to existing functionality/feature headliner Used to tag major items for release notes Kubernetes Issue related to the Kubernetes side of the project
Milestone

Comments

@d47853
Copy link
Member

d47853 commented Jun 29, 2020

Gaffer should scale up when hdfs starts filling up and down when it becomes under-utilised to allow. A nice to have feature would be to be able to do this on a schedule as well - so Gaffer can in effect shut down at night to reduce resources and costs when deployed on a Kubernetes cluster.

@d47853 d47853 added enhancement Improvement to existing functionality/feature Kubernetes Issue related to the Kubernetes side of the project labels Jun 29, 2020
@d47853 d47853 self-assigned this Jun 29, 2020
@d47853 d47853 added the headliner Used to tag major items for release notes label Jun 29, 2020
@ctas582
Copy link
Member

ctas582 commented Jun 29, 2020

Scaling Accumulo's tablet servers should be easy enough, as they are pretty much stateless. However, just want to add a note of caution about scaling the HDFS data nodes. Adding additional data nodes is straight forward, however removing them is more complicated. With the default way we are running HDFS (replication factor 3, with no topology config), we will only be able to remove 2 data nodes at a time, otherwise you are pretty much guaranteed to lose data. You then need to wait for blocks to be fully replicated across the remaining nodes before being able to remove any more.

Specifying a topology should help, as it would allow us to remove any number of data nodes from 2 of the 3 zones at once. However, we currently rely on the StatefulSet operator to add/remove/replace data node Pods. Afaik it doesn't give you control over which exact data node Pod gets removed. I think it's only LIFO. We might need to look into creating our own operator, if we can't find something in the open-source community.

@d47853 d47853 removed their assignment Nov 5, 2020
@GCHQDeveloper314 GCHQDeveloper314 added this to the post-v2.0_backlog milestone Nov 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improvement to existing functionality/feature headliner Used to tag major items for release notes Kubernetes Issue related to the Kubernetes side of the project
Projects
None yet
Development

No branches or pull requests

3 participants