Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v2] Graceful shutdown #659

Merged
merged 5 commits into from
Sep 24, 2024
Merged

[v2] Graceful shutdown #659

merged 5 commits into from
Sep 24, 2024

Conversation

blampe
Copy link
Contributor

@blampe blampe commented Sep 11, 2024

This implements graceful shutdown by propagating interrupts to our child processes.

Killing a workspace pod in the middle of an update will mark the Update as failed ("update canceled" error), and subsequent Updates will resume where it left off as you would expect.

Fixes #607.

@blampe blampe added the impact/no-changelog-required This issue doesn't require a CHANGELOG update label Sep 11, 2024
Copy link

codecov bot commented Sep 11, 2024

Codecov Report

Attention: Patch coverage is 64.78873% with 25 lines in your changes missing coverage. Please review.

Project coverage is 47.27%. Comparing base (2a40b39) to head (afc4d3a).
Report is 1 commits behind head on v2.

Files with missing lines Patch % Lines
agent/cmd/serve.go 30.00% 13 Missing and 1 partial ⚠️
agent/pkg/server/server.go 57.14% 5 Missing and 4 partials ⚠️
agent/cmd/init.go 33.33% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##               v2     #659       +/-   ##
===========================================
+ Coverage   33.11%   47.27%   +14.15%     
===========================================
  Files          27       26        -1     
  Lines        4161     2807     -1354     
===========================================
- Hits         1378     1327       -51     
+ Misses       2610     1316     -1294     
+ Partials      173      164        -9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

agent/cmd/root.go Outdated Show resolved Hide resolved
Comment on lines +140 to +142
if err != nil {
panic(err)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, something happened?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but we shouldn't ignore the error even if we don't expect it.

Copy link
Contributor

@EronWright EronWright left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think the PR could and should be simplified to focus on graceful shutdown, but either way the improvement itself looks good.

agent/cmd/root.go Outdated Show resolved Hide resolved
Comment on lines +33 to +37
_workDir string
_skipInstall bool
_stack string
_host string
_port int
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the "underscore" convention?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/uber-go/guide/blob/master/style.md#prefix-unexported-globals-with-_

Makes global symbols (and their mutation) more obvious to readers.

agent/cmd/serve.go Outdated Show resolved Hide resolved
log.Infow("starting the RPC server", "address", address)
serverOpts := []grpc_zap.Option{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I felt better with this setup code being in serve.go rather than the server package, because a) the setup code tends to be parameterized by flags, b) there might be other RPC services to be mixed into the server, and c) code that watches signals and calls os.Exit should be in the cmd package. Moving the code has made this PR harder to understand.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally right about os.Exit belonging in cmd -- I moved that back.

In general I treat cmd as the place for executables (package main) and try to limit their scope to flag parsing and terminal output. With cobra especially it helps to keep Run functions as small wrappers around other library calls -- for example in this case I need to extract the gRPC setup into something a test can re-use. The server can still be customized with minimal changes to NewGRPC.

@blampe blampe changed the title Graceful shutdown [v2] Graceful shutdown Sep 24, 2024
@blampe blampe merged commit 181286c into v2 Sep 24, 2024
7 checks passed
@blampe blampe deleted the blampe/v2-graceful branch September 24, 2024 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
impact/no-changelog-required This issue doesn't require a CHANGELOG update
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement graceful termination for the server, canceling outstanding operations
2 participants