Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Celery process exits with status code 1 on shutdown after v1.0 upgrades #1492

Open
mikkonie opened this issue Sep 9, 2024 · 3 comments
Open

Celery process exits with status code 1 on shutdown after v1.0 upgrades #1492

mikkonie opened this issue Sep 9, 2024 · 3 comments
Assignees
@mikkonie
Labels
bug Something isn't working environment Issues of dependencies, CI, deployment etc. ongoing Ongoing issue, needs observation or pending on other projects

Comments

@mikkonie
Copy link
Collaborator

mikkonie commented Sep 9, 2024 •
edited
Loading

After doing bihealth/sodar-server#1922, the celery process exits with Error 1 in dev when shutting down.

The Celery server does work otherwise normally in dev, so this may not be a huge issue. Still, I'd like not to get unnecessary errors..

Possibly an incompatibility with the Celery config and the version used?

There is no further info, at least not with the logging level used by default. Dump:

^C
worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)
[2024-09-09 13:58:44,496: INFO/MainProcess] beat: Shutting down...
make: *** [Makefile:50: celery] Error 1
@mikkonie mikkonie added bug Something isn't working environment Issues of dependencies, CI, deployment etc. labels Sep 9, 2024
@mikkonie mikkonie added this to the v1.0.0 milestone Sep 9, 2024
@mikkonie mikkonie self-assigned this Sep 9, 2024
@mikkonie mikkonie mentioned this issue Sep 9, 2024
Closed
@mikkonie
Copy link
Collaborator Author

mikkonie commented Sep 9, 2024

Moving this issue to SODAR Core as the problem can also be observed there.

@mikkonie mikkonie transferred this issue from bihealth/sodar-server Sep 9, 2024
@mikkonie mikkonie changed the title Celery process exits with "Error 1" on shutdown after SODAR Core v1.0 upgrade Celery process exits with status code 1 on shutdown after v1.0 upgrades Sep 9, 2024
@mikkonie mikkonie removed this from the v1.0.0 milestone Sep 9, 2024
@mikkonie mikkonie added the ongoing Ongoing issue, needs observation or pending on other projects label Sep 9, 2024
@mikkonie
Copy link
Collaborator Author

mikkonie commented Sep 9, 2024

No specific error reported with debug log enabled either:

^C
worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)
[2024-09-09 15:26:23,683: DEBUG/MainProcess] | Worker: Closing Hub...
[2024-09-09 15:26:23,683: DEBUG/MainProcess] | Worker: Closing Pool...
[2024-09-09 15:26:23,684: DEBUG/MainProcess] | Worker: Closing Beat...
[2024-09-09 15:26:23,684: DEBUG/MainProcess] | Worker: Closing Consumer...
[2024-09-09 15:26:23,684: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2024-09-09 15:26:23,685: DEBUG/MainProcess] | Consumer: Closing Connection...
[2024-09-09 15:26:23,685: DEBUG/MainProcess] | Consumer: Closing Events...
[2024-09-09 15:26:23,685: DEBUG/MainProcess] | Consumer: Closing Heart...
[2024-09-09 15:26:23,685: DEBUG/MainProcess] | Consumer: Closing Mingle...
[2024-09-09 15:26:23,685: DEBUG/MainProcess] | Consumer: Closing Tasks...
[2024-09-09 15:26:23,686: DEBUG/MainProcess] | Consumer: Closing Control...
[2024-09-09 15:26:23,686: DEBUG/MainProcess] | Consumer: Closing Gossip...
[2024-09-09 15:26:23,686: DEBUG/MainProcess] | Consumer: Closing event loop...
[2024-09-09 15:26:23,686: DEBUG/MainProcess] | Consumer: Stopping event loop...
[2024-09-09 15:26:23,686: DEBUG/MainProcess] | Consumer: Stopping Gossip...
[2024-09-09 15:26:23,693: DEBUG/MainProcess] | Consumer: Stopping Control...
[2024-09-09 15:26:23,699: DEBUG/MainProcess] | Consumer: Stopping Tasks...
[2024-09-09 15:26:23,699: DEBUG/MainProcess] Canceling task consumer...
[2024-09-09 15:26:23,699: DEBUG/MainProcess] | Consumer: Stopping Mingle...
[2024-09-09 15:26:23,700: DEBUG/MainProcess] | Consumer: Stopping Heart...
[2024-09-09 15:26:23,702: DEBUG/MainProcess] | Consumer: Stopping Events...
[2024-09-09 15:26:23,702: DEBUG/MainProcess] | Consumer: Stopping Connection...
[2024-09-09 15:26:23,703: DEBUG/MainProcess] | Worker: Stopping Beat...
[2024-09-09 15:26:23,703: INFO/MainProcess] beat: Shutting down...
[2024-09-09 15:26:23,703: DEBUG/MainProcess] | Worker: Stopping Pool...
[2024-09-09 15:26:24,742: DEBUG/MainProcess] | Worker: Stopping Hub...
[2024-09-09 15:26:24,742: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2024-09-09 15:26:24,742: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2024-09-09 15:26:24,742: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2024-09-09 15:26:24,742: DEBUG/MainProcess] Canceling task consumer...
[2024-09-09 15:26:24,742: DEBUG/MainProcess] Closing consumer channel...
[2024-09-09 15:26:24,742: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2024-09-09 15:26:24,742: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2024-09-09 15:26:24,743: DEBUG/MainProcess] | Consumer: Shutdown Connection...
[2024-09-09 15:26:24,744: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
make: *** [Makefile:36: celery] Error 1

@mikkonie
Copy link
Collaborator Author

mikkonie commented Sep 9, 2024

This was not related to broker_connection_retry. Also, upgrading to celery==5.4.0 did not help.

I tried finding reports of similar behaviour but came up empty.

Since I can observe no problems in actually running the process at the moment, I'm setting this as ongoing and will return to it later if further problems arise. As for now it doesn't seem that serious.

Once I do get back to it (or someone else does), double-checking the config might be a good place to start.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mikkonie mikkonie

Labels
bug Something isn't working environment Issues of dependencies, CI, deployment etc. ongoing Ongoing issue, needs observation or pending on other projects
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mikkonie