On SIGKILL to turbo run, tasks continue running (incompatible with Playwright testing)

[bencmbrook]

Verify canary release

• I verified that the issue exists in the latest Turborepo canary release.

Link to code that reproduces this issue

This is purely in bash, please see simple repro steps below

Which canary version will you have in your reproduction?

Turborepo v2.3.4-canary.5

Enviroment information

CLI:
   Version: 2.3.4-canary.5
   Path to executable: /Users/you/my-turborepo/node_modules/.pnpm/[email protected]/node_modules/turbo-darwin-arm64/bin/turbo
   Daemon status: Running
   Package manager: pnpm9

Platform:
   Architecture: aarch64
   Operating system: macos
   WSL: false
   Available memory (MB): 7268
   Available CPU cores: 12

Environment:
   CI: None
   Terminal (TERM): xterm-256color
   Terminal program (TERM_PROGRAM): vscode
   Terminal program version (TERM_PROGRAM_VERSION): 1.96.2
   Shell (SHELL): /bin/zsh
   stdin: false

Expected behavior

Child processes end when the parent receives SIGKILL. The child processes are the tasks turbo runs, such pnpm run dev in the apps/blog workspace

Actual behavior

Child processes remain running when the turbo run process receives SIGKILL

To Reproduce

1. Run turbo run dev in any turborepo to start a dev server listening on port 3000. Such as the codebase from npx create-turbo@canary -e basic
2. In a second terminal window, run ps | grep "turbo run dev"

   $ ps | grep "turbo run dev"
   
   56369 ttys006    0:00.03 node /Users/you/.nvm/versions/node/v20.18.0/bin/turbo run dev
   56370 ttys006    0:00.00 /Users/you/.nvm/versions/node/v20.18.0/lib/node_modules/turbo/node_modules/turbo-darwin-arm64/bin/turbo run dev
   56727 ttys013    0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox turbo run dev

3. Grab the process ID. In my case, the top-level process ID is 56369, but the output will depend on your Node installation.
4. Run kill -9 56369 (replace 56369 with your process ID)
5. Confirm that the dev server is still running by going to http://localhost:3000 or running lsof -i:3000

To clean up the orphaned process, run kill $(lsof -t -i:3000)

Additional context

Playwright sends SIGKILL when it is closed, so using Playwright + Turborepo results in this situation of hanging processes.

// playwright.config.ts

import { defineConfig } from '@playwright/test';

export default defineConfig({
  // Run your local dev server before starting the tests
  webServer: {
    command: 'turbo run dev',
    url: 'http://127.0.0.1:3000'
  },
});

So, running Ctrl + C on the playwright test causes the web server to stay online when using turbo. I can confirm that this is not the case if you simply set the command to next dev (or pnpm run dev aliased to it).

Here are related Playwright issues, but about Docker, which has a similar issue with SIGKILL.

[anthonyshew]

This sounds like a duplicate of #3711. Would you agree?

[chris-olszewski]

Could you please try your reproduction steps, but target the process group instead of the individual process e.g. kill -9 -56369? This is what Playwright does and it should SIGKILL all of the child processes instead of just turbo.

Here are my results when I target the process group instead of just the individual process:

[0 olszewski@chriss-mbp] /Users/olszewski/code/vercel/turborepo $ pstree 65391
-+= 65391 olszewski node /Users/olszewski/.nvm/versions/node/v20.11.1/bin/turbo dev --ui=stream
 \-+- 65394 olszewski /Users/olszewski/.nvm/versions/node/v20.11.1/lib/node_modules/turbo/node_modules/turbo-darwin-arm64/bin/turbo dev --u
   \-+- 65395 olszewski /private/tmp/error-only/node_modules/.pnpm/[email protected]/node_modules/turbo-darwin-arm64/bin/tu
     |-+= 65415 olszewski node /Users/olszewski/.nvm/versions/node/v20.11.1/bin/pnpm run dev
     | \-+- 65452 olszewski node /private/tmp/error-only/apps/web/node_modules/.bin/../next/dist/bin/next dev --turbopack
     |   \--- 65465 olszewski next-server (v15.1.0) 
     \-+= 65416 olszewski node /Users/olszewski/.nvm/versions/node/v20.11.1/bin/pnpm run dev
       \-+- 65447 olszewski node /private/tmp/error-only/apps/docs/node_modules/.bin/../next/dist/bin/next dev --turbopack
         \--- 65466 olszewski next-server (v15.1.0)
# Just killing the `turbo` process
[0 olszewski@chriss-mbp] /Users/olszewski/code/vercel/turborepo $ kill -9 65391
# Port is still open
[0 olszewski@chriss-mbp] /Users/olszewski/code/vercel/turborepo $ lsof -t -i:3000
65465
# Now we use `-pid` to target the entire process group
[0 olszewski@chriss-mbp] /Users/olszewski/code/vercel/turborepo $ pstree 70765
-+= 70765 olszewski node /Users/olszewski/.nvm/versions/node/v20.11.1/bin/turbo dev --ui=stream
 \-+- 70766 olszewski /Users/olszewski/.nvm/versions/node/v20.11.1/lib/node_modules/turbo/node_modules/turbo-darwin-arm64/bin/turbo dev --u
   \-+- 70767 olszewski /private/tmp/error-only/node_modules/.pnpm/[email protected]/node_modules/turbo-darwin-arm64/bin/tu
     |-+= 70787 olszewski node /Users/olszewski/.nvm/versions/node/v20.11.1/bin/pnpm run dev
     | \-+- 70821 olszewski node /private/tmp/error-only/apps/docs/node_modules/.bin/../next/dist/bin/next dev --turbopack
     |   \--- 70837 olszewski next-server (v15.1.0) 
     \-+= 70788 olszewski node /Users/olszewski/.nvm/versions/node/v20.11.1/bin/pnpm run dev
       \-+- 70824 olszewski node /private/tmp/error-only/apps/web/node_modules/.bin/../next/dist/bin/next dev --turbopack
         \--- 70838 olszewski next-server (v15.1.0) 
[0 olszewski@chriss-mbp] /Users/olszewski/code/vercel/turborepo $ kill -9 -70765
[0 olszewski@chriss-mbp] /Users/olszewski/code/vercel/turborepo $ lsof -t -i:3000     
[1 olszewski@chriss-mbp] /Users/olszewski/code/vercel/turborepo $ #Port 3000 is available as child was killed

This is a result of what SIGKILL does to a process:

The SIGKILL signal is used to cause immediate program termination. It cannot be handled or ignored, and is therefore always fatal. It is also not possible to block this signal.
This signal is usually generated only by explicit request. Since it cannot be handled, you should generate it only as a last resort, after first trying a less drastic method such as C-c or SIGTERM.

When turbo gets a SIGKILL directly we do not have any chance to forward it to our children or do any sort of cleanup. We just immediately stop running.

If you could try out the new gracefulShutdown option they added with SIGINT, that would probably better since we can forward that signal to our children and give them a chance to clean up.

[bencmbrook]

@chris-olszewski thanks, I'll try targeting the group in the repro. I'm also wondering if the children are receiving SIGHUP already, but not responding to it.

@anthonyshew re whether this is a dupe of #3711 -- on the surface it's a different problem, but it's a possibility that issue is the root cause of this issue, in which case we could mark as a dupe, but needs more investigation to make that connection.

[chris-olszewski]

@bencmbrook Can you provide a reproduction that uses Playwright in the way you're using it? That would be helpful for figuring out what is happening in your setup.