feat(core): lock graph creation when running in another process

[AgentEnder]

Current Behavior

Running Nx in multiple processes at the same time with the daemon disabled can cripple a system due to excess memory usage when creating the graph. This is due to plugin workers being started per-parent process when there is no daemon. This change enables a file lock to prevent the simultaneous processing, and read from the cache when the first run completes.

Currently, running nx show projects 30 times in parallel looks something like this:

30 processes exited within 37535ms

Expected Behavior

30 processes exited within 6435ms

Test Script

//@ts-check

const { spawn } = require('child_process');

let alive = new Set();

let start = Date.now();
let iterations = 30;

for (let i = 0; i < iterations; i++) {
  const cp = spawn('npx nx show projects', [], {
    shell: true,
    env: {
      ...process.env,
      NX_DAEMON: 'false',
      NX_VERBOSE_LOGGING: 'true',
    },
  });
  alive.add(i);
  //   cp.stdout.on('data', (data) => {
  //     console.log(`stdout [${i}]: ${data}`);
  //   });
  cp.stderr.on('data', (data) => {
    console.error(`stderr [${i}]: ${data}`);
  });
  cp.on('exit', (code) => {
    console.log(`child process ${i} exited with code ${code}`);
    alive.delete(i);
  });
}

const i = setInterval(() => {
  if (alive.size > 0) {
  } else {
    clearInterval(i);
    console.log(
      `${iterations} processes exited within ${Date.now() - start}ms`
    );
  }
}, 1);

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Updated (UTC)
nx-dev	⬜️ Ignored (Inspect)	Visit Preview	Dec 19, 2024 11:22pm

[nx-cloud]

View your CI Pipeline Execution ↗ for commit 41b174c.

Command	Status	Duration	Result
nx affected --targets=lint,test,build,e2e,e2e-c...	❌ Failed	36m 30s	View ↗
nx run-many -t check-imports check-commit check...	✅ Succeeded	59s	View ↗
nx-cloud record -- nx-cloud conformance:check	✅ Succeeded	1s	View ↗
nx-cloud record -- nx format:check --base=16c8b...	✅ Succeeded	18s	View ↗
nx-cloud record -- nx sync:check	✅ Succeeded	16s	View ↗
nx documentation --no-dte	✅ Succeeded	4m 9s	View ↗

---

☁️ Nx Cloud last updated this comment at 2024-12-20 00:05:25 UTC

[FrozenPandaz]

This should be in a finally at least

[FrozenPandaz]

Write this in rust

[FrozenPandaz]

Use polling

[FrozenPandaz]

We might not need this

[FrozenPandaz]

This should throw an error if the graph is in a bad state