Yarn v2 installation error Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)

[jasonlimantoro]

Describe the bug

Using Yarn v2 workspace (node_modules linker, not PnP), the build will fail if heroku-prebuild is defined in the root package.json

To Reproduce

Steps to reproduce the behavior:

1. Use Yarn v2 workspace
2. Define a heroku-prebuild script in root package.json.
   The script could be as simple as:

   {
       "scripts": {
           "heroku-prebuild": "ls"
       }
   }

3. Deploy it (tested with git push heroku main or dpl --provider=heroku, it doesn't matter)
4. Sufficient to see the error

Versions (please complete the following information):

• Heroku Stack: heroku-20
• Node Version: 14.x
• NPM or Yarn Version: 2.4.0
• Buildpack Version: Latest at the time of writing, Not pinned

Additional context

• If heroku-prebuild is not defined, the installation always works.
• NODE_MODULES_CACHE has been disabled or enabled, the error persists.
• NPM_CONFIG_PRODUCTION and YARN_PRODUCTION has been unset as per the official migration guide, the error still persists
• No other buildpacks interfering
• No output seen on the log for the heroku-prebuild script (suggesting that maybe it wasn't even run)

Log output (trimmed):

-----> Node.js app detected
       
-----> Creating runtime environment
       
       NPM_CONFIG_LOGLEVEL=error
       USE_YARN_CACHE=true
       NODE_VERBOSE=false
       NODE_ENV=production
       NODE_MODULES_CACHE=false
       
-----> Installing binaries
       engines.node (package.json):  unspecified
       engines.npm (package.json):   unspecified (use default)
       engines.yarn (package.json):  unspecified (use default)
       
       Resolving node version 14.x...
       Downloading and installing node 14.16.0...
       Using default npm version: 6.14.11
       Resolving yarn version 1.22.x...
       Downloading and installing yarn (1.22.10)
       Using yarn 2.4.0
       
-----> Restoring cache
       Caching has been disabled because NODE_MODULES_CACHE=false
       
-----> Prebuild
       Running heroku-prebuild (yarn)
       Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)
       
       $ yarn run [--inspect] [--inspect-brk] <scriptName> ...

-----> Build failed
       
       We're sorry this build is failing! You can troubleshoot common issues here:
       https://devcenter.heroku.com/articles/troubleshooting-node-deploys

[mhfs]

having the same issue, any insight into this?

[getlarge]

It seems like when we define heroku-prebuild hook in package.json, yarn run <script> will be called
Since Yarn v2 it seems impossible to call yarn run <script> before installing dependencies and since heroku-prebuild is called before dependencies are installed... you can guess what's next, the error reported by @jasonlimantoro .

[ferm10n]

adding a note for devs in the future with similar headaches...
I kept getting this error even after running yarn install but it turned out to be caused by inconsistent nodeLinker in .yarnrc.yml

it was a sub project, which was a git submodule and a yarn workspace, configured as nodeLinker: node-modules but the worktree .yarnrc.yml was configured for nodeLinker: pnp (default)

fixed by removing nodeLinker: node-modules and running yarn install

[stevegrossi]

I was running into the same issue and came across this workaround: yarnpkg/berry#2701 (comment) It uses a yarn plugin and hook to allow particular scripts to run without the expected node_modules state. For example, we needed a heroku-prebuild script to set up Yarn authentication to a private registry before a yarn install has occurred.

// .yarn/plugins/override-state.js
const ALLOWED_SCRIPTS = ['heroku-prebuild'];

const NODE_MODULES = 'node_modules';
const STATE_FILE = '.yarn-state.yml';

module.exports = {
  name: `plugin-override-state`,
  factory: require => ({
    hooks: {
      async setupScriptEnvironment(project, scriptEnv) {
        if (
          scriptEnv != null &&
          ALLOWED_SCRIPTS.find(
            script => script == scriptEnv.npm_lifecycle_event,
          ) != null
        ) {
          if (project.configuration.get(`nodeLinker`) === `node-modules`) {
            const stateFile = [project.cwd, NODE_MODULES, STATE_FILE].join('/');
            const fs = require('fs');

            if (!fs.existsSync(stateFile)) {
              console.log(
                'Detected command allowed in stateless environment and state file is missing.',
              );
              console.log('Generating empty state file...');

              fs.mkdirSync(require('path').dirname(stateFile), {
                recursive: true,
              });
              fs.appendFileSync(stateFile, '# Autogenerated\n', { flag: 'w+' });
              fs.appendFileSync(stateFile, '__metadata:\n');
              fs.appendFileSync(stateFile, '  version: 1\n');
              fs.appendFileSync(stateFile, '  nmMode: classic\n');
            }
          }
        }
      },
    },
  }),
};

Enabled with

# .yarnrc.yml
plugins:
  - .yarn/plugins/override-state.js

[simkessy]

@ferm10n why is using node-linker an issue here?

[luchillo17]

I had this issue while trying to run Storybook in Backstage repo, fixed it by running the install command directly inside the Storybook folder, not sure why that is the only one that breaks given that we have many plugins that work with just the root install.

[ferm10n]

@ferm10n why is using node-linker an issue here?

@simkessy because there's a conflicting source of truth for how to resolve packages. whether it's through node_modules, or through a pnp file.

[robertmaier]

I still get this error with yarn 4. I do not have a monorepo which could have conflicting module resolving settings.

I'm using prebuild to set up npm credentials using yarn and get the exact same error as @jasonlimantoro

Is there any other workaround?

[colincasey]

@robertmaier without knowing the particulars of how you're setting up the credentials, some options could be:

Use .yarnrc.yml with environment variables

Environment variables can be used in .yarnrc.yml. Setup the private npm registry using the npmRegistries config and supply an auth key or other details using an environment variable that can be set on your app.

Use an inline buildpack

The following steps should work as long as you don't require Node.js to execute your credential setup since this will run before the heroku/nodejs buildpack executes.

1. Add the inline buildpack before the heroku/nodejs buildpack

   heroku buildpacks:add --index 1 heroku-community/inline
   

2. Create a bin/detect file in your app with execute permissions containing the following:

   #!/usr/bin/env bash
   exit 0

3. Create a bin/compile file in your app with execute permissions containing the following:

   #!/usr/bin/env bash
   echo "Add code to setup credentials here"  

Use a custom Yarn plugin

If you do need Node.js to be installed to run your credential setup, then the workaround of adding a custom Yarn plugin as outlined by @stevegrossi here may be more appropriate.