Custom resource fails to build in yarn workspace with error “TypeScript executable not found”

[acusti]

How did you install the Amplify CLI?

yarn

If applicable, what version of Node.js are you using?

v18.7.0

Amplify CLI Version

10.6.2

What operating system are you using?

macOS 13.1

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No manual changes made

Describe the bug

i have a monorepo using yarn 1.22.19 workspaces in which i have installed amplify. i also have a custom resource as a part of my amplify setup defined via a cdk-stack.ts file. when i try to run amplify push, i get the following error:

✖ There was an error pulling the backend environment main.
🛑 TypeScript executable not found.

this error is generated from the utils/build-custom-resources.ts file: https://github.com/aws-amplify/amplify-cli/blob/dev/packages/amplify-category-custom/src/utils/build-custom-resources.ts#L89-L92, and it occurs because that file is manually attempting to find the tsc executable within the backend/custom/{CustomResourceName}/node_modules/.bin directory:

  const targetDir = path.resolve(path.join(pathManager.getBackendDirPath(), categoryName, resource.resourceName));
  // ...
  const localTscExecutablePath = path.join(targetDir, 'node_modules', '.bin', 'tsc');

so if the tsc executable has been hoisted to the workspace route, that path fails the fs.existsSync(localTscExecutablePath) check and the process throws an error.

Expected behavior

i expected the buildResource util to work even if dependencies have been hoisted.

Reproduction steps

1. create a yarn workspace and a directory to serve as the amplify root (e.g. /web in my example) and run amplify init in that root directory
2. add a custom resource to your amplify install
3. run yarn from the workspace root and confirm that the tsc executable exists in the workspace root’s node_modules/.bin/ directory but not in the custom resource’s node_modules/.bin/ directory
4. run amplify push

Project Identifier

Project Identifier: 4cce2c1871372609297d98c2261bc65e

Log output

2023-01-25T14:49:46.766Z|info : amplify-provider-awscloudformation.aws-cfn.describeStack.cfn.describeStacks([{"StackName":"[***]rmation:us-[***]st-[***]08:stack/[***]ify-[*****]ry-[***]in-[***]29-[*****]kStripe-[*****]CK70/[*****]a0-[***]c7-[***]ec-[***]51-[******]5f"}])
2023-01-25T14:49:46.767Z|info : amplify-provider-awscloudformation.aws-cfn.describeStack.cfn.describeStacks([{"StackName":"[***]rmation:us-[***]st-[***]08:stack/[***]ify-[*****]ry-[***]in-[***]29-[******]ors-[*****]F4S/[****]e30-[***]ef-[***]ec-[***]66-[*****]7ad"}])
2023-01-25T14:49:47.898Z|info : amplify-provider-awscloudformation.aws-s3.s3.getFile([{"Key":"[***]ify-[***]json","Bucket":"[***]ify-[*****]ry-[***]in-[***]29-[*****]nt"}])
2023-01-25T14:49:54.707Z|error : TypeScript executable not found.
MissingOverridesInstallationRequirementsError: TypeScript executable not found.

Additional information

No response

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[josefaidt]

Hey @acusti 👋 thanks for raising this! Do you have the amplify/ directory as part of your monorepo's workspaces?

[acusti]

i do! i originally just had:

["<amplify-root>", "<amplify-root>/amplify/backend", "<amplify-root>/amplify/backend/custom/*", "<amplify-root>/amplify/backend/function/*/src"]

(where <amplify-root> is a placeholder for the actual root directory where i ran amplify init) as the amplify part of my workspaces, but i just tried adding "<amplify-root>/amplify" as an additional workspace and running yarn`, and i get the same thing.

[josefaidt]

Hey @acusti thanks for clarifying! May I ask what is the intended use case for including these in your workspaces? Is it purely to share modules and reduce the node_modules footprint on your machine? Overall I would recommend not including these directories in your workspaces, especially with functions as it may push a symlink to a shared module and will cause issues with subsequent amplify pull attempts (see #1077)

[acusti]

what is the intended use case for including these in your workspaces?

do you mean what is the intended use case for including amplify/custom/* and amplify/function/* in my workspaces? if so, that’s easy. i want to manage my dependencies from a central place (a single yarn.lock file makes it trivial to verify in diffs that i haven’t caused any dependencies to get out of sync with each other) and, most importantly, i don’t want node_modules to be shipped to lambda for my amplify functions. i write my lambdas in typescript and use "amplify:<functionName>" npm run scripts to compile / build each lambda function via esbuild into a plain JS output that has been tree-shaken to only include code that is actually used. i go into more detail on that in this blog post: https://www.acusti.ca/blog/2022/03/30/writing-your-amplify-functions-in-typescript-via-esbuild-and-yarn-workspaces/

[josefaidt]

Hey @acusti thanks for clarifying, and that seems reasonable to me! I really enjoy the writeup as well!

I'll mark this as a bug 🙂 thank you for taking the time to file the PR!!

[acusti]

as a quick follow-up, i used amplify-dev built from my branch (see #11854) and my custom resource built without issue

[Amplifiyer]

@acusti I'm not able to reproduce the issue. I followed your steps, created a yarn workspace with just two directories, amplify app root and the custom resource root like

"workspaces": [
    "app1", "app1/amplify/backend/custom/customResourceb4d43d64"
  ],

Then running yarn install from the root, deletes all the packages from the customResourceb4d43d64/node_modules/<pkgs> but not the bin directory customResourceb4d43d64/node_modules/.bin/. Even if I delete the node_modules directory from the customResourceb4d43d64 and then run yarn install from the root, it still creates the customResourceb4d43d64/node_modules/.bin/ directory and symlinks to the root's node_modules's bin.

(base) ➜  <my yarn ws> tree app1/amplify/backend/custom/customResourceb4d43d64/node_modules/ -a
app1/amplify/backend/custom/customResourceb4d43d64/node_modules/
└── .bin
    ├── tsc -> ../../../../../../../node_modules/typescript/bin/tsc
    └── tsserver -> ../../../../../../../node_modules/typescript/bin/tsserver

2 directories, 2 files

and so amplify push works just fine.

I'm also using yarn 1.22.19. Is there any other configuration that I'm missing?

[josefaidt]

To reproduce, use the following package.json in your project root

{
  "private": true,
  "version": "0.0.1",
  "workspaces": [
    "packages/*",
    "apps/*",
    "amplify/backend/*",
    "amplify/backend/custom/*"
  ],
  "devDependencies": {
    "typescript": "^4.9.5"
  }
}

create your Amplify project and add a custom resource:

amplify init -y
amplify add custom # select cdk

you will notice tsc binary in amplify/backend/custom/<resource-name>/node_modules/.bin is symlinked to the project root. Then we'll want to simulate a fresh clone of the project with:

git init
git clean -fXd

From here, attempt to re-initialize your project

amplify init

Observe it will fail building resources with the error described here

✖ There was an error initializing your environment.
🛑 TypeScript executable not found.

Resolution: Please add it as a dev-dependency in the package.json file for this resource.
Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

Session Identifier: 14bb6d84-96a9-4569-889e-205979032685

[josefaidt]

Hey @acusti this issue appears to be fixed with the release of Amplify CLI 10.7.0! Can you try upgrading and see if this issue is still occurring for you?

As a note, the issue lies with the lack of node_modules on amplify init/amplify pull, which is reproducible in 10.6.2 but is resolved in 10.7.0. We'll keep the PR open as well as I think it is a great addition!

[acusti]

@josefaidt thanks for the update! i just upgraded to the latest amplify CLI, and it did indeed resolve my issue. also, i updated my PR (#11854) based on your suggestion in #11889, which i thought was a good one.

[josefaidt]

Hey @acusti glad to hear that resolved the issue for you! And thank you for updating the PR! We'll close this issue for now, but the team will evaluate the PR 🙂