ESBuild architecture mismatch on arm64 mac

[Connorelsea]

Thanks for creating this project! It seems really useful and well-organized. I started digging through it and trying to get it running locally and ran into some issues with ESBuild.

TLDR: arm64 mac and arm64 linux docker image mismatch esbuild dependency architecture, not sure how node_modules are shared

After following the setup instructions, and then trying to run docker-compose up, many of the services start successfully. The API service fails with the following error.

I researched this and many have similar problems in different projects, but the solutions I have found don't work so far. I have tried entering the docker container bash, deleting node modules, and re-running the install in hopes that the correct architecture of esbuild would be installed at that location. But that did not work. I also attempted switching the build process to use yarn and the supportedArchitectures yarnrc feature but this introduced breaking errors about the async iterator symbol TSC problems during docker build that I could not find a fix for.

I am not that familiar with docker so I have been trying to slowly decipher the process that takes place here, and at what points the node_modules are shared and what could be done to prevent that or force reinstall inside the container. Any further insight into the process here or ideas about how to resolve this?

Thanks again! Even if I can't get this working it's giving me a lot of good ideas about how to structure a project like this.

[cerinoligutom]

Thanks for checking this out! Glad to know it's helping people out there 🙂

As for your issue, not quite if this is mac-specific (or localized to your machine) but can you try installing the dependencies on your host machine using the latest version of pnpm w/ Node v20 (use nvm preferably to switch around node versions in case you're not yet on v20) then run docker-compose up. See if anything changes. Just want to confirm if this is a package manager issue.

esbuild isn't on my node_modules so not sure what is going on here.

I'm on Windows and using WSL to develop so mine's running fine but I'm curious about this issue and would like to know what's causing this exactly as I've never encountered it 🤔

[Connorelsea]

I should have mentioned this, but I had originally tried the install on the host with NPM instead of PNPM, but I was using Node 20 and NVM. Just did a clean clone and attempted setup with only PNPM on the host. Got the same output, pasted the full version of it to a gist so it's a bit easier to parse. https://gist.github.com/Connorelsea/4f2d4762c25faf79e40ff16103a24f9e

Edit: Also seems like esbuild is a dependency of tsx but I could be interpreting that incorrectly

[cerinoligutom]

Can you try this:

1. Delete all the lockfiles you've generated so far (e.g. package-lock.json, pnpm-lock.yaml)
2. Install dependencies with pnpm on the host machine (pnpm install)
3. Build with docker-compose using the no-cache flag (docker-compose build --no-cache)

Let me know if you get a different result this time 🤞

[Connorelsea]

Just tested that approach and am getting the same result. It was a fresh git clone of the project so only the pnpm-lock was present. I also tried force deleting the containers in the docker desktop UI and then re-running pnpm install and the build --no-cache and got the same result on that attempt as well.

[cerinoligutom]

I see. The idea was for your machine to generate the lockfile and see if it'd work.

Let's try a different approach and isolate the docker container.

Do these file changes:

docker-compose.yml

Comment the entire api service.

src/config/environment.ts

Change the database (pg and redis) URLs to localhost.

src/config/supertokens.ts

Change the supertokens connection URL to localhost.

---

After you do that:

1. Run docker-compose up in a terminal. (Should run the whole suite but without the api service)
2. Run the app on the host machine (pnpm start)

Let me know if it works.

[Connorelsea]

Thanks so much for the detailed help! That worked and the API is able to run locally with all the services running in the container. Got the following when running the API on host.

Debugger listening on ws://0.0.0.0:9229/3bc11e7d-09d1-4d3d-afa4-9ecec925987a
For help, see: https://nodejs.org/en/docs/inspector
[01:32:36] Finished 'tscCheck' after 1.94 s
[01:32:36] Finished 'lint' after 2.22 s
==============================
NODE_ENV: development
==============================
Server is now up @ 8080

Definitely will be continuing to think about how this could be happening though, will update if I discover anything new.

[cerinoligutom]

Neat! The idea now is that the Node application is running on the host machine so the other cli instructions in the README about getting inside the api container should now be done on the host with this setup.

Anyhow, I might have to rethink developing inside a docker container when esbuild is involved 🤔 The pros of developing inside a container is you don't need to think of the host machine setup since the container should be self-sufficient and you can get into coding right away. Really convenient at work when onboarding someone new into the team. They can just run docker-compose up and can get into hacking right away after the DB setup. Didn't know the new Mac machines have a beef with this particular image (alpine). I don't have a MacBook yet but do you happen to have an M1 or M2 machine?

P.S. You're also correct on your guess that esbuild is a dependency of tsx. You can see that here or directly on the repo's package.json.

[Connorelsea]

I think I may have found a solution! I have the API working now inside the container and no need for host to install dependencies anymore to run the container.

After reading a lot on the topic I came across a strategy to move node_modules up a directory in the container so you can do both local and in-container development with two different sets of node_modules. And ensuring the node_modules being used in the container are being installed there on build rather than being copied over. I have some code for this that I can post momentarily and we can discuss. But it likely needs to be cleaned up since I was doing a lot of testing of different approaches, and make sure there are no dire drawbacks to this approach. I also need to look into the production build process and see if this approach needs to be taken there or if the new dev-time build approach breaks anything about the prod build process.

I found this solution here, listed as solution B, and modified it to fit this use case: https://www.docker.com/blog/keep-nodejs-rockin-in-docker/

New Dockerfile.dev

# https://github.com/nodejs/docker-node/blob/master/docs/BestPractices.md

FROM node:20-alpine

EXPOSE 8080

WORKDIR /usr/src/app

# Global installs

RUN npm install -g pnpm
RUN npm install -g gulp

# Copy

COPY package.json ./
# COPY pnpm-lock.yaml ./
COPY patches ./patches
# RUN pnpm install

### Isolate node modules

# install dependencies first, in a different location for easier app bind mounting for local development
# due to default /opt permissions we have to create the dir with root and change perms
# RUN mkdir /usr/src && chown node:node /usr/src
RUN chown node:node /usr/src
WORKDIR /usr/src
COPY patches ./patches
# the official node image provides an unprivileged user as a security best practice
# but we have to manually enable it. We put it here so npm installs dependencies as the same
# user who runs the app. 
# https://github.com/nodejs/docker-node/blob/master/docs/BestPractices.md#non-root-user
USER node
COPY package.json package-lock.json* ./
# Ensure cache is cleared
RUN rm -rf /usr/src/node_modules
# Prune possibly needed but sometimes breaking? Needs more tests, might not be needed
# RUN pnpm store prune

# Run install
RUN pnpm install
ENV PATH /usr/src/node_modules/.bin:$PATH

# Change back to root

USER root

# Copy files from host to container
WORKDIR /usr/src/app
COPY . .

# Build


RUN pnpm build

RUN ls -al

CMD [ "pnpm", "start" ]

New docker-compose.yml

version: '3'
services:
  api:
    build:
      context: .
      dockerfile: Dockerfile.dev
    user: node
    command: pnpm start
    restart: always
    volumes:
      - ./:/usr/src/app
      # IMPORTANT: If you are using Windows, you might want to uncomment the entry below.
      # https://jdlm.info/articles/2019/09/06/lessons-building-node-app-docker.html#the-node_modules-volume-trick
      # - ./docker-volumes/node_modules:/usr/src/app/node_modules # https://stackoverflow.com/a/32785014
      
      # this is a workaround to prevent host node_modules from accidently getting mounted in container
      # in case you want to use node/npm both outside container for test/lint etc. and also inside container
      # this will overwrite the default node_modules dir in container so it won't conflict with our
      # /opt/node_app/node_modules location. # https://github.com/BretFisher/node-docker-good-defaults/blob/69c923bc646bc96003e9ada55d1ec5ca943a1b19/docker-compose.yml#L30-L34
      - notused:/usr/src/app/node_modules
    depends_on:
      - supertokens
      - db
    ports:
      - '8080:8080'
      - '9229:9229'
    tty: true
    environment:
      POSTGRES_CONNECTION_URL: postgresql://postgres:password@db:5432/db
      REDIS_CONNECTION_URL: redis://redis

  db:
    image: postgres:15-alpine
    user: root
    restart: always
    ports:
      - '5432:5432'
    environment:
      POSTGRES_PASSWORD: password
      POSTGRES_DB: db
    volumes:
      - ./docker-volumes/postgres:/var/lib/postgresql/data

  # https://www.pgadmin.org/docs/pgadmin4/6.5/container_deployment.html
  pgadmin:
    image: dpage/pgadmin4:7.4
    user: root
    restart: always
    ports:
      - '8888:80'
    environment:
      PGADMIN_DEFAULT_EMAIL: [email protected]
      PGADMIN_DEFAULT_PASSWORD: password
    volumes:
      - ./docker-volumes/pgadmin:/var/lib/pgadmin

  redis:
    image: redis:7-alpine
    user: root
    restart: always
    ports:
      - '6379:6379'
    volumes:
      - ./docker-volumes/redis:/data

  redis-commander:
    image: rediscommander/redis-commander:latest
    user: root
    restart: always
    ports:
      - '8889:8081'
    environment:
      - REDIS_HOSTS=local:redis:6379

  supertokens:
    image: supertokens/supertokens-postgresql:6.0
    user: root
    restart: always
    environment:
      # IMPORTANT:
      # Make sure to adjust this accordingly if you plan to test
      # other than the local database
      POSTGRESQL_CONNECTION_URI: 'postgresql://postgres:password@db:5432/db'
      POSTGRESQL_TABLE_SCHEMA: 'supertokens'

      # IMPORTANT:
      # On Production, make sure to set a secure API KEY
      # Read more here: https://supertokens.io/docs/session/common-customizations/core/api-keys
      API_KEYS: 'graphql-starter-supertokens-api-key'
    ports:
      - '3567:3567'

volumes:
  notused:

[cerinoligutom]

Interesting find 🤔 This seems similar to the Windows issue with the note I left there on the docker-compose.yml to isolate the node_modules but that has an extra step of moving the node_modules up a directory.

Have you tried not touching anything on the Docker files (dockerfile and dockerfile.dev) and just uncommenting line 14 on docker-compose.yml?

https://github.com/zeferinix/GraphQL-Starter/blob/d259c881164ef85d7d87b4721d3c551eb8ad4469/docker-compose.yml#L14

It should have the same effect of isolating the node_modules for the container and the host as described on the link in the compose file.

[Connorelsea]

I tried that originally and it didn't work but I don't remember what the result was so I did a few more tests. When I try to first edit the docker-compose.yml to uncomment that link, and then follow setup instructions, I get a weird error about gulp not being available. Adding global gulp install to the dockerfile in this case does not fix that issue so I am not yet sure how to resolve.

The second scenario I attempted was the opposite, running all setup normally, changing the docker-compose.yml, and then re-building. In that case the gulp error does not occur but the esbuild error still does occur. I could try looking into why but it is a bit hard because I can't tell if that changed setting is even taking effect or what exactly the effect should be.

I will continue trying to investigate.

[Connorelsea]

Adding the following to the dockerfile actually does make a difference but only changes the error to a slightly different error but still related to gulp lol.

[cerinoligutom]

Weird. gulp is installed globally on the image so it should be there.

I'll be getting a Mac machine sooner or later so I'll take a closer look on this once that comes but do kindly continue sharing your findings. Will this keep this issue open until I can agree on a solution (or tradeoff worst-case scenario) that works across these platforms.

Just throwing in ideas but maybe one of the other Node image variants (e.g. buster-slim, hydrogen, bookworm) might work? I really only picked alpine due to how lightweight it is + reduces potential attack vectors from a security standpoint since there's no bloat included (or unnecessary dependencies) for the Node application.