devspace dev fails after Error executing command 'touch /.devspace/start': touch: cannot touch '/.devspace/start': No such file or directory

[vpatov]

What happened?

Running devspace dev results in an error and stops, because of

11:47:07 start_dev: initial sync: upstream: exec after initial sync: rpc error: code = Unknown desc = Error executing command 'touch /.devspace/start': touch: cannot touch '/.devspace/start': No such file or directory
11:47:07  => exit status 1
11:47:07 fatal exit status 1

First run

Here is the output of the initial run of devspace dev --debug:

11:46:31 dev:my-app Waiting for pod to become ready...
11:46:31 dev:my-app Start selecting a single container with selector label selector: app=my-app
11:46:42 dev:my-app Printing init container logs of pod my-app-d5f477dc9-llnqd
11:46:42 dev:my-app attempting to connect
11:46:42 dev:my-app connection error/timeout: [Errno 111] Connect call failed ('10.96.45.0', 5432)
11:46:42 dev:my-app attempting to connect
11:46:42 dev:my-app connection error/timeout: [Errno 111] Connect call failed ('10.96.45.0', 5432)
11:46:42 dev:my-app attempting to connect
11:46:42 dev:my-app connection error/timeout: [Errno 111] Connect call failed ('10.96.45.0', 5432)
11:46:42 dev:my-app attempting to connect
11:46:42 dev:my-app connection error/timeout: [Errno 111] Connect call failed ('10.96.45.0', 5432)
11:46:42 dev:my-app attempting to connect
11:46:42 dev:my-app connection error/timeout: [Errno 111] Connect call failed ('10.96.45.0', 5432)
11:46:42 dev:my-app attempting to connect
11:46:42 dev:my-app connection error/timeout: [Errno 111] Connect call failed ('10.96.45.0', 5432)
11:46:42 dev:my-app attempting to connect
11:46:42 dev:my-app connection error/timeout: [Errno 111] Connect call failed ('10.96.45.0', 5432)
11:46:42 dev:my-app attempting to connect
11:46:42 dev:my-app connected
11:46:43 dev:my-app DevSpace is waiting, because Pod my-app-d5f477dc9-llnqd has status: Init:1/3
11:46:53 dev:my-app Pod my-app-d5f477dc9-llnqd: Startup probe failed: Get "http://10.244.1.58:8000/healthz": dial tcp 10.244.1.58:8000: connect: connection refused (Unhealthy)
11:47:01 dev:my-app Selected pod my-app-d5f477dc9-llnqd
11:47:01 dev:my-app sync  Start selecting a single container with selector pod name: my-app-d5f477dc9-llnqd
11:47:02 dev:my-app sync  Starting sync...
11:47:02 dev:my-app sync  Inject devspacehelper...
11:47:04 dev:my-app sync  Start syncing
11:47:04 dev:my-app sync  Sync started on: ./my_app <-> /my-app/my_app
11:47:04 dev:my-app sync  Waiting for initial sync to complete
11:47:04 dev:my-app sync  Initial Sync - Retrieve Initial State
11:47:04 dev:my-app sync  Downstream - Start collecting changes
11:47:05 dev:my-app sync  Helper - Use inotify as watching method in container
11:47:05 dev:my-app sync  Downstream - Done collecting changes
11:47:05 dev:my-app sync  Initial Sync - Done Retrieving Initial State
11:47:05 dev:my-app sync  Initial Sync - Calculate Delta from Remote State
11:47:05 dev:my-app sync  Initial Sync - Done Calculating Delta (Download: 106, Upload: 85)
11:47:05 dev:my-app sync  Downstream - Initial sync completed
11:47:06 dev:my-app sync  Upstream - Start applying 191 changes
11:47:06 dev:my-app sync  Upstream - Handling 106 removes
11:47:06 dev:my-app sync  Upstream - Remove '<redacted>...'
11:47:06 dev:my-app sync  Upstream - Done Handling removes
11:47:06 dev:my-app sync  Start hashing 9 files
11:47:06 dev:my-app sync  Done hashing 9 files
11:47:06 dev:my-app sync  Upstream - Upload File '<redacted>'
11:47:06 dev:my-app sync  Upstream - Upload 76 create change(s) (Uncompressed ~172.77 KB)
11:47:06 dev:my-app sync  Upstream - Done Uploading
11:47:06 dev:my-app sync  Upstream - Successfully processed 182 change(s)
11:47:06 dev:my-app sync  Upstream - Done applying changes
11:47:07 dev:my-app sync  Downstream - Start collecting changes
11:47:07 dev:my-app sync  Downstream - Done collecting changes
11:47:07 dev:my-app sync  Downstream - Start applying 0 changes
11:47:07 dev:my-app sync  Downstream - Done applying changes
11:47:07 dev:my-app sync  Sync Error on /home/vpatov/mycompany/my-app/my_app: upstream: exec after initial sync: rpc error: code = Unknown desc = Error executing command 'touch /.devspace/start': touch: cannot touch '/.devspace/start': No such file or directory
 => exit status 1
11:47:07 dev:my-app sync  Sync stopped
11:47:07 dev:my-app sync  Initial sync completed
11:47:07 dev:my-app Stopped dev my-app
11:47:07 start_dev: initial sync: upstream: exec after initial sync: rpc error: code = Unknown desc = Error executing command 'touch /.devspace/start': touch: cannot touch '/.devspace/start': No such file or directory
11:47:07  => exit status 1
11:47:07 fatal exit status 1

Second run (works)

If I rerun devspace dev --debug a second time, it succeeds and sync works, but it still displays the error:

12:05:19 dev:my-app Waiting for pod to become ready...
12:05:19 dev:my-app Start selecting a single container with selector label selector: app=my-app
12:05:20 dev:my-app Selected pod my-app-d5f477dc9-llnqd
12:05:20 dev:my-app sync  Start selecting a single container with selector pod name: my-app-d5f477dc9-llnqd
12:05:20 dev:my-app sync  Starting sync...
12:05:21 dev:my-app sync  Start syncing
12:05:21 dev:my-app sync  Sync started on: ./my_app <-> /my-app/my_app
12:05:21 dev:my-app sync  Waiting for initial sync to complete
12:05:21 dev:my-app sync  Initial Sync - Retrieve Initial State
12:05:21 dev:my-app sync  Downstream - Start collecting changes
12:05:21 dev:my-app sync  Helper - Use inotify as watching method in container
12:05:21 dev:my-app sync  Downstream - Done collecting changes
12:05:21 dev:my-app sync  Initial Sync - Done Retrieving Initial State
12:05:21 dev:my-app sync  Initial Sync - Calculate Delta from Remote State
12:05:21 dev:my-app sync  Initial Sync - Done Calculating Delta (Download: 0, Upload: 0)
12:05:21 dev:my-app sync  Downstream - Initial sync completed
12:05:21 dev:my-app sync  Upstream - Initial sync completed
12:05:21 dev:my-app sync  Initial sync took: 622.089548ms
12:05:21 dev:my-app sync  Initial sync completed
12:05:21 debug Wait for dev to finish
12:05:22 dev:my-app sync  Sync Error on /home/vpatov/mycompany/my-app/my_app: upstream: exec after initial sync: rpc error: code = Unknown desc = Error executing command 'touch /.devspace/start': touch: cannot touch '/.devspace/start': No such file or directory
 => exit status 1
12:05:22 dev:my-app sync  Sync stopped
12:05:22 dev:my-app sync  Restarting because: upstream: exec after initial sync: rpc error: code = Unknown desc = Error executing command 'touch /.devspace/start': touch: cannot touch '/.devspace/start': No such file or directory
 => exit status 1
12:05:22 dev:my-app sync  Start selecting a single container with selector pod name: my-app-d5f477dc9-llnqd
12:05:22 dev:my-app sync  Helper - Streams are closed
12:05:22 dev:my-app sync  Helper - Streams are closed
12:05:22 dev:my-app sync  Starting sync...
12:05:23 dev:my-app sync  Start syncing
12:05:23 dev:my-app sync  Sync started on: ./my_app <-> /my-app/my_app
12:05:23 dev:my-app sync  Waiting for initial sync to complete
12:05:23 dev:my-app sync  Initial Sync - Retrieve Initial State
12:05:23 dev:my-app sync  Downstream - Start collecting changes
12:05:23 dev:my-app sync  Helper - Use inotify as watching method in container
12:05:23 dev:my-app sync  Downstream - Done collecting changes
12:05:23 dev:my-app sync  Initial Sync - Done Retrieving Initial State
12:05:23 dev:my-app sync  Initial Sync - Calculate Delta from Remote State
12:05:23 dev:my-app sync  Initial Sync - Done Calculating Delta (Download: 0, Upload: 0)
12:05:23 dev:my-app sync  Downstream - Initial sync completed
12:05:23 dev:my-app sync  Upstream - Initial sync completed
12:05:23 dev:my-app sync  Initial sync took: 637.195352ms
12:05:23 dev:my-app sync  Initial sync completed

What did you expect to happen instead?

I expect devspace dev to work the first time around

How can we reproduce the bug? (as minimally and precisely as possible)

My devspace.yaml:

version: v2beta1
name: my-app

# metallb let's us access the cluster's services from our local machine
dependencies:
  metallb:
    path: ./hack/metallb
    namespace: metallb-system

images:
  my-app:
    image: my-app-local-image
    target: production
    buildKit:
      args:
        - "--ssh"
        - "default=${SSH_AUTH_SOCK}"

commands:
  prep:
    description: "Setup the cluster for development. This only needs to be run once per cluster."
    command: |-

      # Update helm repos
      helm repo add bitnami https://charts.bitnami.com/bitnami
      helm repo add cert-manager https://charts.jetstack.io
      helm repo update

      # Install helm dependency apps.
      helm upgrade cert-manager cert-manager/cert-manager --namespace cert-manager --create-namespace --set crds.enabled=true --install

pipelines:
  dev:
    run: |-
      run_dependencies metallb
      kubectl apply -f ./hack/my-app/namespace.yaml
      # Use dev secrets so that we don't need externalsecrets / doppler
      kubectl apply -f ./hack/my-app/dev-secrets.yaml

      build_images my-app
      create_deployments my-app

      SERVICE_IP=$(kubectl get svc my-app-kubernetes-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

      # TODO this is hack. Rather than add the metallb IP to /etc/hosts, we should use cloudflare DNS.
      sudo sed -i '/console\.dev\.mycompany\.com/d' /etc/hosts
      echo "${SERVICE_IP}    console.dev.mycompany.com" | sudo tee -a /etc/hosts

      start_dev my-app

  deploy:
    run: |-

      # TODO figure out how to install metallb as part of devspace run prep, while still using
      # dependency syntax/structure (convenient to be able to invoke it this way).
      # It only needs to happen once per cluster, and it doesn't need to be torn down and rebuilt
      # for every cloud console deploy.
      run_dependencies metallb

      kubectl apply -f ./hack/my-app/namespace.yaml
      # Use dev secrets so that we don't need externalsecrets / doppler
      kubectl apply -f ./hack/my-app/dev-secrets.yaml

      build_images my-app
      create_deployments my-app

      SERVICE_IP=$(kubectl get svc my-app-kubernetes-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

      # TODO this is hack. Rather than add the metallb IP to /etc/hosts, we should use cloudflare DNS.
      sudo sed -i '/console\.dev\.mycompany\.com/d' /etc/hosts
      echo "${SERVICE_IP}    console.dev.mycompany.com" | sudo tee -a /etc/hosts

deployments:
  my-app:
    helm:
      releaseName: my-app
      chart:
        path: ../local-iac/my-app/
      valuesFiles:
        - ../local-iac/my-app/values.yaml
        - ../local-iac/my-app/values-dev-kind.yaml
      values:
        image:
          tag: ${runtime.images.my-app.tag}
          repository: ${runtime.images.my-app.image}

dev:
  my-app:
    labelSelector:
      app: my-app
    namespace: my-app
    container: my-app
    sync:
      - path: ./my_app:/my-app/my_app
        startContainer: true

Local Environment:

• DevSpace Version: DevSpace version : 6.3.14
• Operating System: Ubuntu 22.04.5 LTS
• ARCH of the OS: x86_64
  Kubernetes Cluster:
• Cloud Provider: kind v0.25.0 go1.22.9 linux/amd64
• Docker:

Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:41:00 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978

• Kubernetes Version:

Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0

Anything else we need to know?

We started using devspace very recently, so we are flexible with how we configure things. It's very possible I am simply misusing/misunderstanding the config. Any help would be appreciated, thank you!

[vpatov]

I am able to workaround this issue by adding the following to my dockerfile:

RUN mkdir /.devspace
RUN touch /.devspace/start

This doesn't seem like the "right" thing to do though.

[lizardruss]

Thanks for creating this issue. Does this container run as a non-root user that may not have permissions to create the .devspace directory?

I suspect your Dockerfile workaround works since it's running at build time instead of in the container.

[vpatov]

✗  kubectl exec deploy/my-app -c my-app -it -- whoami 
root

This means the container is running as root, yes?

It seems like the touch command is failing not because of insufficient permissions, but because touch isn't meant to create missing directories on the fly? I get the same error if I try to run the command as root myself, but it works fine if I do mkdir /.devspace first.

---

It seems that the restart helper script creates the directory first, which may explain why things work after running devspace a second time?

devspace/pkg/devspace/build/builder/restart/restart.go

Line 49 in 92765ea

mkdir -p $tmpDir