feat: restart workers when on source changes (#1013)

* Adds filesystem watcher with tests. * Refactoring. * Formatting. * Formatting. * Switches to absolute path in tests. * Fixes race condition from merge conflict. * Fixes race condition. * Fixes tests. * Fixes markdown lint errors. * Switches back to absolute paths. * Reverts back to relative file paths. * Fixes golangci-lint issues. * Uses github.com/dunglas/go-fswatch instead. * Stops watcher before stopping workers. * Updates docs. * Avoids segfault in tests. * Fixes watcher segmentation violations on shutdown. * Adjusts watcher latencies and tests. * Adds fswatch to dockerfiles * Fixes fswatch in alpine. * Fixes segfault (this time for real). * Allows queueing new reload if file changes while workers are reloading. * Makes tests more consistent. * Prevents the watcher from getting stuck if there is an error in the worker file itself. * Reverts changing the image. * Puts fswatch version into docker-bake.hcl. * Asserts instead of panicking. * Adds notice Co-authored-by: Kévin Dunglas <[email protected]> * Update dev.Dockerfile Co-authored-by: Kévin Dunglas <[email protected]> * Update Dockerfile Co-authored-by: Kévin Dunglas <[email protected]> * Update Dockerfile Co-authored-by: Kévin Dunglas <[email protected]> * Update alpine.Dockerfile Co-authored-by: Kévin Dunglas <[email protected]> * Update alpine.Dockerfile Co-authored-by: Kévin Dunglas <[email protected]> * Update dev-alpine.Dockerfile Co-authored-by: Kévin Dunglas <[email protected]> * Update dev-alpine.Dockerfile Co-authored-by: Kévin Dunglas <[email protected]> * Update dev.Dockerfile Co-authored-by: Kévin Dunglas <[email protected]> * Update docs/config.md Co-authored-by: Kévin Dunglas <[email protected]> * Runs fswatch version. * Removes .json. * Replaces ms with s. * Resets the channel after closing it. * Update watcher_options.go Co-authored-by: Kévin Dunglas <[email protected]> * Update watcher_test.go Co-authored-by: Kévin Dunglas <[email protected]> * Asserts no error instead. * Fixes a race condition where events are fired after frankenphp has stopped. * Updates docs. * Update watcher_options_test.go Co-authored-by: Kévin Dunglas <[email protected]> * Allows queuing events while watchers are reloading. * go fmt * Refactors stopping and draining logic. * Allows extended watcher configuration with dirs, recursion, symlinks, case-sensitivity, latency, monitor types and regex. * Updates docs. * Adds TODOS. * go fmt. * Fixes linting errors. * Also allows wildcards in the longform and adjusts docs. * Adds debug log. * Fixes the watcher short form. * Refactors sessions and options into a struct. * Fixes an overflow in the 'workersReadyWG' on unexpected terminations. * Properly logs errors coming from session.Start(). * go fmt. * Adds --nocache. * Fixes lint issue. * Refactors and resolves race condition on worker reload. * Implements debouncing with a timer as suggested by @withinboredom. * Starts watcher even if no workers are defined. * Updates docs with file limit warning. * Adds watch config unit tests. * Adjusts debounce timings. * go fmt. * Adds fswatch to static builder (test). * Adds a short grace period between stopping and destroying the watcher sessions. * Adds caddy test. * Adjusts sleep time. * Swap to edant/watcher. * Fixes watch options and tests. * go fmt. * Adds TODO. * Installs edant/watcher in the bookworm image. * Fixes linting. * Refactors the watcher into its own module. * Adjusts naming. * ADocker image adjustments and refactoring. * Testing installation methods. * Installs via gcc instead. * Fixes pointer formats. * Fixes lint issues. * Fixes arm alpine and updates docs. * Clang format. * Fixes dirs. * Adds watcher version arg. * Uses static lib version. * Adds watcher to tests and sanitizers. * Uses sudo for copying the shared lib. * Removes unnused func. * Refactoring. * Update .github/workflows/sanitizers.yaml Co-authored-by: Kévin Dunglas <[email protected]> * Adds fpic. * Fixes linting. * Skips tests in msan. * Resets op_cache in every worker thread after termination * Review fixes part 1. * Test: installing libstc++ instead of gcc. * Test: using msan ignorelist. * Test: using msan ignorelist. * Test: using msan ignorelist. * Allows '/**/' for global recursion and '**/' for relative recursion. * Reverts using the ignorelist. * Calls opcache directly. * Adds --watch to php-server command * Properly free CStrings. * Sorts alphabetically and uses curl instead of git. * Labeling and formatting. * Update .github/workflows/sanitizers.yaml Co-authored-by: Kévin Dunglas <[email protected]> * Update .github/workflows/sanitizers.yaml Co-authored-by: Kévin Dunglas <[email protected]> * Update .github/workflows/tests.yaml Co-authored-by: Kévin Dunglas <[email protected]> * Update .github/workflows/tests.yaml Co-authored-by: Kévin Dunglas <[email protected]> * Update caddy/caddy.go Co-authored-by: Kévin Dunglas <[email protected]> * Update docs/config.md Co-authored-by: Kévin Dunglas <[email protected]> * Update frankenphp_with_watcher_test.go Co-authored-by: Kévin Dunglas <[email protected]> * Update watcher/watcher.h Co-authored-by: Kévin Dunglas <[email protected]> * Update frankenphp.c Co-authored-by: Kévin Dunglas <[email protected]> * Update watcher/watcher.go Co-authored-by: Kévin Dunglas <[email protected]> * Update docs/config.md Co-authored-by: Kévin Dunglas <[email protected]> * Update frankenphp_with_watcher_test.go Co-authored-by: Kévin Dunglas <[email protected]> * Update testdata/files/.gitignore Co-authored-by: Kévin Dunglas <[email protected]> * Update watcher/watcher-c.h Co-authored-by: Kévin Dunglas <[email protected]> * Update watcher/watcher.c Co-authored-by: Kévin Dunglas <[email protected]> * Fixes test and Dockerfile. * Fixes Dockerfiles. * Resets go versions. * Replaces unsafe.pointer with uintptr_t * Prevents worker channels from being destroyed on reload. * Minimizes the public api by only passing a []string. * Adds support for directory patterns and multiple '**' globs. * Adjusts label. * go fmt. * go mod tidy. * Fixes merge conflict. * Refactoring and formatting. * Cleans up unused vars and functions. * Allows dirs with a dot. * Makes test nicer. * Add dir tests. * Moves the watch directive inside the worker directive. * Adds debug log on special events. * Removes line about symlinks. * Hints at multiple possible --watch flags. * Adds ./**/*.php as default watch configuration. * Changes error to a warning. * Changes the default to './**/*.{php,yaml,yml,twig,env}' and supports the {bracket} pattern. * Fixes linting. * Fixes merge conflict and adjust values. * Adjusts values. --------- Co-authored-by: a.stecher <[email protected]> Co-authored-by: Kévin Dunglas <[email protected]>
dunglas · Oct 7, 2024 · 8d9b6e7 · 8d9b6e7
1 parent aa585f7
commit 8d9b6e7
Show file tree

Hide file tree

Showing 31 changed files with 1,018 additions and 42 deletions.
diff --git a/.github/workflows/sanitizers.yaml b/.github/workflows/sanitizers.yaml
@@ -23,13 +23,14 @@ jobs:
       matrix:
         sanitizer: ['asan', 'msan']
     env:
-      CFLAGS: -g -O0 -fsanitize=${{ matrix.sanitizer == 'asan' && 'address' || 'memory' }}  -DZEND_TRACK_ARENA_ALLOC
+      CFLAGS: -g -O0 -fsanitize=${{ matrix.sanitizer == 'asan' && 'address' || 'memory' }} -DZEND_TRACK_ARENA_ALLOC
       LDFLAGS: -fsanitize=${{ matrix.sanitizer == 'asan' && 'address' || 'memory' }}
       CC: clang
       CXX: clang++
       USE_ZEND_ALLOC: 0
       LIBRARY_PATH: ${{ github.workspace }}/php/target/lib
       LD_LIBRARY_PATH: ${{ github.workspace }}/php/target/lib
+      EDANT_WATCHER_VERSION: next
     steps:
       -
         name: Remove local PHP
@@ -95,6 +96,20 @@ jobs:
       -
         name: Add PHP to the PATH
         run: echo "$(pwd)/php/target/bin" >> "$GITHUB_PATH"
+      -
+        uses: actions/checkout@v4
+        name: Checkout watcher
+        with:
+          repository: e-dant/watcher
+          ref: ${{ env.EDANT_WATCHER_VERSION }}
+          path: 'edant/watcher'
+      -
+        name: Compile edant/watcher
+        run: |
+          cd edant/watcher/watcher-c/
+          clang -o libwatcher.so ./src/watcher-c.cpp -I ./include -I ../include -std=c++17 -Wall -Wextra -fPIC -shared ${{ matrix.sanitizer == 'msan' && '-fsanitize=memory -fno-omit-frame-pointer -fno-optimize-sibling-calls' || '' }}
+          sudo cp libwatcher.so /usr/local/lib/libwatcher.so
+          sudo ldconfig
       -
         name: Set Set CGO flags
         run: |

diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
@@ -23,6 +23,7 @@ jobs:
     env:
       GOEXPERIMENT: cgocheck2
       GOMAXPROCS: 10
+      EDANT_WATCHER_VERSION: next
     steps:
       -
         uses: actions/checkout@v4
@@ -43,6 +44,20 @@ jobs:
         env:
           phpts: ts
           debug: true
+      -
+        uses: actions/checkout@v4
+        name: Checkout watcher
+        with:
+          repository: e-dant/watcher
+          ref: ${{ env.EDANT_WATCHER_VERSION }}
+          path: 'edant/watcher'
+      -
+        name: Compile edant/watcher
+        run: |
+          cd edant/watcher/watcher-c/
+          gcc -o libwatcher.so ./src/watcher-c.cpp -I ./include -I ../include -std=c++17 -O3 -Wall -Wextra -fPIC -shared
+          sudo cp libwatcher.so /usr/local/lib/libwatcher.so
+          sudo ldconfig
       -
         name: Set CGO flags
         run: |

diff --git a/Dockerfile b/Dockerfile
@@ -85,6 +85,16 @@ COPY --link *.* ./
 COPY --link caddy caddy
 COPY --link internal internal
 COPY --link testdata testdata
+COPY --link watcher watcher
+
+# install edant/watcher (necessary for file watching)
+ARG EDANT_WATCHER_VERSION=next
+WORKDIR /usr/local/src/watcher
+RUN curl -L https://github.com/e-dant/watcher/archive/refs/heads/$EDANT_WATCHER_VERSION.tar.gz | tar xz
+WORKDIR /usr/local/src/watcher/watcher-$EDANT_WATCHER_VERSION/watcher-c
+RUN gcc -o libwatcher.so ./src/watcher-c.cpp -I ./include -I ../include -std=c++17 -O3 -Wall -Wextra -fPIC -shared && \
+	cp libwatcher.so /usr/local/lib/libwatcher.so && \
+	ldconfig /usr/local/lib
 
 # See https://github.com/docker-library/php/blob/master/8.3/bookworm/zts/Dockerfile#L57-L59 for PHP values
 ENV CGO_CFLAGS="-DFRANKENPHP_VERSION=$FRANKENPHP_VERSION $PHP_CFLAGS"
@@ -104,6 +114,13 @@ FROM common AS runner
 
 ENV GODEBUG=cgocheck=0
 
+# copy watcher shared library
+COPY --from=builder /usr/local/lib/libwatcher* /usr/local/lib/
+# fix for the file watcher on arm
+RUN apt-get install -y --no-install-recommends libstdc++6 && \
+	apt-get clean && \
+	ldconfig
+
 COPY --from=builder /usr/local/bin/frankenphp /usr/local/bin/frankenphp
 RUN setcap cap_net_bind_service=+ep /usr/local/bin/frankenphp && \
 	frankenphp version
diff --git a/alpine.Dockerfile b/alpine.Dockerfile
@@ -58,21 +58,24 @@ ENV PATH=/usr/local/go/bin:$PATH
 RUN apk add --no-cache --virtual .build-deps \
 	$PHPIZE_DEPS \
 	argon2-dev \
+	# Needed for the custom Go build
+	bash \
 	brotli-dev \
 	coreutils \
 	curl-dev \
+	# Needed for the custom Go build
+	git \
 	gnu-libiconv-dev \
 	libsodium-dev \
+	# Needed for the file watcher
+	libstdc++ \
 	libxml2-dev \
 	linux-headers \
 	oniguruma-dev \
 	openssl-dev \
 	readline-dev \
 	sqlite-dev \
-	upx \
-	# Needed for the custom Go build
-	git \
-	bash
+	upx
 
 # FIXME: temporary workaround for https://github.com/golang/go/issues/68285
 WORKDIR /
@@ -103,6 +106,16 @@ COPY --link *.* ./
 COPY --link caddy caddy
 COPY --link internal internal
 COPY --link testdata testdata
+COPY --link watcher watcher
+
+# install edant/watcher (necessary for file watching)
+ARG EDANT_WATCHER_VERSION=next
+WORKDIR /usr/local/src/watcher
+RUN curl -L https://github.com/e-dant/watcher/archive/refs/heads/$EDANT_WATCHER_VERSION.tar.gz | tar xz
+WORKDIR /usr/local/src/watcher/watcher-$EDANT_WATCHER_VERSION/watcher-c
+RUN gcc -o libwatcher.so ./src/watcher-c.cpp -I ./include -I ../include -std=c++17 -O3 -Wall -Wextra -fPIC -shared && \
+	cp libwatcher.so /usr/local/lib/libwatcher.so && \
+	ldconfig /usr/local/lib
 
 # See https://github.com/docker-library/php/blob/master/8.3/alpine3.20/zts/Dockerfile#L53-L55
 ENV CGO_CFLAGS="-DFRANKENPHP_VERSION=$FRANKENPHP_VERSION $PHP_CFLAGS"
@@ -122,6 +135,11 @@ FROM common AS runner
 
 ENV GODEBUG=cgocheck=0
 
+# copy watcher shared library (libgcc and libstdc++ are needed for the watcher)
+COPY --from=builder /usr/local/lib/libwatcher* /usr/local/lib/
+RUN apk add --no-cache libstdc++ && \
+	ldconfig /usr/local/lib
+
 COPY --from=builder /usr/local/bin/frankenphp /usr/local/bin/frankenphp
 RUN setcap cap_net_bind_service=+ep /usr/local/bin/frankenphp && \
 	frankenphp version
diff --git a/build-static.sh b/build-static.sh
@@ -142,6 +142,16 @@ if [ "${os}" = "linux" ]; then
 		CGO_LDFLAGS="${CGO_LDFLAGS} -lstdc++"
 	fi
 fi
+
+# install edant/watcher for file watching (static version)
+git clone --branch="${EDANT_WATCHER_VERSION:-next}" https://github.com/e-dant/watcher watcher
+cd watcher/watcher-c
+gcc -c -o libwatcher.o ./src/watcher-c.cpp -I ./include -I ../include -std=c++17 -Wall -Wextra -fPIC
+ar rcs libwatcher.a libwatcher.o
+cp libwatcher.a "../../buildroot/lib/libwatcher.a"
+cd ../../
+CGO_LDFLAGS="${CGO_LDFLAGS} -lstdc++ ${PWD}/buildroot/lib/libwatcher.a"
+
 export CGO_LDFLAGS
 
 LIBPHP_VERSION="$(./buildroot/bin/php-config --version)"

diff --git a/caddy/caddy.go b/caddy/caddy.go
@@ -62,6 +62,8 @@ type workerConfig struct {
 	Num int `json:"num,omitempty"`
 	// Env sets an extra environment variable to the given value. Can be specified more than once for multiple environment variables.
 	Env map[string]string `json:"env,omitempty"`
+	// Directories to watch for file changes
+	Watch []string `json:"watch,omitempty"`
 }
 
 type FrankenPHPApp struct {
@@ -85,7 +87,7 @@ func (f *FrankenPHPApp) Start() error {
 
 	opts := []frankenphp.Option{frankenphp.WithNumThreads(f.NumThreads), frankenphp.WithLogger(logger), frankenphp.WithMetrics(metrics)}
 	for _, w := range f.Workers {
-		opts = append(opts, frankenphp.WithWorkers(repl.ReplaceKnown(w.FileName, ""), w.Num, w.Env))
+		opts = append(opts, frankenphp.WithWorkers(repl.ReplaceKnown(w.FileName, ""), w.Num, w.Env, w.Watch))
 	}
 
 	_, loaded, err := phpInterpreter.LoadOrNew(mainPHPInterpreterKey, func() (caddy.Destructor, error) {
@@ -134,7 +136,6 @@ func (f *FrankenPHPApp) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
 				}
 
 				f.NumThreads = v
-
 			case "worker":
 				wc := workerConfig{}
 				if d.NextArg() {
@@ -178,6 +179,13 @@ func (f *FrankenPHPApp) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
 							wc.Env = make(map[string]string)
 						}
 						wc.Env[args[0]] = args[1]
+					case "watch":
+						if !d.NextArg() {
+							// the default if the watch directory is left empty:
+							wc.Watch = append(wc.Watch, "./**/*.{php,yaml,yml,twig,env}")
+						} else {
+							wc.Watch = append(wc.Watch, d.Val())
+						}
 					}
 
 					if wc.FileName == "" {

diff --git a/caddy/caddy_test.go b/caddy/caddy_test.go
@@ -574,3 +574,31 @@ func TestAutoWorkerConfig(t *testing.T) {
 			"frankenphp_testdata_index_php_ready_workers",
 		))
 }
+
+func TestWorkerWithInactiveWatcher(t *testing.T) {
+	tester := caddytest.NewTester(t)
+	tester.InitServer(`
+		{
+			skip_install_trust
+			admin localhost:2999
+			http_port 9080
+
+			frankenphp {
+				worker {
+					file ../testdata/worker-with-watcher.php
+					num 1
+					watch ./**/*.php
+				}
+			}
+		}
+
+		localhost:9080 {
+			root ../testdata
+			rewrite worker-with-watcher.php
+			php
+		}
+		`, "caddyfile")
+
+	tester.AssertGetResponse("http://localhost:9080", http.StatusOK, "requests:1")
+	tester.AssertGetResponse("http://localhost:9080", http.StatusOK, "requests:2")
+}
diff --git a/caddy/php-server.go b/caddy/php-server.go
@@ -29,7 +29,7 @@ import (
 func init() {
 	caddycmd.RegisterCommand(caddycmd.Command{
 		Name:  "php-server",
-		Usage: "[--domain <example.com>] [--root <path>] [--listen <addr>] [--worker /path/to/worker.php<,nb-workers>] [--access-log] [--debug] [--no-compress] [--mercure]",
+		Usage: "[--domain <example.com>] [--root <path>] [--listen <addr>] [--worker /path/to/worker.php<,nb-workers>] [--watch <paths...>] [--access-log] [--debug] [--no-compress] [--mercure]",
 		Short: "Spins up a production-ready PHP server",
 		Long: `
 A simple but production-ready PHP server. Useful for quick deployments,
@@ -48,6 +48,7 @@ For more advanced use cases, see https://github.com/dunglas/frankenphp/blob/main
 			cmd.Flags().StringP("root", "r", "", "The path to the root of the site")
 			cmd.Flags().StringP("listen", "l", "", "The address to which to bind the listener")
 			cmd.Flags().StringArrayP("worker", "w", []string{}, "Worker script")
+			cmd.Flags().StringArrayP("watch", "", []string{}, "Directory to watch for file changes")
 			cmd.Flags().BoolP("access-log", "a", false, "Enable the access log")
 			cmd.Flags().BoolP("debug", "v", false, "Enable verbose debug logs")
 			cmd.Flags().BoolP("mercure", "m", false, "Enable the built-in Mercure.rocks hub")
@@ -73,6 +74,10 @@ func cmdPHPServer(fs caddycmd.Flags) (int, error) {
 	if err != nil {
 		panic(err)
 	}
+	watch, err := fs.GetStringArray("watch")
+	if err != nil {
+		panic(err)
+	}
 
 	var workersOption []workerConfig
 	if len(workers) != 0 {
@@ -90,6 +95,7 @@ func cmdPHPServer(fs caddycmd.Flags) (int, error) {
 
 			workersOption = append(workersOption, workerConfig{FileName: parts[0], Num: num})
 		}
+		workersOption[0].Watch = watch
 	}
 
 	if frankenphp.EmbeddedAppPath != "" {

diff --git a/dev-alpine.Dockerfile b/dev-alpine.Dockerfile
@@ -15,6 +15,8 @@ ENV PHPIZE_DEPS="\
 	pkgconfig \
 	re2c"
 
+SHELL ["/bin/ash", "-eo", "pipefail", "-c"]
+
 RUN apk add --no-cache \
 	$PHPIZE_DEPS \
 	argon2-dev \
@@ -29,6 +31,9 @@ RUN apk add --no-cache \
 	zlib-dev \
 	bison \
 	nss-tools \
+	# file watcher
+	libstdc++ \
+	linux-headers \
 	# Dev tools \
 	git \
 	clang \
@@ -58,6 +63,15 @@ RUN git clone --branch=PHP-8.3 https://github.com/php/php-src.git . && \
 	echo "opcache.enable=1" >> /usr/local/lib/php.ini && \
 	php --version
 
+# install edant/watcher (necessary for file watching)
+ARG EDANT_WATCHER_VERSION=next
+WORKDIR /usr/local/src/watcher
+RUN git clone --branch=$EDANT_WATCHER_VERSION https://github.com/e-dant/watcher .
+WORKDIR /usr/local/src/watcher/watcher-c
+RUN gcc -o libwatcher.so ./src/watcher-c.cpp -I ./include -I ../include -std=c++17 -O3 -Wall -Wextra -fPIC -shared && \
+	cp libwatcher.so /usr/local/lib/libwatcher.so && \
+	ldconfig /usr/local/lib
+
 WORKDIR /go/src/app
 COPY . .
 

diff --git a/dev.Dockerfile b/dev.Dockerfile
@@ -15,6 +15,8 @@ ENV PHPIZE_DEPS="\
 	pkg-config \
 	re2c"
 
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
 # hadolint ignore=DL3009
 RUN apt-get update && \
 	apt-get -y --no-install-recommends install \
@@ -63,6 +65,15 @@ RUN git clone --branch=PHP-8.3 https://github.com/php/php-src.git . && \
 	echo "opcache.enable=1" >> /usr/local/lib/php.ini && \
 	php --version
 
+# install edant/watcher (necessary for file watching)
+ARG EDANT_WATCHER_VERSION=next
+WORKDIR /usr/local/src/watcher
+RUN git clone --branch=$EDANT_WATCHER_VERSION https://github.com/e-dant/watcher .
+WORKDIR /usr/local/src/watcher/watcher-c
+RUN gcc -o libwatcher.so ./src/watcher-c.cpp -I ./include -I ../include -std=c++17 -O3 -Wall -Wextra -fPIC -shared && \
+	cp libwatcher.so /usr/local/lib/libwatcher.so && \
+	ldconfig /usr/local/lib
+
 WORKDIR /go/src/app
 COPY . .
 

diff --git a/docker-bake.hcl b/docker-bake.hcl
@@ -14,6 +14,10 @@ variable "GO_VERSION" {
     default = "1.22"
 }
 
+variable EDANT_WATCHER_VERSION {
+	default = "next"
+}
+
 variable "SHA" {}
 
 variable "LATEST" {
@@ -115,6 +119,7 @@ target "default" {
     }
     args = {
         FRANKENPHP_VERSION = VERSION
+        EDANT_WATCHER_VERSION = EDANT_WATCHER_VERSION
     }
 }
 
@@ -140,6 +145,7 @@ target "static-builder" {
     }
     args = {
         FRANKENPHP_VERSION = VERSION
+        EDANT_WATCHER_VERSION = EDANT_WATCHER_VERSION
     }
     secret = ["id=github-token,env=GITHUB_TOKEN"]
 }