Thruster caches x-sendfile response and gives a 404 on subsequent requests if the file is removed (rails full page caching)

[sodonnel]

I have a very old blog I am trying to bring up to Rails 8 with Thruster.

The app uses Rails full page caching, ie:

caches_page :index, :show

On the posts controller. What I have found is that on the first load of the posts index, everything works great - rails writes the cache file into public/posts/index.html. However I think Thruster then caches the static file on the next load.

Later, when I add a new post, the cache gets swept, removing posts.html from the public folder.

At this stage, if I navigate back to /posts, the rails action does not run as Thruster immediately returns a 404. So Thruster must check if the cached file it has on disk still exists before serving it, and if it is not there, it does not fall back to the Rails action. At this point it is stuck as there is no way to expire the cache and rebuild the index.

Is there any setting I might be missing to make this work out of the box, or some sort of "no cache" header I can set to tell Thruster to not cache this sort of asset?

[sodonnel]

This problem is related to x-sendfile. I enabled debug, and this is the trace where the request gets cached for the first time:

{"time":"2024-11-16T16:51:44.912372513Z","level":"DEBUG","msg":"X-Sendfile sending file","path":"/rails/public/posts.html"}
{"time":"2024-11-16T16:51:44.912768281Z","level":"DEBUG","msg":"Cache: added item","key":6566495710094405687,"size":367,"expires_at":"2025-11-16T22:40:56.912664736Z"}
{"time":"2024-11-16T16:51:44.912808727Z","level":"DEBUG","msg":"Added response to cache","path":"/posts","key":6566495710094405687,"expires":"2025-11-16T22:40:56.912664736Z","size":367}
{"time":"2024-11-16T16:51:44.914714285Z","level":"INFO","msg":"Request","path":"/posts","status":200,"dur":4,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":7807,"resp_content_type":"text/html","remote_addr":"172.17.0.1:40044","user_agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","cache":"miss","query":""}

Notice "size":367 is being cached. However the response is much larger than that - about 8kb compressed, 31kb uncompressed. So I am guessing it is caching the header response with x-sendfile location, but I am not sure on that.

After the cache is expired on the rails side (/rails/public/posts.html is now deleted), on the next request for /post Thruster seems to go back looking for th file again, but its not there. I am not sure if Rails / Rack is giving the 404, or Thruster generates it.

{"time":"2024-11-16T16:54:06.957689567Z","level":"DEBUG","msg":"X-Sendfile sending file","path":"/rails/public/posts.html"}
{"time":"2024-11-16T16:54:06.957746705Z","level":"INFO","msg":"Request","path":"/posts","status":404,"dur":0,"method":"GET","req_content_length":0,"req_content_type":"","resp_content_length":19,"resp_content_type":"text/plain; charset=utf-8","remote_addr":"172.17.0.1:43166","user_agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","cache":"hit","query":""}

Disabling x-sendfile fixes the problem, at least in part (more on that below). What does Thruster cache for a x-sendfile response with a cachable header? It looks like its caching the wrong thing (not the file content, but the header to serve that file), causing it to need to go back to the Rails server to read the file again on every request, but not giving Rails a chance to recreate that file or change the response.

Even with x-sendfile disabled the behavior is a little unexpected. When Rails answers /post, the default header returned specifies "no-cache/private". However once Rails does the full page caching the Static middleware serves the file, same as for assets. There is only a single configurable http header for this, so all static files get served with 1 year cache age. That can be worked around by adding some extra middleware in Rails. This is not a Thruster problem. I am only mentioning it in case someone comes across this in the future.

[sodonnel]

Just notice #27 has reported something similar.