Rails deployment should merge old and new assets

[rahearn]

Story

As a devops engineer,
I would like to deploy an update that can serve old assets,
So that I can have a true zero-downtime deploy that doesn't break in-flight requests

Inspired by https://kamal-deploy.org as well as some pain points we hit on the covid tests backup deploys

Solution

Tasks

Preview Give feedback

1. Merge assets from deploy n-1 with current via CI cacheing
2. Bundle those assets in cloud.gov deployment (skip compilation in buildpack?)
3. Clean up assets from deploy n-2+ so the app size doesn't get out of control

Acceptance Criteria

• Deploying asset updates allows for the app to serve both new and old digest fingerprints

[rahearn]

Blocked until cloudfoundry/ruby-buildpack#1006 has been merged, released, and in cloud.gov's default buildpacks list.

Not fully blocked, we can still have the default clean behavior even without that PR being released

[rahearn]

Current issues:

• need to deploy at least 1/week to prevent cache eviction, need a fallback
• potential issues in high-turnover assets falling off the 1.hour clean cutoff between plan and apply resulting in failed deploys
• need to add a key to the cache name to force upload when files change (hash .manifest.json?)

[rahearn]

Issues of failed applies due to assets falling off the 1.hour cutoff left as an exercise for the implementing team, and maybe to revisit in the future in gitlab ci