blobfixture: create utility for managing blob fixtures #140368

jeffswenson · 2025-02-03T13:00:16Z

This introduces the blobfixture package. It will be used by the DR team to manage the lifetime of fixtures created for restore roachtests. Currently, fixture generation is a manual process so many of our fixtures date back to 22.2. blobfixture has two features which allow for automated fixture generation:

There is an explicit step to mark a fixture as ready. This prevents a restore roachtest from attempting to use a fixture that is in the process of creation.
It garbage collects old fixtures. The GC is easy to retrofit once we start tracking fixture ready time. GC is superior to using a TTL because we don't throw away fixtures if fixture creation breaks and we can delete fixtures more aggressively than if we depended on TTL, which helps minimize the cost of the fixtures.

Release note: none
Part of: #139159

cockroach-teamcity · 2025-02-03T13:00:27Z

This change is

Previously, backup fixture creation was a manual process. Now, fixture creation will run as a roachtest and use the fixture registry utility added by cockroachdb#140368. Fixture generation was reworked to use import instead of old backup fixtures. This makes it trivial to bootstrap fixtures in new clouds and in new fixture buckets. Release note: none Part of: cockroachdb#139159

msbutler

This is awesome!! Left mostly nits

pkg/roachprod/blobfixture/registry.go

msbutler · 2025-02-03T21:02:25Z

pkg/roachprod/blobfixture/registry.go

+
+func (r *Registry) deleteBlobsMatchingPrefix(prefix string) error {
+	return r.storage.List(context.Background(), prefix, "", func(path string) error {
+		return r.storage.Delete(context.Background(), prefix+path)


i hope this scales!

The list is going to be fast. We may need to parallelize the deletes, but I'm trying to write the simplest possible code on the first pass.

pkg/roachprod/blobfixture/registry_test.go

msbutler · 2025-02-03T21:09:44Z

pkg/roachprod/blobfixture/registry.go

+		// 'nodelocal://' support is a little hacky and is only intended to support
+		// testing fixture generation in local tests.
+		node := uri.Host
+		root := path.Join(os.Getenv("HOME"), "local", node, "data", "extern")


i assume you used this in demo? It may be worth adding a baby unit test to ensure this works.

I ended up removing all support for nodelocal for now. I did use it when testing fixture generation. I think it would be possible to make this work in the future by creating a fixture directory somewhere on the local machine and linking it into the external data directory when the roachprod cluster is created. But I don't want to do that right now.

msbutler · 2025-02-03T21:09:57Z

pkg/cloud/nodelocal/test_nodelocal_storage.go

@@ -49,17 +49,14 @@ func ReplaceNodeLocalForTesting(root string) func() {
 func TestingMakeNodelocalStorage(
 	root string, settings *cluster.Settings, es cloudpb.ExternalStorage,
 ) cloud.ExternalStorage {
-	if !buildutil.CrdbTestBuild {


could you provide more context on this change? It seems unrelated to the rest of this commit.

I removed this from the PR and may do something like this in the future if I add local support.

The motivation for this change is the roachtest and roachprod binaries are not considered test binaries. So calling this would panic.

srosenberg · 2025-02-03T21:16:50Z

pkg/roachprod/blobfixture/registry.go

+// GC deletes fixtures that were leaked or obsolete. See the comment on the
+// fixturesToGc for details about the GC policy.
+func (r *Registry) GC(ctx context.Context, l *logger.Logger) error {
+	// List objects in /roachtest/v25.1/metadata


The comment is perhaps a hint that this package should really be inside roachtest proper. Roachprod is used outside of roachtest, so we want to keep them loosely coupled, when possible.

I don't have strong opinions as to where this goes, but I put it in roachprod because I considered adding a roachprod command that restores fixtures into a roachprod cluster or prints the uri for the latest copy of a fixture.

Ah, in that case it makes sense. I also wondered if there would be a self-contained command (via CLI).

msbutler

I'm very excited for this.

Previously, backup fixture creation was a manual process. Now, fixture creation will run as a roachtest and use the fixture registry utility added by cockroachdb#140368. Fixture generation was reworked to use import instead of old backup fixtures. This makes it trivial to bootstrap fixtures in new clouds and in new fixture buckets. Release note: none Part of: cockroachdb#139159

This introduces the `blobfixture` package. It will be used by the DR team to manage the lifetime of fixtures created for restore roachtests. Currently, fixture generation is a manual process so many of our fixtures date back to 22.2. blobfixture has two features which allow for automated fixture generation: 1. There is an explicit step to mark a fixture as ready. This prevents a restore roachtest from attempting to use a fixture that is in the process of creation. 2. It garbage collects old fixtures. The GC is easy to retrofit once we start tracking fixture ready time. GC is superior to using a TTL because we don't throw away fixtures if fixture creation breaks and we can delete fixtures more aggressively than if we depended on TTL, which helps minimize the cost of the fixtures. Release note: none Part of: 139159

Previously, backup fixture creation was a manual process. Now, fixture creation will run as a roachtest and use the fixture registry utility added by cockroachdb#140368. Fixture generation was reworked to use import instead of old backup fixtures. This makes it trivial to bootstrap fixtures in new clouds and in new fixture buckets. Release note: none Part of: cockroachdb#139159

jeffswenson · 2025-02-10T13:41:43Z

Thanks for the reviews!

bors r+

craig · 2025-02-10T14:54:01Z

Build succeeded:

Previously, backup fixture creation was a manual process. Now, fixture creation will run as a roachtest and use the fixture registry utility added by cockroachdb#140368. Fixture generation was reworked to use import instead of old backup fixtures. This makes it trivial to bootstrap fixtures in new clouds and in new fixture buckets. Release note: none Part of: cockroachdb#139159

jeffswenson requested a review from msbutler February 3, 2025 13:00

jeffswenson requested review from a team as code owners February 3, 2025 13:00

jeffswenson requested review from herkolategan and DarrylWong and removed request for a team February 3, 2025 13:00

jeffswenson force-pushed the jeffswenson-fixture-registr branch from 0905435 to 05449e0 Compare February 3, 2025 13:20

jeffswenson mentioned this pull request Feb 3, 2025

roachtest: update backup fixtures #140369

Open

msbutler reviewed Feb 3, 2025

View reviewed changes

srosenberg reviewed Feb 3, 2025

View reviewed changes

srosenberg requested a review from golgeek February 3, 2025 21:39

jeffswenson force-pushed the jeffswenson-fixture-registr branch from 05449e0 to 6e51494 Compare February 4, 2025 14:22

msbutler approved these changes Feb 4, 2025

View reviewed changes

jeffswenson force-pushed the jeffswenson-fixture-registr branch from 6e51494 to 118501a Compare February 4, 2025 16:13

craig bot merged commit 85ecd9c into cockroachdb:master Feb 10, 2025
22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

blobfixture: create utility for managing blob fixtures #140368

blobfixture: create utility for managing blob fixtures #140368

jeffswenson commented Feb 3, 2025 •

edited

Loading

cockroach-teamcity commented Feb 3, 2025

msbutler left a comment

msbutler Feb 3, 2025

jeffswenson Feb 4, 2025

msbutler Feb 3, 2025

jeffswenson Feb 4, 2025

msbutler Feb 3, 2025

jeffswenson Feb 4, 2025

srosenberg Feb 3, 2025

jeffswenson Feb 3, 2025

srosenberg Feb 3, 2025

msbutler left a comment

jeffswenson commented Feb 10, 2025

craig bot commented Feb 10, 2025

blobfixture: create utility for managing blob fixtures #140368

blobfixture: create utility for managing blob fixtures #140368

Conversation

jeffswenson commented Feb 3, 2025 • edited Loading

cockroach-teamcity commented Feb 3, 2025

msbutler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msbutler left a comment

Choose a reason for hiding this comment

jeffswenson commented Feb 10, 2025

craig bot commented Feb 10, 2025

jeffswenson commented Feb 3, 2025 •

edited

Loading