🐛 cache: clone maps to prevent data race when concurrently creating caches using the same options #3078

chrischdi · 2025-01-20T11:04:57Z

This PR adds clones to the cache packages defaultOpts function.
This is necessary to prevent data races when creating caches in parallel using the same options.

Note: we found this in CAPI and workaround using: kubernetes-sigs/cluster-api#11707

Data races before:

==================
WARNING: DATA RACE
Write at 0x00c00046c900 by goroutine 29:
  runtime.mapaccess2()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/runtime/map.go:479 +0x28c
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:523 +0x650
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Previous read at 0x00c00046c900 by goroutine 28:
  runtime.mapdelete()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/runtime/map.go:741 +0x43c
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:472 +0x144
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Goroutine 29 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40

Goroutine 28 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40
==================
==================
WARNING: DATA RACE
Read at 0x00c00046c930 by goroutine 28:
  runtime.mapdelete()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/runtime/map.go:741 +0x43c
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:487 +0x418
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Previous write at 0x00c00046c930 by goroutine 29:
  runtime.mapaccess2_faststr()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/runtime/map_faststr.go:117 +0x42c
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:509 +0x1270
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Goroutine 28 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40

Goroutine 29 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40
==================
==================
WARNING: DATA RACE
Write at 0x00c0004a6b90 by goroutine 29:
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:523 +0x660
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Previous read at 0x00c0004a6b90 by goroutine 28:
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:472 +0x308
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Goroutine 29 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40

Goroutine 28 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40
==================
==================
WARNING: DATA RACE
Read at 0x00c0004c4590 by goroutine 28:
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:487 +0xcf0
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Previous write at 0x00c0004c4590 by goroutine 29:
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:509 +0x1280
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Goroutine 28 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40

Goroutine 29 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40
==================
==================
WARNING: DATA RACE
Read at 0x00c00046c960 by goroutine 28:
  runtime.mapassign_fast64ptr()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/runtime/map_fast64.go:214 +0x36c
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:493 +0xe68
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Previous write at 0x00c00046c960 by goroutine 29:
  runtime.mapaccess2_faststr()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/runtime/map_faststr.go:117 +0x42c
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:539 +0xa4c
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Goroutine 28 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40

Goroutine 29 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40
==================
==================
WARNING: DATA RACE
Read at 0x00c0004c4810 by goroutine 28:
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:493 +0xe78
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Previous write at 0x00c0004c4810 by goroutine 29:
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:539 +0xa5c
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Goroutine 28 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40

Goroutine 29 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40
==================
==================
WARNING: DATA RACE
Write at 0x00c00046c900 by goroutine 28:
  runtime.mapaccess2()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/runtime/map.go:479 +0x28c
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:523 +0x650
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Previous read at 0x00c00046c900 by goroutine 29:
  runtime.mapiterinit()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/runtime/map.go:877 +0x27c
  sigs.k8s.io/controller-runtime/pkg/cache.defaultOpts()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache.go:472 +0x2a8
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace.func1()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:35 +0xe8

Goroutine 28 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40

Goroutine 29 (running) created at:
  sigs.k8s.io/controller-runtime/pkg/cache.TestDefaultOptsRace()
      /Users/schlotterc/go/src/sigs.k8s.io/controller-runtime/pkg/cache/cache_race_test.go:33 +0x4e4
  testing.tRunner()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1690 +0x184
  testing.(*T).Run.gowrap1()
      /Users/schlotterc/.bin/go-archive/go1.23.2.darwin-arm64/src/testing/testing.go:1743 +0x40

Reproduction using the following command and unit test:

go test -race -count=1 -timeout 300s -run ^TestDefaultOptsRace$ sigs.k8s.io/controller-runtime/pkg/cache

File pkg/cache/cache_race_test.go:

package cache

import (
	"sync"
	"testing"

	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/labels"
	"k8s.io/client-go/rest"
	"sigs.k8s.io/controller-runtime/pkg/client"
)

func TestDefaultOptsRace(t *testing.T) {
	opts := Options{
		Mapper: &fakeRESTMapper{},
		ByObject: map[client.Object]ByObject{
			&corev1.Pod{}: {
				Label: labels.SelectorFromSet(map[string]string{"from": "pod"}),
				Namespaces: map[string]Config{"default": {
					LabelSelector: labels.SelectorFromSet(map[string]string{"from": "pod"}),
				}},
			},
		},
		DefaultNamespaces: map[string]Config{"default": {}},
	}

	m := sync.RWMutex{}
	wg := sync.WaitGroup{}
	// Block until we finished the setup
	m.Lock()
	for range 2 {
		wg.Add(1)
		go func() {
			m.RLock()
			defaultOpts(&rest.Config{}, opts)
			wg.Done()
		}()
	}
	m.Unlock()
	wg.Wait()
}

sbueringer · 2025-01-20T13:27:48Z

/cherry-pick release-0.20

k8s-infra-cherrypick-robot · 2025-01-20T13:27:51Z

@sbueringer: once the present PR merges, I will cherry-pick it on top of release-0.20 in a new PR and assign it to you.

In response to this:

/cherry-pick release-0.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

sbueringer · 2025-01-20T13:27:53Z

/cherry-pick release-0.19

k8s-infra-cherrypick-robot · 2025-01-20T13:27:56Z

@sbueringer: once the present PR merges, I will cherry-pick it on top of release-0.19 in a new PR and assign it to you.

In response to this:

/cherry-pick release-0.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

sbueringer

Just a question + the linter finding

sbueringer · 2025-01-20T13:39:07Z

pkg/cache/defaulting_test.go

+
+	// Start go routines which re-use the above options struct.
+	wg := sync.WaitGroup{}
+	for range 2 {


2 is enough to regularly hit the issue with the race detector?

(only if -race is used of course)

sbueringer · 2025-01-20T13:40:15Z

/assign @alvaroaleman

…hes using the same options

k8s-ci-robot · 2025-01-20T14:08:49Z

LGTM label has been added.

Git tree hash: 4209a5d785e31a029ebf7b0ecfb70dba12fdbf61

k8s-ci-robot · 2025-01-20T14:08:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alvaroaleman, chrischdi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alvaroaleman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-infra-cherrypick-robot · 2025-01-20T14:11:13Z

@sbueringer: new pull request created: #3079

In response to this:

/cherry-pick release-0.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-infra-cherrypick-robot · 2025-01-20T14:11:48Z

@sbueringer: new pull request created: #3080

In response to this:

/cherry-pick release-0.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 20, 2025

k8s-ci-robot requested review from inteon and vincepri January 20, 2025 11:05

k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jan 20, 2025

chrischdi mentioned this pull request Jan 20, 2025

Drop map cloning in clustercache kubernetes-sigs/cluster-api#11711

Closed

chrischdi force-pushed the pr-cache-fix-map-data-race branch from a495593 to f5684ee Compare January 20, 2025 13:26

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 20, 2025

sbueringer reviewed Jan 20, 2025

View reviewed changes

k8s-ci-robot assigned alvaroaleman Jan 20, 2025

cache: clone maps to prevent data race when concurrently creating cac…

a1adc6b

…hes using the same options

chrischdi force-pushed the pr-cache-fix-map-data-race branch from f5684ee to a1adc6b Compare January 20, 2025 13:55

alvaroaleman approved these changes Jan 20, 2025

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 20, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 20, 2025

k8s-ci-robot merged commit 5a8edf6 into kubernetes-sigs:main Jan 20, 2025
10 checks passed

k8s-infra-cherrypick-robot mentioned this pull request Jan 20, 2025

[release-0.20] 🐛 cache: clone maps to prevent data race when concurrently creating caches using the same options #3079

Merged

k8s-infra-cherrypick-robot mentioned this pull request Jan 20, 2025

[release-0.19] 🐛 cache: clone maps to prevent data race when concurrently creating caches using the same options #3080

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 cache: clone maps to prevent data race when concurrently creating caches using the same options #3078

🐛 cache: clone maps to prevent data race when concurrently creating caches using the same options #3078

chrischdi commented Jan 20, 2025 •

edited

Loading

sbueringer commented Jan 20, 2025

k8s-infra-cherrypick-robot commented Jan 20, 2025

sbueringer commented Jan 20, 2025

k8s-infra-cherrypick-robot commented Jan 20, 2025

sbueringer left a comment

sbueringer Jan 20, 2025

chrischdi Jan 20, 2025

chrischdi Jan 20, 2025

sbueringer commented Jan 20, 2025

k8s-ci-robot commented Jan 20, 2025

k8s-ci-robot commented Jan 20, 2025

k8s-infra-cherrypick-robot commented Jan 20, 2025

k8s-infra-cherrypick-robot commented Jan 20, 2025

🐛 cache: clone maps to prevent data race when concurrently creating caches using the same options #3078

🐛 cache: clone maps to prevent data race when concurrently creating caches using the same options #3078

Conversation

chrischdi commented Jan 20, 2025 • edited Loading

sbueringer commented Jan 20, 2025

k8s-infra-cherrypick-robot commented Jan 20, 2025

sbueringer commented Jan 20, 2025

k8s-infra-cherrypick-robot commented Jan 20, 2025

sbueringer left a comment

Choose a reason for hiding this comment

sbueringer Jan 20, 2025

Choose a reason for hiding this comment

chrischdi Jan 20, 2025

Choose a reason for hiding this comment

chrischdi Jan 20, 2025

Choose a reason for hiding this comment

sbueringer commented Jan 20, 2025

k8s-ci-robot commented Jan 20, 2025

k8s-ci-robot commented Jan 20, 2025

k8s-infra-cherrypick-robot commented Jan 20, 2025

k8s-infra-cherrypick-robot commented Jan 20, 2025

chrischdi commented Jan 20, 2025 •

edited

Loading