Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: potential deadlock #125

Closed
wants to merge 1 commit into from
Closed

Conversation

buroa
Copy link

@buroa buroa commented Dec 23, 2024

This will fix the potential deadlock in #123

I'm not entirely sure why these excess locks were added. Status() was basically renamed to ready(), and then Setup(), SetupSerially(), Remove(), and Check() have added the additional lock mutexes below the c.ready(), which were not present with the previous Status() code.

You can see them before the change here:

Status()
ready()
Setup()
and so on...

=== RUN   TestLibCNIType020
POTENTIAL DEADLOCK: Recursive locking:
current goroutine 22 lock 0x1400010c100
cni.go:159 go-cni.(*libcni).Networks { c.RLock() } <<<<<
cni.go:158 go-cni.(*libcni).Networks { func (c *libcni) Networks() []*Network { }
cni.go:226 go-cni.(*libcni).attachNetworks { var firstError error }
cni.go:175 go-cni.(*libcni).Setup { result, err := c.attachNetworks(ctx, ns) }
cni_test.go:89 go-cni.TestLibCNIType020 {  }

Previous place where the lock was grabbed (same goroutine)
cni.go:169 go-cni.(*libcni).Setup { c.RLock() } <<<<<
cni.go:168 go-cni.(*libcni).Setup { } }
cni_test.go:89 go-cni.TestLibCNIType020 {  }

FAIL    github.com/containerd/go-cni    0.012s
FAIL

Fixes potential deadlock in containerd#123

Signed-off-by: Steven Kreitzer <[email protected]>
@buroa buroa force-pushed the buroa/fix-deadlock branch from ee0ed6e to a94c4cc Compare December 23, 2024 18:52
smira added a commit to smira/go-cni that referenced this pull request Dec 24, 2024
There were at least two code paths which might acquire `.RLock()`
recursively:

1. `Setup*()` -> `createResult()`
2. `Setup*()` -> `Networks()`.

On its own, it's not a problem, but if `.Load()` is called concurrently,
it might do `.Lock()` in between recursive `.RLock()`s, which would lead
to a deadlock.

See containerd#125

Fix by introducing a contract on the way mutex is acquired.

Add a test facility to verify for such mistakes via build tags:

* `go test -race` or `go test -tags deadlocks` activates deadlock
  detection

Signed-off-by: Andrey Smirnov <[email protected]>
smira added a commit to smira/go-cni that referenced this pull request Dec 24, 2024
There were at least two code paths which might acquire `.RLock()`
recursively:

1. `Setup*()` -> `createResult()`
2. `Setup*()` -> `Networks()`.

On its own, it's not a problem, but if `.Load()` is called concurrently,
it might do `.Lock()` in between recursive `.RLock()`s, which would lead
to a deadlock.

See containerd#125

Fix by introducing a contract on the way mutex is acquired.

Add a test facility to verify for such mistakes via build tags:

* `go test -race` or `go test -tags deadlocks` activates deadlock
  detection

Signed-off-by: Andrey Smirnov <[email protected]>
@@ -183,8 +181,6 @@ func (c *libcni) SetupSerially(ctx context.Context, id string, path string, opts
if err := c.ready(); err != nil {
return nil, err
}
c.RLock()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is totally correct (removing the lock), at least it's not clear if it is, see #126

@buroa
Copy link
Author

buroa commented Dec 24, 2024

Closing in favor of #126

@buroa buroa closed this Dec 24, 2024
smira added a commit to smira/pkgs that referenced this pull request Dec 24, 2024
smira added a commit to smira/pkgs that referenced this pull request Dec 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants