cache volumes: volume leaks when node CreateVolumeResponse is lost

[pohly]

Suppose the following happens:

• Controller CreateVolume calls CreateVolume on several nodes for a cache volume
• all nodes create the volume, but one CreateVolumeResponse back to the controller is lost

I suspect we currently leak that volume here:

pmem-csi/pkg/pmem-csi-driver/controllerserver-master.go

Line 212 in ff52c5a

klog.Warningf("failed to create volume name:%s id:%s on %s: %s", node, req.Name, volumeID, err.Error())

It prints a warning (but who will read it?) and then proceeds, leading to an overall success for CreateVolume.

When DeleteVolume is called, it will not be called for the node where the response was lost -> volume leak.

@avalluri: does that sound right?

[avalluri]

The current code treats error response from node controllers as a failure(not just this case, same for all calls to nod controller), and we continue as if no volume was created on that node, hence we do not decrease the cache volume count.

We might look into the error if its transport error. But I am not sure what are the options we have in this case.

[pohly]

We might look into the error if its transport error. But I am not sure what are the options we have in this case.

We would have to assume that a volume might have been created. It's not going to be pretty. One more reason to remove cache volume support.

[pohly]

Fixed by PR #838