convolution layer when cudnn_off = False will cause random result

[xiaodao2049]

Hi,every one,
I'm training a net in which some layers are convolution layer, and I set a random seed, but when the
cudnn_off = False the result will be random, and if True, the result is reproduceable.

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Cuda, Bug

[vdantu]

@xiaodao2049 : I think the reason is that there are multiple implementations of convolution when using cundd and cudnn chooses the best algorithm based on your available system resources at runtime. You could try using "MXNET_ENFORCE_DETERMINISM" (Refer this PR for full context) and see if this helps. If not you could turn off autotune by setting "MXNET_CUDNN_AUTOTUNE_DEFAULT" to 0 and see if that helps.

@apeforest , please correct me if any part of my answer is misrepresented :)

@mxnet-label-bot add [question, cudnn]

[vdantu]

@mxnet-label-bot update [question]

[xiaodao2049]

@vdantu thanks for your reply. I have tried both solution you mentioned,MXNET_ENFORCE_DETERMINISM doesn't work, maybe my mx version is too old. Setting "MXNET_CUDNN_AUTOTUNE_DEFAULT" to 0 thrown an std::bad_alloc error. I will upgrade my mxnet and try the first solution again.