[tests] Added smoke tests for conv solvers (ConvAsmImplicitGemmV4R1DynamicFwd_1x1 and more). Some fixes.

[atamazov]

This is the last PR in series that aims to cover all convolution Solvers with smoke tests.

@junliume Proposed reviewers: @averinevg (mostly to share knowledge), @shurale-nkn. https://github.com/ROCmSoftwarePlatform/MIOpen/labels/testing https://github.com/ROCmSoftwarePlatform/MIOpen/labels/urgency_high

This PR adds smoke tests for conv solvers:

• ConvAsmImplicitGemmV4R1DynamicFwd_1x1
• ConvMPBidirectWinograd_xdlops (with tuning)
• ConvMPBidirectWinograd<M,N>
• ConvOclBwdWrW2 (with tuning)
• ConvOclBwdWrW2NonTunable
• ConvOclDirectFwd (with tuning)
• ConvOclDirectFwd1x1 (with tuning)
• GemmBwd1x1_stride1
• GemmBwd1x1_stride2
• GemmBwdRest
• GemmFwd1x1_0_1
• GemmFwd1x1_0_1_int8
• GemmFwd1x1_0_2
• GemmFwdRest
• GemmWrw1x1_stride1
• GemmWrwUniversal
• ConvHipImplicitGemmBwdXdlops
  • ⚠️ WORKAROUND_ISSUE_2173 introduced.

By products:

• [tuning] Added support for MIOPEN_DEBUG_TUNING_ITERATIONS_MAX to the legacy tuning
• [tests][NFC]
  • Refactored MIOPEN_FIND_ENFORCE=SEARCH_DB_UPDATE MIOPEN_DEBUG_TUNING_ITERATIONS_MAX=5 -> ${ENFORCE_TUNING_5}
  • Reworked W/A from PR 147, see https://github.com/ROCmSoftwarePlatform/MIOpen/pull/147/files#r1056274289
  • Removed duplicate test for ConvBinWinogradRxSf2x3g1
    • resolves https://github.com/ROCmSoftwarePlatform/MIOpen/pull/1906/files#r1131423652
  • Removed duplicate test for ConvBinWinogradRxSf3x2
  • Fix: f16 tests disabled for WinogradRxSf2x3/f3x2 on gfx900
• [NFC] Fix: Added ConvMPBidirectWinograd<2-3> to the list of Winograd solvers.
  • This is NFC so far because Multi-Pass Bidirectional Winograd is disabled by default

TODO

• Add GFX110X_ENABLED where necessary (minimum)
• Add GFX110X_ENABLED for solvers enabled in [GFX11] Add support for gfx1101 target ID #1947
• Add smoke test for ConvHipImplicitGemmBwdXdlops from [CK API] 2d backward data convolution composable kernel integration #1874
• Remove GFX110X_ENABLED for solvers listed in (2) at Deprecate some old static solvers for new ASICs #2080
• Add GFX94X_ENABLED and MIOPEN_DEBUG_ENABLE_DEPRECATED_SOLVERS=1 where necessary, see Deprecate some old static solvers for new ASICs #2080
  • ⚠️ If a solver fails, do not fix, just disable for new GPUs unconditionally (regardless of environment)

[atamazov]

@averinevg @shurale-nkn @johnny-keker @junliume Solver tests updated for MI300. Ready for review & CI testing.

[junliume]

@atamazov another round of CI has just started.

[atamazov]

I think fresh develop needs to be merged once more.

[atamazov]

@junliume @johnny-keker Please start another CI round, thanks!

[atamazov]

@junliume @johnny-keker Can you please send me failing logs? Please also let me know which docker image is currently used on CI. Thanks!

[junliume]

@junliume @johnny-keker Can you please send me failing logs? Please also let me know which docker image is currently used on CI. Thanks!

Hi @atamazov it failed in smoke_solver_ConvHipImplicitGemmBwdXdlops test for the stage Fp32 Hip Debug gfx90a. Here is the error message:

[2023-05-03T19:26:30.939Z] /home/jenkins/workspace/MLLibs_MIOpen_PR-1911/build/bin/test_conv2d --float --cmode conv --pmode default --group-count 1 --disable-forward --disable-backward-weights --input 128 64 7 7 --weights 64 64 3 3 --batch_size 128 --input_channels 64 --output_channels 64 --spatial_dim_elements 7 7 --filter_dims 3 3 --pads_strides_dilations 1 1 1 1 1 1 --trans_output_pads 0 0 --in_layout NHWC --fil_layout NHWC --out_layout NHWC --deterministic 0 --tensor_vect 0 --vector_length 1 --output_type int32 --int8_vectorize 0 
[2023-05-03T19:26:30.939Z] MIOpen(HIP): Info [GetWorkSpaceSize] 0
[2023-05-03T19:26:30.939Z] MIOpen(HIP): Info [FindConvBwdDataAlgorithm] requestAlgoCount = 1, workspace = 0
[2023-05-03T19:26:30.939Z] MIOpen(HIP): Info [Measure] RamDb::Prefetch time: 0.16479 ms
[2023-05-03T19:26:30.939Z] MIOpen(HIP): Info [TryLoad] Find-db regenerating.
[2023-05-03T19:26:30.939Z] MIOpen(HIP): Info [GetPerfDbPathFile] Found exact perf database file
[2023-05-03T19:26:30.939Z] MIOpen(HIP): Info [FindSolutionImpl] ConvHipImplicitGemmBwdXdlops
[2023-05-03T19:26:30.939Z] MIOpen(HIP): Warning [FindSolutionImpl] Perf Db: load skipped: ConvHipImplicitGemmBwdXdlops, enforce: SEARCH_DB_UPDATE(4)
[2023-05-03T19:26:30.939Z] MIOpen(HIP): Info [FindSolutionImpl] Starting search: ConvHipImplicitGemmBwdXdlops, enforce: SEARCH_DB_UPDATE(4)
[2023-05-03T19:26:30.939Z] UndefinedBehaviorSanitizer:DEADLYSIGNAL
[2023-05-03T19:26:30.939Z] ==189537==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address (pc 0x7fda3951ae1e bp 0x7ffe35f2cce0 sp 0x7ffe35f2c990 T189537)
[2023-05-03T19:26:30.939Z] ==189537==The signal is caused by a READ memory access.
[2023-05-03T19:26:30.939Z] ==189537==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
[2023-05-03T19:26:30.939Z]     #0 0x7fda3951ae1e  (/home/jenkins/workspace/MLLibs_MIOpen_PR-1911/build/lib/libMIOpen.so.1+0x1cfc1e1e)

[atamazov]

@junliume Thanks! Which docker image is currently used on CI?

[atamazov]

@junliume wrt #1911 (comment):

• Issue [MI200][MI100][MI300] UB of ConvHipImplicitGemmBwdXdlops #2173 created (I do not have MI200)
• WORKAROUND_ISSUE_2173 introduced.

Merged from the recent develop as well. Can you please start CI again. Thanks!

[atamazov]

• Blocked by [MIGraphX] tuning error message with "Attempt to cast AnyInvokeParams to invalid type" #2239

[atamazov]

@junliume Now this PR is unblocked because (the fix for #2239 is here).

[atamazov]

TODO: Enable W/A for Debug builds only

[junliume]

@atamazov could we revisit this PR? Update it and I can kick off a whole set of CI on it.

[atamazov]

@junliume Sure, as soon as I have time. Quite a lot of work required.