OpenCL support

[zjin-lcf]

It is not clear if ethier, for example, supports OpenCL when your library is built with OpenCL. Thanks for your clarification.

nrspre ethier 2

...
Initializing device
[CUDA] mode is not enabled, defaulting to [Serial] mode
active occa mode: Serial

[stgeke]

I am not sure if I understand your question.

[zjin-lcf]

I know you are busy with your library. I hope my reply will make sense to you.

Users can select CUDA, HIP or OpenCL when building your library. So a user types ./makenrs and selects OpenCL only. After the library is built successfully. He goes to the example ' ethier'. Then he types 'nrspre ethier 2'. The message shown above indicates that OCCA is in the serial mode. A user may expect that the mode is OpenCL. Is that right ?

[stgeke]

First compile nekRS and enable all the backends (CUDA, HIP, OPENCL) you're planning to use. Note, the SERIAL (CPU) backend is always enabled. Then select in <case.par> which backend you want to use.

…

On Fri, Feb 19, 2021 at 3:50 PM zjin-lcf ***@***.***> wrote: I know you are busy with your library. I hope my reply will make sense to you. Users can select CUDA, HIP or OpenCL when building your library. So a user types ./makenrs and selects OpenCL only. After the library is built successfully. He goes to the example ' ethier'. Then he types 'nrspre ethier 2'. The message shown above indicates that OCCA is in the serial mode. A user may expect that the mode is OpenCL. Is that right ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#233 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEN2M75MJO6BNXOCRXHA2H3S7Z3DBANCNFSM4X4PJJVQ> .

[zjin-lcf]

Thanks. I list some results here, which might be useful for you.

Somehow, the OpenCL option '-cl-strict-aliasing' is not recognized by the Nvidia OpenCL compiler in SDK 11.0. So I removed it in the source files.

On a P100 GPU, the OpenCL and CUDA results are shown below. The commands are
nrspre ethier 1
nrsmpi ethier 1

The number of list items in runtime statistics are different between OpenCL and CUDA execution. In addition, the elapsed time (x.e06) seems very long.

OpenCL

copying solution to nek
calling nek_userchk ...
         100  0.20000000000000015        3.6318977613725603E-010   6.6530611027273271E-010   1.0822979399295472E-011   1.3985686402906877E-011  L2 err

      100  2.0000E-01 Write checkpoint
       FILE:/home/5zj/mini-app/nekRS-OpenCL/examples/ethier/ethier0.f00002                                                       
 min/max:  -1.0000       1.0000      -1.0000       1.0000      -1.0000       1.0000

      100  2.0000E-01 done :: Write checkpoint
                              file size =     0.37    MB
                              avg data-throughput =    28.1MB/s
                              io-nodes =     1

runtime statistics

  setup                 0.000000e+00 s
  computation           2.528723e+01 s
  makef                 0.000000e+00 s
  velocitySolve         0.000000e+00 s
  pressureSolve         0.000000e+00 s
    preconditioner      0.000000e+00 s
      coarse grid       5.010069e-03 s

elapsedTime: 3.90489e+06 s

CUDA

      100  2.0000E-01 Write checkpoint
       FILE:/home/5zj/mini-app/nekRS/examples/ethier/ethier0.f00002                                                    
 min/max:  -1.0000       1.0000      -1.0000       1.0000      -1.0000       1.0000
 min/max:  0.33743      0.54871     -0.61511E-01  0.15500      0.36826E-01  0.25682
 min/max: -0.11849E-01  0.79325E-02
 min/max:  0.33743      0.54871
 min/max:  0.33743      0.54871

      100  2.0000E-01 done :: Write checkpoint
                              file size =      1.1    MB
                              avg data-throughput =    33.7MB/s
                              io-nodes =     1

runtime statistics

  setup                 3.813165e+02 s
  checkpointing         6.863123e-02 s
  udfExecuteStep        3.790905e-02 s
  computation           6.386870e-01 s
  makef                 2.108112e-02 s
  velocitySolve         1.245573e-01 s
  pressureSolve         2.505355e-01 s
    preconditioner      1.394878e-01 s
      coarse grid       3.377919e-03 s
  makeq                 2.746064e-02 s
    udfSEqnSource       2.576896e-03 s
  scalarSolve           1.591737e-01 s

  residual projection   2.539165e-02 s
  dotp                  8.604947e-02 s

elapsedTime: 3.90469e+06 s

[stgeke]

It looks like the device timers for openCL are broken. Please file an issue.
Indeed elapsedTime looks unreasonable. Not sure what's going wrong here. I cannot reproduce it locally.

[zjin-lcf]

I will try to run your example on other computing platforms. Thanks.

[stgeke]

Can you please reference this in a new issue (there is a GitHub function for that).

[zjin-lcf]

I added a post as a new question in Discussion. Thanks again.