RESOURCE_EXHAUSTED error when running linalg.matmul

[EmilySillars]

Full console output after running /toolchain/bin/snitch_cluster.vlt build/runtime/samples/pomelo/pomelo:

[fesvr] Wrote 36 bytes of bootrom to 0x1000
[fesvr] Wrote entry point 0x80000000 to bootloader slot 0x1020
[fesvr] Wrote 56 bytes of bootdata to 0x1024
[Tracer] Logging Hart          8 to logs/trace_hart_00000008.dasm
[Tracer] Logging Hart          0 to logs/trace_hart_00000000.dasm
[Tracer] Logging Hart          1 to logs/trace_hart_00000001.dasm
[Tracer] Logging Hart          2 to logs/trace_hart_00000002.dasm
[Tracer] Logging Hart          3 to logs/trace_hart_00000003.dasm
[Tracer] Logging Hart          4 to logs/trace_hart_00000004.dasm
[Tracer] Logging Hart          5 to logs/trace_hart_00000005.dasm
[Tracer] Logging Hart          6 to logs/trace_hart_00000006.dasm
[Tracer] Logging Hart          7 to logs/trace_hart_00000007.dasm
RESOURCE_EXHAUSTED; while invoking native function test_pamplemousse.matmulTiny;
[ 1]   native hal.fence.create:0 -
[ 0]   native test_pamplemousse.matmulTiny:0 -

• main.c file: https://github.com/CAPS-UMU/Quidditch/blob/zigzag/runtime/samples/pomelo/main.c
• MLIR file: https://github.com/CAPS-UMU/Quidditch/blob/zigzag/runtime/samples/pomelo/pamplemousse.mlir
• cmakelists.txt file: https://github.com/CAPS-UMU/Quidditch/blob/zigzag/runtime/samples/pomelo/CMakeLists.txt

More details:

• arith.mulf works

func.func @simple_mul(%arg0: tensor<3x3xf64>, %arg1: tensor<3x3xf64>) -> tensor<3x3xf64> {
  %0 = "arith.mulf"(%arg0, %arg1) : (tensor<3x3xf64>, tensor<3x3xf64>) -> tensor<3x3xf64>
  return %0 : tensor<3x3xf64>
  }

• linalg.add works

func.func @matmulTiny(%lhs: tensor<3x3xf64>, %rhs: tensor<3x3xf64>, %acc: tensor<3x3xf64>) -> tensor<3x3xf64> {
  %result = linalg.add
    ins(%lhs, %rhs: tensor<3x3xf64>, tensor<3x3xf64>)
    outs(%acc: tensor<3x3xf64>)
  -> tensor<3x3xf64>
  return %result: tensor<3x3xf64>
  }

• linalg.matmul causes RESOURCES_EXHAUSTED error

func.func @matmulTiny(%lhs: tensor<3x3xf64>, %rhs: tensor<3x3xf64>, %acc: tensor<3x3xf64>) -> tensor<3x3xf64> {
  %result = linalg.matmul
    ins(%lhs, %rhs: tensor<3x3xf64>, tensor<3x3xf64>)
    outs(%acc: tensor<3x3xf64>)
  -> tensor<3x3xf64>
  return %result: tensor<3x3xf64>
  }

• Matrix sizes 4x4 and 64x64 also fail.

[compor]

Can you point to the branch where all this is set up so one can have a go locally, please?

A few things to try out:

1. Does this behaviour persist when you use the LLVM backend (not the Quidditch backend), i.e., using LLVM in the quidditch_module invocation?
2. trying the build settings suggested here, specially into the iree-configuration CMakeLists.txt for the runtime/.
3. Can we get some more detail from the .dasm traces? The flow is detailed in runtime/tests/CMakeLists.txt and it invokes spike on each trace and then gen_trace.py. Let me know if this is not clear enough.

[EmilySillars]

Yes, here is my fork at the point when I first encountered this error: https://github.com/CAPS-UMU/Quidditch/tree/49b48d2d38c88e89944b44eb858acd341e0c358a

Will respond to points 1-3 soon!

[EmilySillars]

1. Would that meaning rebuilding + running after removing line 1 and un-commenting line 3 of the CMakeLists.txt file here?

2. From the page you linked, are you suggesting using the flags --iree-flow-trace-dispatch-tensors and --trace_execution inside the runtime directory's CMakeLists.txt file? If yes, which line in this file would you add these flags?

3. Is this the same as invoking the executable using ctest? For example, ctest -R pomelo from inside the <build-dir>/runtime folder? When I run this ctest command, the program never terminates:

(venv) [hoppip@inf-155-54-205-141 runtime]$ ctest -R pomelo
Test project /home/hoppip/Quidditch/build/runtime
      Start 38: pomelo-setup
 1/12 Test #38: pomelo-setup .....................   Passed    0.00 sec
      Start 37: pomelo

This is why I started trying to run the executable directly with verilator, and got RESOURCES_EXHAUSTED.
I can see .dasm files inside <build-dir>/runtime/samples/pomelo/logs; are these the trace files you want more info on?

[EmilySillars]

Added a PR to reproduce the behavior here: #144