Error in eigenValue: invalid matrix entries!

[spatel81]

using nekRS v23.1.1 (fda7a95)

I'm running a case with about 200K elements on Polaris. Using 2 Nodes. 4 gpus per node (8 ranks in total). poly. order = 7. I see the following error appear:

`============= BUILDING pMG7 ==================
autotuning gs for wordSize=4 nFields=1
local: 1.93e-04s (578.3GB/s)
pack/unpack host + hostBuffer MPI using pw: 1.38e-03s
pack/unpack device + hostBuffer MPI using pw: 5.37e-04s
pack/unpack device + hostBuffer MPI using nbc: 4.87e-04s
pack/unpack device + deviceBuffer MPI using pw: 2.26e-04s
MPI min/max/avg: 2.57e-05s 5.90e-05s 4.26e-05s / avg bi-bw: 29.4GB/s/rank
autotuning gs for wordSize=4 nFields=1
local: 2.50e-04s (589.1GB/s)
pack/unpack host + hostBuffer MPI using pw: 1.62e-03s
pack/unpack device + hostBuffer MPI using pw: 7.33e-04s
pack/unpack device + hostBuffer MPI using nbc: 6.12e-04s
pack/unpack device + deviceBuffer MPI using pw: 2.82e-04s
MPI min/max/avg: 2.83e-05s 6.52e-05s 4.58e-05s / avg bi-bw: 34.4GB/s/rank
autotuning gs for wordSize=4 nFields=1 userKnlOverlap
local: 6.98e-04s
pack/unpack host + hostBuffer MPI using pw: 1.62e-03s
pack/unpack device + hostBuffer MPI using pw: 1.10e-03s
pack/unpack device + hostBuffer MPI using nbc: 9.90e-04s
pack/unpack device + deviceBuffer MPI using pw: 7.17e-04s
MPI min/max/avg: 2.81e-05s 6.63e-05s 4.63e-05s / avg bi-bw: 33.8GB/s/rank
testing overlap in smoothSchwarz: 1.65e-03s 1.66e-03s
estimating maxEigenvalue ... Error in eigenValue: invalid matrix entries!
stack trace:
  /home/sspatel/neup/spatel/nekRS/install/lib/libnekrs.so : pMGLevel::maxEigSmoothAx()+0x1629
  /home/sspatel/neup/spatel/nekRS/install/lib/libnekrs.so : pMGLevel::setupSmoother(elliptic_t*)+0x60d
  /home/sspatel/neup/spatel/nekRS/install/lib/libnekrs.so : pMGLevel::pMGLevel(elliptic_t*, int, setupAide, int, bool)+0x226
  /home/sspatel/neup/spatel/nekRS/install/lib/libnekrs.so : ellipticMultiGridSetup(elliptic_t*, precon_t*)+0x51e
  /home/sspatel/neup/spatel/nekRS/install/lib/libnekrs.so : ellipticPreconditionerSetup(elliptic_t*, ogs_t*)+0x243
  /home/sspatel/neup/spatel/nekRS/install/lib/libnekrs.so : ellipticSolveSetup(elliptic_t*)+0x23a0
  /home/sspatel/neup/spatel/nekRS/install/lib/libnekrs.so : nrsSetup(int, setupAide&, nrs_t*)+0x5fcf`

FWIW, same errors appears for poly order 3 and 5.

I am also seeing some warnings in Lanczos:

parCon (tol = 2.000000e-01) finished in 0.522681 s
Warning: Lanczos reached a residual of 0.000020 (target: 0.000010) after 50 x 50 iterations in Level=0!
Warning: Lanczos reached a residual of 7908.432895 (target: 0.000010) after 50 x 50 iterations in Level=2!
parRSB finished in 4.19131 seconds.

Does this mean there is something wrong with the Mesh? any help is much appreciated.

[yslan]

I suspect the boundary condition is not setup properly. This is similar to Nek5000's Error in generalv

Warning: Lanczos reached a residual of 7908.432895 (target: 0.000010) after 50 x 50 iterations in Level=2!
This is unusual. The residual is super high. However, parRSB should still provide a valid (but poor?) partition

[spatel81]

Thanks @yslan. There are a handful of nek5K subroutines that get called in nRS for the mesh read-in, does it call nek5K's generalv subroutine?

here is the partion info...looks balanced

 nElements   max/min/bal: 25196 25196 1.00
 nMessages   max/min/avg: 6 2 4.00
 msgSize     max/min/avg: 138390 8905 53557.12
 msgSizeSum  max/min/avg: 374114 99780 217239.50

[yslan]

I need to check when I get back to my laptop...
generalev is for setting up the fast diagonalization, which is not called by NekRS. NekRS has its own calls to Eigensolver.