Hb solver #546

kostrzewa · 2022-07-08T08:39:27Z

No description provided.

…al, it's now possible to specify a solver for the heatbath, different from the one used in the MD evolution (computation of the forces). In order to do that I made the following changes to the code: 1. I added `int HB_solver; // solver for the HB only (optional) ` and ` solver_params_t HB_solver_params; // parameters of the HB solver ` to the `monomial` struct in `monomial/monomial.h`. 2. I wrote the declaration `%x HB_MSOLVER` in `read_input.l`. There I also wrote ` {SPC}*HB_Solver{EQL} { solver_caller=YY_START; BEGIN(NDMSOLVER); } ` in `<NDRATMONOMIAL,NDRATCORMONOMIAL,NDCLRATMONOMIAL,NDCLRATCORMONOMIAL,NDDETRATMONOMIAL,NDCLDETRATMONOMIAL>`, which I defined the block `<HB_MSOLVER>`, analogously to `<MSOLVER>`, replacing all the occurrencies of `mnl->solver =` with `mnl->HB_solver =`. ** To do**: do the same with the solver_parameters 3. I added `INVALID_SOLVER` to the `SOLVER_TYPE` enum in `solver/solver_type.h`. It's value is `26`. 4. I added the macro `#define _default_HB_solver_flag 26 // this is INVALID_SOLVER (see solver/solver_types.h)` in `default_input_values.h`. 5. In `monomial/cloverdetratio_monomial.c` I changed the function `cloverdetratio_heatbath()` replacing all the occurrencies of `mnl->solver` and `mnl->solver_params` with `mnl->HB_solver` and `mnl->HB_solver_params` respectively. 6. In `monomial/monomial.c`: - In `add_monomial()`, I added `monomial_list[no_monomials].HB_solver = _default_HB_solver_flag;` in the function `add_monomial()`. - In `init_monomials()`, when the HB solver is invalid, i.e. `(monomial_list[i].HB_solver == INVALID_SOLVER`, I set the `HB_solver` and ` Hb_solver_params` equal to the ones used in the MD evolution. In short: if the HB solver is not specified in the input file, the solver type and its parameters are copied from what was specified for the MD. (note that the `hbfunction` is the same, but now uses the HB solver and its parameters)

`<DETMONOMIAL,CLDETMONOMIAL,CLDETRATMONOMIAL,CLDETRATRWMONOMIAL,NDRATMONOMIAL,NDRATCORMONOMIAL,NDCLRATMONOMIAL,NDCLRATCORMONOMIAL,RATMONOMIAL,RATCORMONOMIAL,CLRATMONOMIAL,CLRATCORMONOMIAL,NDDETRATMONOMIAL,NDCLDETRATMONOMIAL>` which parses, for each monomial `mnl`, the `HB_solver_params` attributes. I did it by copying and pasting the same block for `mnl->solver_params` and preplacing those occurrences with `mnl->HB_solver_params`. Note that the default values don't need to be specified because the default corresponds to the solver used in the MD evolution.

…atbath()` replacing all the occurrencies of `mnl->solver` and `mnl->solver_params` with `mnl->HB_solver` and `mnl->HB_solver_params` respectively.

read_input.l

kostrzewa · 2022-07-11T09:35:56Z

read_input.l

@@ -2446,6 +2498,10 @@ static inline double fltlist_next_token(int * const list_end){
   solver_caller=YY_START;
   BEGIN(NDMSOLVER);
  }
+  {SPC}*HB_Solver{EQL} {
+   solver_caller=YY_START;
+   BEGIN(NDMSOLVER);


are you sure this is correct? Do we need to be able to set the HB solver also for the ND monomial?

kostrzewa · 2022-07-11T09:40:02Z

@simone-romiti opened a pull request to review this, please see the inline comments

kostrzewa · 2022-07-11T09:40:43Z

monomial/monomial.c

@@ -248,6 +249,11 @@ int init_monomials(const int V, const int even_odd_flag) {
        }
      }
      else if(monomial_list[i].type == CLOVERDETRATIO) {


you need the same thing for DETRATIO

`HB_UseExternalInverter, HB_UseSloppyPrecision, HB_UseCompression`

` {SPC}*HB_Solver{EQL} { solver_caller=YY_START; BEGIN(NDMSOLVER); ` We don't need to be able to set the HB solver also for the ND monomial.

… HB_solver also for the DETRATIO monomial.

HB_solver specified only for the cloverdetratio and detratio monomials

- Removed extra spaces in blockS (flex syntax error) modified: read_input.l

Marcogarofalo · 2022-07-13T17:34:36Z

monomial/monomial.c

+        if (monomial_list[i].HB_solver == _default_HB_solver_flag) {
+          monomial_list[i].HB_solver = monomial_list[i].solver;
+        }
+        monomial_list[i].HB_solver_params = monomial_list[i].solver_params;


please check this. As far as I can tell HB_solver_params need always to be initialised , so I moved outside the if guard.

Using the monomial

BeginMonomial CLOVERDETRATIO Timescale = 0 kappa = 0.1400645 2KappaMu = 0.000224103200 # numerator shift rho = 0.0016 # denominator shift, should match CLOVERDET shift rho2 = 0.0128 CSW = 1.74 AcceptancePrecision = 1.e-21 ForcePrecision = 1.e-18 Name = cloverdetratio2light useexternalinverter = quda MaxSolverIterations = 500 solver = mg usesloppyprecision = single # HB_solver = cg # HB_usesloppyprecision = single # HB_UseExternalInverter = quda # HB_MaxSolverIterations = 50000 EndMonomial

I got the same onlinemeas after one trajectory:

commenting out he HB_solver lines

==> HB_solver/onlinemeas.000001 <== 6 1 30 2.754953e-30 3.374063e-31 6 1 31 3.400805e-32 -2.501565e-31 6 1 32 -2.318780e-33 0.000000e+00

keeping only the comment in the line # HB_UseExternalInverter = quda i.e. using the tmLQCD cg

==> HB_solver/onlinemeas.000001_with_HB_CPU_cg <== 6 1 30 2.754159e-30 3.371860e-31 6 1 31 3.399023e-32 -2.501224e-31 6 1 32 -2.242958e-33 0.000000e+00

keeping HB_solver commented out

==> HB_solver_same_input/onlinemeas.000001 <== 6 1 30 2.754809e-30 3.377900e-31 6 1 31 3.415679e-32 -2.501734e-31 6 1 32 -2.284831e-33 0.000000e+00

using 6b31350 ad of course HB_solver commented out

==> add_actions/onlinemeas.000001 <== 6 1 30 2.755184e-30 3.367085e-31 6 1 31 3.416626e-32 -2.500371e-31 6 1 32 -2.259483e-33 0.000000e+00

these these were done in
/qbigwork/garofalo/test_tmLQCD_QUDA/cA211.08.32

well, there should be defaults set for a few members in HB_solver_params as is done for solver_params. One way of doing this might indeed be to simply copy solver_params, but only if this does not lead to things being overwritten.

I think this will overwrite settings. Any defaults should be set in add_monomial, which is called while reading the input file. init_monomials, instead, is called after the input file has already been parsed.

A good way would be to set monomial_list[no_monomials].HB_solver_params = monomial_list[no_monomials].solver_params in add_monomial after the corresponding defaults in solver_params have been set

kostrzewa · 2022-07-13T18:22:21Z

.editorconfig

@@ -0,0 +1,59 @@
+[*]


please delete

kostrzewa · 2022-07-13T18:22:34Z

configure_old

@@ -0,0 +1,10320 @@
+#! /bin/sh


please delete

kostrzewa · 2022-07-13T18:22:58Z

tmLQCD.kdev4

@@ -0,0 +1,4 @@
+[Project]


please delete

kostrzewa · 2022-07-13T18:23:17Z

workspace.code-workspace

@@ -0,0 +1,17 @@
+{


please delete

kostrzewa · 2022-07-13T18:23:55Z

sample-input/sample-hmc-tmcloverdet-GPU.input

@@ -0,0 +1,75 @@
+# this is a sample input file for a single cloverdet


this should not be here, I thought I removed this with the GPU removal commit

sorry I made, some confusion with the directory.

Marcogarofalo · 2022-07-14T10:25:01Z

read_input.l

+  CG {
+    if(myverbose) printf("  HB Solver set to \"%s\" line %d monomial %d\n", yytext, line_of_file, current_monomial);
+    mnl->HB_solver = CG;
+    BEGIN(solver_caller);


Please check this line. We found that BEGIN(solver_caller) is necessary to continue reading the input file correctly but I do not understand the logic.

Indeed. This allows the parser to return to the point it came from (after having parsed the solver string). To be precise, this is what happens in flex:

<CLDETRATMONOMIAL,DETRATMONOMIAL>{ {SPC}*HB_Solver{EQL} { solver_caller=YY_START; BEGIN(HB_MSOLVER); -> jump to HB_MSOLVER parser } [...] --> <HB_MSOLVER>{ CG { if(myverbose) printf(" HB Solver set to \"%s\" line %d monomial %d\n", yytext, line_of_file, current_monomial); mnl->HB_solver = CG; BEGIN(solver_caller); --> jump back } [...] } <-- return to this point {SPC}*HB_MaxSolverIterations{EQL}{DIGIT}+ { sscanf(yytext, " %[a-zA-Z_] = %d", name, &a); mnl->HB_maxiter = a; if(myverbose) printf(" HB_MaxSolverIterations set to %d line %d monomial %d\n", a, line_of_file, current_monomial); } [...]

While looking at this I realised that there are actually a few bugs in the parser. There are instances of name_caller where instead there should be instances of solver_caller...

I'm actually surprised that the parsing of the monomials works correctly when the MG (or mixedbicgstab) is used as a solver... :)

Marcogarofalo · 2022-07-14T17:36:39Z

regarding the performance:

mg heatbath
With the cg in cloverdetratio2light heatbath we move the time to setup the mg in the cloverdetratio3light
With the cg in cloverdetratio2light and cloverdetratio3light heatbath

There is a slight improvement overall but mainly we are moving the time for MG_Preconditioner_Setup between monomials

kostrzewa · 2022-07-14T17:44:55Z

Can you provide some more details on where these numbers were obtained and what is the ensemble in question?

kostrzewa · 2022-07-14T17:53:14Z

The only problematic case is the heatbath of the cloverdetratio2light monomial (at least with our current choice of mass preconditioning). You should be looking at the solve time rather than setup+solve as the setup has to be done anyway (as you say).

The problem is most acute on M100, where the heatbath solve in the cloverdetratio2light takes several hundred seconds (much less on other machines). Also don't forget that the setup is only run once per job (when acceptance is sufficiently high).

See the cA211.08.64 run:

19:51 bkostrze@login02 /m100_work/INF22_lqcd123_0/romiti/runs/cA211.08.64_therm/jobscript/logs 
 $ grep "Time for cloverdetratio_heatbath" log_cA211.08.64_therm_7288769_4294967294.out | grep ratio2
# : Time for cloverdetratio_heatbath 1.062495e+03 s level: 1 proc_id: 0 /HMC/cloverdetratio2light:cloverdetratio_heatbath
# : Time for cloverdetratio_heatbath 6.671943e+02 s level: 1 proc_id: 0 /HMC/cloverdetratio2light:cloverdetratio_heatbath

First call is with the setup, the second call without. On a 32c64 test lattice (or another test lattice), the problem will not be this massive, of course. Similary, on Juwels Booster the MG performs much better, so even if it does 300-400 iterations, the impact on runtime is not too bad (using CG would still be faster though).

Marcogarofalo · 2022-07-14T17:53:56Z

Sorry I mean that test was on the same ensamble of the previous comment

/qbigwork/garofalo/test_tmLQCD_QUDA/cA211.08.32

There is a subdirectory add_actions which is my reference version with mg hb, please look at the last log file.

For the cg hb there is a directory HB_solver and i was quoting the second last log file(cg hb only in cloverdet2light) and the last log file with cg hb also in cloverdet3light.

…ere's a separate lib file now

kostrzewa · 2023-01-11T18:51:02Z

The problem with the github action was due to the github runner moving to Ubuntu 22.04 and there, the MPI libs are distributed separately from the compiler wrapper and utilities.

There's still another problem with the DDalphaAMG build because DDalphaAMG has issues with GCC 11.3 and how it organizes linking. I don't have the time to fix that problem right now.

I'll try to check the documentation tomorrow (not sure if I'll manage) and will pull this in then.

kostrzewa · 2023-01-11T18:52:50Z

I've improved the github actions through "artifact upload" (https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts) which makes the various config.log (and output.data) accessible through the web interface post-build.

…our CI

kostrzewa · 2023-02-04T13:10:00Z

alright, finally fixed our CI ...

kostrzewa · 2023-03-01T12:03:08Z

Damn, there's an issue with the initialisation of HB_maxiter for the "default" case (no HB solver defined).

With

BeginMonomial CLOVERDETRATIO
  Timescale = 2
  kappa =    0.1400086
  2KappaMu = 0.000215613244
  rho = 0.0
  rho2 = 0.0015
  CSW = 1.7112
  AcceptancePrecision =  1.e-20
  ForcePrecision = 1.e-18
  Name = cloverdetratio3light
  solver = cg
  MaxSolverIterations = 100000
  UseExternalInverter = quda
  UseSloppyPrecision = single
EndMonomial

which should thus allow for 100k iterations in the HB, acceptance and derivative steps, I get:

# TM_QUDA: Using single prec. as sloppy!
# TM_QUDA: Called _loadGaugeQuda for gauge_id: 0.000000
# TM_QUDA: Theta boundary conditions will be applied to gauge field
# TM_QUDA: Time for reorder_gauge_toQuda 1.359407e-02 s level: 4 proc_id: 0 /HMC/cloverdetratio3light:cloverdetratio_heatbath/solve_degenerate/invert_eo_degenerate_quda/reorder_gauge_toQuda
# TM_QUDA: Time for loadGaugeQuda 2.442663e-01 s level: 4 proc_id: 0 /HMC/cloverdetratio3light:cloverdetratio_heatbath/solve_degenerate/invert_eo_degenerate_quda/loadGaugeQuda
# TM_QUDA: Using mixed precision CG!
# TM_QUDA: Using EO preconditioning!
# TM_QUDA: Time for loadCloverQuda 1.201410e-01 s level: 4 proc_id: 0 /HMC/cloverdetratio3light:cloverdetratio_heatbath/solve_degenerate/invert_eo_degenerate_quda/loadCloverQuda
# TM_QUDA: mu = 0.000770000000, kappa = 0.140008600000, csw = 1.711200000000
# TM_QUDA: Time for reorder_spinor_eo_toQuda 4.586891e-03 s level: 4 proc_id: 0 /HMC/cloverdetratio3light:cloverdetratio_heatbath/solve_degenerate/invert_eo_degenerate_quda/reorder_spinor_eo_toQuda
# QUDA: WARNING: Exceeded maximum iterations 5000
# QUDA: CG: Convergence at 5000 iterations, L2 relative residual: iterated = 2.350305e-09, true = 2.342661e-09 (requested = 1.000000e-10)

kostrzewa · 2023-03-01T12:07:55Z

@simone-romiti can you find the logic bug? It seems to me that it should work correctly:

during read_input, add_monomial is called and sets the default params
in the remaining read_input for this monomial, MaxSolverIterations (in other words mnl->maxiter) is set
during init_monomials, for DETRATIO and CLOVERDETRATIO, the parameters from the "default" solver are taken over for the HB_solver. In particular, this should include mnl->HB_maxiter = mnl->maxiter

simone-romiti · 2023-03-01T14:11:43Z

The flow I see is the following, let me know it you agree on that:

As you said, for each monomial in the input file, at line 2147 of read_input.l, add_monomial is called and sets the default params (definition at line 59 of monomial.c).
After that, the "flex blocks" with the name aliases of the monomial start. At lines 2174/2175 of read_input.l I see that CLOVERDETRATIO and DETRATIO correspond respectively to CLDETRATMONOMIAL and DETRATMONOMIAL.
To me it seems MaxSolverIterations is initalized for CLDETRATMONOMIAL but not if we have DETRATMONOMIAL. So I think we should add the latter to line 2403 of read_input.l.
Still, we are using CLOVERDETRATIO here. I think the problem may be at lines 257 and 292 of monomial.c, where it should be monomial_list[i].HB_maxiter =monomial_list[i].maxiter;.

kostrzewa · 2023-03-01T15:03:18Z

Thanks for checking, I agree with your conclusions. When I reviewed monomial.c, I missed that no_monomials was used instead of i for setting HB_maxiter.

I also agree that DETRATMONOMIAL should be added to line 2403 in read_input.l.

simone-romiti added 2 commits July 6, 2022 14:53

kostrzewa force-pushed the quda_work_add_actions branch from da55dc9 to bff599c Compare July 8, 2022 08:40

In monomial/detratio_monomial.c I changed the function `detratio_he…

40482d0

…atbath()` replacing all the occurrencies of `mnl->solver` and `mnl->solver_params` with `mnl->HB_solver` and `mnl->HB_solver_params` respectively.

kostrzewa commented Jul 11, 2022

View reviewed changes

read_input.l Outdated Show resolved Hide resolved

kostrzewa commented Jul 11, 2022

View reviewed changes

simone-romiti and others added 8 commits July 12, 2022 09:54

Reply to #546 (comment). I've prepended HB_ to the parsing strings:

c39d63e

`HB_UseExternalInverter, HB_UseSloppyPrecision, HB_UseCompression`

Reply to #546 (comment). I removed the block

327f2d8

` {SPC}*HB_Solver{EQL} { solver_caller=YY_START; BEGIN(NDMSOLVER); ` We don't need to be able to set the HB solver also for the ND monomial.

Reply to #546 (comment). I've added the default initialization of the…

860e5ad

… HB_solver also for the DETRATIO monomial.

modified: read_input.l

2534c6c

HB_solver specified only for the cloverdetratio and detratio monomials

- declaration of DETRATIO

141c9a6

- Removed extra spaces in blockS (flex syntax error) modified: read_input.l

add HB_maxiter

3cf0922

fix reading of HB_solver and HB_MaxSolverIterations

7438daa

always init HB_solver_params

d0d772d

Marcogarofalo reviewed Jul 13, 2022

View reviewed changes