Leaving out terms in localization matrix?

[wkitlasten]

Hi all, I seem to recall if a par:obs (pargp:obgnme) pair is left out of a localization matrix it will ultimately be treated as 1. For example, if the localization matrix provided only specifies obs0 is not influenced by par1 and obs1 is not influenced by par0 (below), all other obs will be influenced by all pars? Plus, obs0 and obs1 will be influenced by all other pars?

        par0  par1
obs0       1     0
obs1       0     1

So the above would essentially be equivalent to(?):

        par0  par1  par2  par3
obs0       1     0     1     1
obs1       0     1     1     1
obs2       1     1     1     1
obs3       1     1     1     1

My question arises because the parameter upgrades using the "incomplete" localization matrix seem to be significantly faster than those with the "full" (nobs:npar) localization matrix... so just wondering if I misunderstood this, or messed something else up somewhere.

[jtwhite79]

The incomplete localizer is pretty new (and therefore dangerous!). my thinking was to be conservative, any par-obs (group) pairs not included in the localizer should zero - I think this explains why your solves are faster with an incomplete localizer? Does that jive with what you are seeing?

[jtwhite79]

I took a quick look this morning and it does appear that parameters (or par groups) that are left out of the localizer are treated as fully localized, and therefore are implicitly "fixed" and not adjusted. I just added some messaging to the rec file to warn of this condition...Can I assume you are using the latest dev build of ies?

[wkitlasten]

Never safe to assume I am goind anything I should be doing. I am using:

version: 5.2.1
binary compiled on Feb  7 2023 at 19:28:10

I ran the same problem with 2 different localizers: waq_14_s1_200_loc.jcb and waq_14_s1_200_full_loc.jcb. Partial was created with my faulty assumption that terms would be 1 unless specifically set to 0. Full has every pargp:obgnme combo. Both runs take the same amount of time, so my original impression that the partial was faster is somewhat moot (could have been computer resources, how I built the localizer 1 term at a time vs a new trick I learned, etc).

But isolating just the change in localizer still still has me wondering whether terms left out are treated at 0 or1. The parameter change summary from the two runs look almost identical (some very small changes in later iteration, likely due redrawing the noise+obs ensemble is my best guess). My concern now is that pp0_grainmax_layer13 has the highest mean change but is not in the partial localizer:

Partial from the rec:

ies_localizer: waq_14_s1_200_loc.jcb

...

   ---  parameter group change summmary  ---    
   (compared to the initial ensemble using active realizations)
group                          mean change  std change  n at/near bnds  % at/near bnds   init CV   curr CV n CV decr
pp0_grainmax_layer13                286104     0.72352             966               0   4.6e+05   1.1e+02       100
tritium_bcage.sfr_1                1.1e+05         6.5            1621               0   9.7e+03        46      2142
head_measured_151_21               5.5e+04         1.4               1               0   4.7e+03        27         2

Full:

ies_localizer: waq_14_s1_200_full_loc.jcb

...

   ---  parameter group change summmary  ---    
   (compared to the initial ensemble using active realizations)
group                          mean change  std change  n at/near bnds  % at/near bnds   init CV   curr CV n CV decr
pp0_grainmax_layer13                286104     0.72352             966               0   4.6e+05   1.1e+02       100
tritium_bcage.sfr_1                1.1e+05         6.5            1621               0   9.7e+03        46      2142
head_measured_151_21               5.5e+04         1.4               1               0   4.7e+03        27         2

Partial from pyemu/pandas:

>>> ploc=pyemu.Matrix.from_binary(os.path.join('waq_14_s1_200_ies','waq_14_s1_200_loc.jcb')).to_dataframe()
>>> ploc.loc[:,'pp0_grainmax_layer13']
Traceback (most recent call last):
  File "C:\Users\wesk\Miniconda3\envs\py37\lib\site-packages\pandas\core\indexes\base.py", line 3361, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas\_libs\index.pyx", line 76, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pp0_grainmax_layer13'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\wesk\Miniconda3\envs\py37\lib\site-packages\pandas\core\indexing.py", line 925, in __getitem__
    return self._getitem_tuple(key)
  File "C:\Users\wesk\Miniconda3\envs\py37\lib\site-packages\pandas\core\indexing.py", line 1100, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "C:\Users\wesk\Miniconda3\envs\py37\lib\site-packages\pandas\core\indexing.py", line 838, in _getitem_lowerdim
    section = self._getitem_axis(key, axis=i)
  File "C:\Users\wesk\Miniconda3\envs\py37\lib\site-packages\pandas\core\indexing.py", line 1164, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "C:\Users\wesk\Miniconda3\envs\py37\lib\site-packages\pandas\core\indexing.py", line 1113, in _get_label
    return self.obj.xs(label, axis=axis)
  File "C:\Users\wesk\Miniconda3\envs\py37\lib\site-packages\pandas\core\generic.py", line 3761, in xs
    return self[key]
  File "C:\Users\wesk\Miniconda3\envs\py37\lib\site-packages\pandas\core\frame.py", line 3458, in __getitem__
    indexer = self.columns.get_loc(key)
  File "C:\Users\wesk\Miniconda3\envs\py37\lib\site-packages\pandas\core\indexes\base.py", line 3363, in get_loc
    raise KeyError(key) from err
KeyError: 'pp0_grainmax_layer13'

Full:

>>> floc=pyemu.Matrix.from_binary(os.path.join('waq_14_s1_200_ies','waq_14_s1_200_full_loc.jcb')).to_dataframe()
>>> floc.loc[:,'pp0_grainmax_layer13']
tritium_measured_99_nan    1.0
tritium_measured_49_nan    1.0
tritium_measured_61_nan    1.0
tritium_measured_90_nan    1.0
tritium_measured_16_nan    1.0
                          ...
head_measured_1513_48      1.0
head_measured_382_13       1.0
head_measured_1512_1       1.0
less_flw                   1.0
flow                       1.0
Name: pp0_grainmax_layer13, Length: 1629, dtype: float64

pp0_grainmax_layer13 shouldn't change if it isn't in the localizer? But it seems to be changing with the partial localizer?

Sorry I can't dig into a working example until next week. I can send you the ensembles (each 1GB binaries) and the other files, but I prefer to check my understanding before asking you to wade through my junk!

Cheers

[jtwhite79]

hmm. what other advanced options are you using? multimodal? autoadaloc? Im wondering if there is an interaction between other options that might be causing this...

[wkitlasten]

Here is one of the control files (for the partial). Maybe the "ies_localizer_forgive_extra" is the culprit?

pcf version=2

• control data keyword
  pestmode estimation
  noptmax 3
  nphinored 5
  svdmode 1
  maxsing 20000
  eigthresh 1e-06
  eigwrite 1
  ies_parameter_ensemble ..\wairau\waq_14_s1_200_results\ax_prior_waq_14_s1_200.0.par.jcb
  ies_drop_conflicts True
  overdue_giveup_fac 4
  overdue_giveup_minutes 180
  ies_save_binary True
  ies_ordered_binary False
  ies_localizer waq_14_s1_200_loc.jcb
  max_run_fail 2
  ies_pdc_sigma_distance 4
  ies_verbose_level 1
  ies_localizer_forgive_extra True
  ies_num_threads 30
  ies_subset_size -10
  ies_upgrades_in_memory False
  ies_bad_phi 10000000000.0
  ies_restart_observation_ensemble ..\wairau\waq_14_s1_200_results\ax_prior_waq_14_s1_200.0.obs.jcb
  ies_num_reals 500
  ies_observation_ensemble ..\wairau\waq_14_s1_200_results\hwn_waq_14_s1_200.obs+noise.jcb
• parameter groups external
  haxwn_waq_14_s1_200.pargp_data.csv
• parameter data external
  haxwn_waq_14_s1_200.par_data.csv
• observation data external
  haxwn_waq_14_s1_200.obs_data.csv
• model command line
  python forward_run.py
• model input external
  haxwn_waq_14_s1_200.tplfile_data.csv
• model output external
  haxwn_waq_14_s1_200.insfile_data.csv

[wkitlasten]

sending you some junk via email...