Implement adaptive localization

[dafeda]

Resolves: #4411

Note that the current implementation is naive in that it loops through all parameters.
My understanding is that this is the most accurate but too computationally expensive in practice.
Perhaps we can use this as reference to compare more efficient methods against.

I've added two new keywords; LOCALIZATION and LOCALIZATION_CORRELATION_THRESHOLD which may be set as for example:

ANALYSIS_SET_VAR STD_ENKF LOCALIZATION True
ANALYSIS_SET_VAR STD_ENKF LOCALIZATION_CORRELATION_THRESHOLD 0.2

Pre review checklist

• Added appropriate release note label
• PR title captures the intent of the changes, and is fitting for release notes.
• Commit history is consistent and clean, in line with the contribution guidelines.

Adding labels helps the maintainers when writing release notes. This is the list of release note labels.

[frode-aarstad]

Just looking at the code it looks good.

• Might be worth it to add some comments to the functions if its not obvious whats going on.
• Is it possible to write a test for this specific improvement? ie: correlated_parameter_response_pairs

[codecov-commenter]

Codecov Report

Merging #4243 (6bb7092) into main (b828844) will increase coverage by 0.07%.
The diff coverage is 89.15%.

@@            Coverage Diff             @@
##             main    #4243      +/-   ##
==========================================
+ Coverage   59.19%   59.26%   +0.07%     
==========================================
  Files         440      440              
  Lines       30667    30729      +62     
  Branches     3135     3135              
==========================================
+ Hits        18153    18212      +59     
- Misses      11729    11732       +3     
  Partials      785      785              

Impacted Files	Coverage Δ
...ert/gui/ertwidgets/analysismodulevariablespanel.py	19.81% <12.50%> (-0.57%)	⬇️
src/ert/_c_wrappers/analysis/analysis_module.py	82.50% <93.10%> (+1.12%)	⬆️
src/ert/_c_wrappers/analysis/__init__.py	100.00% <100.00%> (ø)
src/ert/analysis/_es_update.py	90.00% <100.00%> (+2.06%)	⬆️
src/ert/_c_wrappers/enkf/config/gen_data_config.py	71.42% <0.00%> (-12.50%)	⬇️
...rc/ert/_c_wrappers/enkf/config/enkf_config_node.py	90.20% <0.00%> (-0.52%)	⬇️
src/ert/gui/ertwidgets/validationsupport.py	98.63% <0.00%> (+19.17%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[tommyod]

We can speed up the computations by not forming the diagonal matrices. Here's an example showing what I mean, where Sigma_A and Sigma_Y are never formed as diagonal matrices. Runs around 20x faster than the current code on these sizes.

ensemble_size = 10
S = np.random.randn(1000, ensemble_size)
X_local = np.random.randn(1000, ensemble_size)

# Standard deviations
Sigma_Y = np.std(S, axis=1, ddof=1)
Sigma_A = np.std(X_local, axis=1, ddof=1)

# State-measurement covariance matrix
Y_prime = S - S.mean(axis=1, keepdims=True)
A = X_local - X_local.mean(axis=1, keepdims=True)
C_AY = A @ Y_prime.T / (ensemble_size - 1)

# Absolute values of the correlation matrix
c_AY = np.abs((C_AY / Sigma_Y.reshape(1, -1)) / Sigma_A.reshape(-1, 1))

alternatively, if C_AA and C_YY are needed:

ensemble_size = 10
S = np.random.randn(1000, ensemble_size)
X_local = np.random.randn(1000, ensemble_size)

# Estimate covariance matrix for outputs
Y_prime = S - S.mean(axis=1, keepdims=True)
C_YY = Y_prime @ Y_prime.T / (ensemble_size - 1)
Sigma_Y = np.sqrt(np.diag(C_YY))

A = X_local - X_local.mean(axis=1, keepdims=True)
C_AA = A @ A.T / (ensemble_size - 1)
Sigma_A = np.sqrt(np.diag(C_AA))

# State-measurement covariance matrix
C_AY = A @ Y_prime.T / (ensemble_size - 1)

# State-measurement correlation matrix
c_AY = np.abs((C_AY / Sigma_Y.reshape(1, -1)) / Sigma_A.reshape(-1, 1))

[Blunde1]

We can speed up the computations by not forming the diagonal matrices. Here's an example showing what I mean, where Sigma_A and Sigma_Y are never formed as diagonal matrices. Runs around 20x faster than the current code on these sizes.

I don't see there being anything in the fit() or update() that is slower than the previous dense matrix inversion of Sigma_Y (only needs to be formed once, outside loops) and Sigma_A, so I fully support this. It would then even be possible to hope that adaptive localization runs in reasonable time even on Troll. It is at least worth to test it.

[dafeda]

Has been merged to main.