[BUG] Discrepancies observed between subprocess and simu mode with variance_method=bootstrap

[jeandut]

Using below test config with newton raphson strategy we observe discrepancies between simu and non simu mode leading in the remote/subprocess mode to instabilities in the hessian raising errors and in the simu mode to no error at all:

name: "Real-world experiments"

# initial_seed is used to generate seed for each run
initial_seed: 42

data:
  ndim: 10
  cate: 0.7
  scale_t: 10.0
  shape_t: 3.0
  propensity: "linear"
  standardize_features: False

defaults:
  - /[email protected]: pooled_iptw
  - /[email protected]: fl_iptw
  - _self_

models:
  IPTW:
    effect: "ATE"
    cox_fit_kwargs:
      robust: False
  FedECA:
    ndim: 10
    num_rounds_list: [50, 50]
    # I know this is awful but couldn't find some corresponding Omegaconf magic easily
    bootstrap_seeds = [42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241]

# config fit FedECA
fit_fedeca:
  n_clients: 2
  variance_method: naive
  split_method: "split_control_over_centers"
  split_method_kwargs: {"treatment_info": "treatment_allocation"}
  data_path: "./tmp"
  # We start on the second url the first is reserved for the server
  urls: [urls]
  server_org_id: 'Org3MSP'

models_common:
  treated_col: "treatment_allocation"
  event_col: "event"
  duration_col: "time"

parameters:
  n_samples: 1_000
  n_reps: 1

hydra:
  sweep:
    dir: "real-world"
  sweeper:
    params:
      # 9 will not be kept but used as a warm-up for the server
      ++fit_fedeca.n_clients: 2
      ++fit_fedeca.backend_type: subprocess, remote, simu
      ++fit_fedeca.variance_method: bootstrap

[jeandut]

It turns out this is a side effect of running them one after the other because the reinstantiated object is not properly cleaned and thus the second fedeca being run by hydra is "damaged" by the previous run. It is linked to #27

[jeandut]

Fix should therefore either use #27 output or at least patch reset_experiment function by modifying at the FedECA level (not at the Experiment level):

def reset_experiment(self):
    super().reset_experiment()
    # Add FedECA specific stuff such as cleaning propensity models

Fixed by #28

[jeandut]

Note that this fixes discrepancies in some cases but with increasing seeds it breaks for some reasons. This is getting more and more mysterious...

[jeandut]

#29 solved the seed issue !
However there still is some residual leaking it seems !

[jeandut]

We still obtain different results with individual runs vs running all backend_type with hydra sequentially.