random_state for theorist fit function

[chadcwilliams]

I have noticed a discrepancy with the random_state parameters. Specifically, the experimentalist can take the parameter as an input at each call, the experiment runners (from synthetic data) takes the parameter at initialization, and the theorist (at least BMS) does not have this parameter.

Would it be a good idea to make sure all of these can take random_state at each call?

This is what is currently looks like:

    s = experimentalist(s, num_samples=10, random_state=42)
    s = experiment_runner(s, added_noise=0.1)
    s = theorist(s)

Would be nice if we could:

    s = experimentalist(s, num_samples=10, random_state=42)
    s = experiment_runner(s, added_noise=0.1, random_state=42)
    s = theorist(s, random_state=42)

[younesStrittmatter]

There is a outstanding PR here: AutoResearch/autora-synthetic#10.
I think we should integrate this there:
[] Revert back to "added_noise" to have a consistent name for this argument throughout all experiment_runners
[] Rename experiment_runner to run
[] Pass the random_state to the run function instead of the "initialization" of the experiment

[younesStrittmatter]

we can work on this branch: https://github.com/AutoResearch/autora-synthetic/tree/9-chore-rename-input-arguments-of-runners-to-us-with-state

[chadcwilliams]

Sounds good. I am happy to lead the charge if we all agree it's best.

[younesStrittmatter]

Yes, I think we can implement these changes

[younesStrittmatter]

The newest AutoResearch/autora-synthetic#10 has all the changes we discussed: Rename experiment_runner -> run, random_state into the run function, observation_noise -> added_noise

[younesStrittmatter]

For the theorists: Internaly the theorist function you are calling does this:
new_model = estimator.set_params(**kwargs).fit(X, y)
so if the estimator.set_params function has the keyword argument random_state, then this would work. But I am not sure if sklearn BaseEstimator generally do have this keyword

[chadcwilliams]

I tried this with BMS, which has a seed parameter in the fit function and indeed got an error. The set_params() seems to be setting the parameters of the class specific to initializing, and not to fitting.

ValueError                                Traceback (most recent call last)
Cell In[12], line 4
      1 print("\033[1mPrevious Model:\033[0m")
      2 print(f"{s.model}\n")
----> 4 s = theorist(s, seed=42)
      6 print("\n\033[1mUpdated Model:\033[0m")
      7 print(s.model)

File c:\Users\cwill\GitHub\virtualEnvs\autoraEnv\lib\site-packages\autora\state\delta.py:1033, in delta_to_state.._f(state_, **kwargs)
   1031 @wraps(f)
   1032 def _f(state_: S, **kwargs) -> S:
-> 1033     delta = f(state_, **kwargs)
   1034     assert isinstance(delta, Mapping), (
   1035         "Output of %s must be a `Delta`, `UserDict`, " "or `dict`." % f
   1036     )
   1037     new_state = state_ + delta

File c:\Users\cwill\GitHub\virtualEnvs\autoraEnv\lib\site-packages\autora\state\delta.py:768, in inputs_from_state.._f(state_, **kwargs)
    761 if ("conditions" in arguments and isinstance(arguments["conditions"], pd.DataFrame) and
    762     "variables" in [i.name for i in fields(state_)]):
    763     arguments["conditions"] = (
    764         align_dataframe_to_ivs(arguments["conditions"],
    765                                getattr(state_, "variables").independent_variables)
    766     )
...
    232     )
    234 if delim:
    235     nested_params[key][sub_key] = value

ValueError: Invalid parameter 'seed' for estimator x. Valid parameters are: ['epochs', 'prior_par', 'ts'].

Would it be possible to adapt the State to take multiple arg types? I haven't thought this through yet, so there could be a major flaw in that thinking...
new_model = estimator.set_params(**init_kwargs).fit(X, y, **fit_kwargs)

It would make things a bit more complicated I think:
theorsit(s, init_kwargs={'n_epochs':100}, fit_kwargs={'seed':42})

[hollandjg]

We should look at https://scikit-learn.org/stable/developers/develop.html#fitting and think about how to pass data-dependent and data-independent parameters.