Skip to content

Commit

Permalink
Adding auxiliary_experiments_by_purpose to COPY_DB_IDS_ATTRS_TO_SKIP (
Browse files Browse the repository at this point in the history
#2934)

Summary:
Pull Request resolved: #2934

Adding `auxiliary_experiments_by_purpose` to COPY_DB_IDS_ATTRS_TO_SKIP.

## Context:

In the follow-up diff D64281332, I'm making changes to make [play_preference_game](https://www.internalfb.com/code/fbsource/[e706ec5635b4cb836d54c8ade1dfddf297806aab]/fbcode/ax/fb/utils/preference/preference.py?lines=499) use [get_preference_exploration_gs](https://www.internalfb.com/code/fbsource/[d119de48295da9f1682c1b1bc1a632abd0f3605c]/fbcode/ax/fb/modelbridge/generation_strategies.py?lines=1558) instead of its own custom logic.

The error this diff is trying to fix happens *after* the preference game is finished and [when we try to save the PE experiment](https://www.internalfb.com/code/fbsource/[b8af5cabad15579a0e81776a78b43b144976b420]/fbcode/ax/fb/utils/preference/preference.py?lines=393)(the link is based on v0.5 of D64281332).

The logic inside `interactive_optimize` used there might be a bit convoluted with local async candidate generation, but the (synchrounous version) logic's pseudo code looks like this:
```
pe_experiment["bo_aux_exp"] = [bo_exp]
gs = get_preference_exploration_gs(pe_experiment)
for i in (nums_pref_questions):
	pref_game_arms = gs.gen(n=2)
	user_response = elicit_user_preference(pref_game_arms)
	pe_experiment.new_batch_trial(user_response)
	save_experiment(pe_experiment)
save_experiment(pe_experiment) # <- the error happens
```

The error we see is (stack trace pasted at: P1656339174):

>SQADecodeError: Error encountered while traversing source ['auxiliary_experiments_by_purpose', <AuxiliaryExperimentPurposeFB.BO_EXPERIMENT: 'bo_experiment'>, '0', 'experiment', 'runner', 'None'] and target ['auxiliary_experiments_by_purpose', <AuxiliaryExperimentPurposeFB.BO_EXPERIMENT: 'bo_experiment'>, '0', 'experiment', 'runner', '<ax.fb.runners.json.JSONRunner object at 0x7f17a6997f10>']: Encountered two objects of different types: <class 'NoneType'> and <class 'ax.fb.runners.json.JSONRunner'>.Consider adding to COPY_DB_IDS_ATTRS_TO_SKIP if appropriate.

I'm not entirely sure where the runner in the new object is coming from as I have never explicitly set the runner of the BO experiment, and the [trial (of the PE experiment) is marked running with `no_runner_required=True`](https://www.internalfb.com/code/fbsource/[b8af5cabad15579a0e81776a78b43b144976b420]/fbcode/ax/fb/utils/preference/preference.py?lines=185) (the link is based on v0.5 of D64281332). Note that the aux experiment is the BO experiment, which is not expected to change at all during the course of the preference game.

There are other related errors that happened in `copy_db_ids` that I was able to get rid of (e.g., by manually dropping the [place-holder optimization_config that we are not using](https://www.internalfb.com/diff/D63794133) in the [generated GR](https://www.internalfb.com/code/fbsource/[b8af5cabad15579a0e81776a78b43b144976b420]/fbcode/ax/fb/modelbridge/preference.py?lines=309)), but it seems likt to fix this one, the more appropriate solution is to add `auxiliary_experiments_by_purpose` to `COPY_DB_IDS_ATTRS_TO_SKIP`. The alternative would be to figure out where the runner is coming from and manually remove that before saving. However, for a real experiment, we probably don't wanna remove the runner of the BO experiment since we'd need that to deploy the QE experiment, for example.

Reviewed By: lena-kashtelyan

Differential Revision: D64520734

fbshipit-source-id: 7c5675c633615047391afa482cd70610b299850a
  • Loading branch information
ItsMrLin authored and facebook-github-bot committed Oct 22, 2024
1 parent 7d37679 commit 63dedd6
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions ax/storage/sqa_store/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@
"_steps",
"analysis_scheduler",
"_nodes",
# For auxiliary experiments, we don't expect any updates, so we
# don't need to recur into them during `copy_db_ids`.
"auxiliary_experiments_by_purpose",
}
SKIP_ATTRS_ERROR_SUFFIX = "Consider adding to COPY_DB_IDS_ATTRS_TO_SKIP if appropriate."

Expand Down

0 comments on commit 63dedd6

Please sign in to comment.