Test the effects of a random noise vector in the model

[dfsnow]

Following up on discussions in the office, we should test the perturbation effect of a vector of random noise in the model. Specifically, we should look at feature importance and the stability of estimates.

[Damonamajor]

@dfsnow do we want this in something like the equivalent of an EI-issue? Where should it be stored?
The same question for #273

[Damonamajor]

@dfsnow We are also probably going to have a decent number of comparisons / test etc. Do we want to have some set parameters or a baseline to test these things against throughout the modeling season?

[dfsnow]

@dfsnow do we want this in something like the equivalent of an EI-issue? Where should it be stored? The same question for #273

For something like this issue, I think it's sufficient to just post findings in the issue comments (including any supporting tables and plots). No need for separate analyses.

@dfsnow We are also probably going to have a decent number of comparisons / test etc. Do we want to have some set parameters or a baseline to test these things against throughout the modeling season?

We'll run a baseline model as soon as #263 is merged. We can use that for comparison for now (until we have 2024 data).

[dfsnow]

Adding a random vector of noise was actually pretty useful! It didn't alter the global performance statistics, but it did introduce some instability in individual predictions that helped us debug.

Adding random noise caused a few predictions to flip between very low ($100K) and very high ($10M) values.
This turned out to be caused by the inclusion of some very high (albeit plausible) erroneous sales in the training data that made it past our sales validation. We've since corrected the underlying sales, which in turn stabilized our predictions.