Use Monte Carlo methods to estimate error on model parameters

[dwysocki]

There is no accepted general method for computing standard error on Lasso coefficients. To remedy this, we might make use of bootstrapping.

Here we can discuss the possibilities for this addition, such as:

• how to implement
• which method to utilize

[earlbellinger]

Bootstrapping won't work - instead we need to use Monte Carlo. It's easy enough: with some number of times --num_perturbations, we perturb each observation's magnitude with normal noise whose standard deviation is that observation's uncertainty. Then we report medians and standard deviations for period (if unspecified), amplitudes, phases, fitted magnitudes, and also parameters like Phi_31.

Why params like Phi_31? Because 1/N sum_N (Phi_1 - 3 Phi_3) % 2pi is not equal to [(1/N sum_N Phi_1) - (3/N sum_N Phi_3)] % 2pi.

[dwysocki]

Just some comments on this:

• note that we now output the fitted magnitudes to files, they've been removed from the table
  • this is actually very convenient, because instead of outputting (phase, mag) we can output (phase, mag, error)
• any time an error is omitted (e.g. period is already provided, or --num-perturbations=0), we should put a 0 for its value, instead of omitting it
• also, keep with naming convention, and call it --num-perturbations

[earlbellinger]

I like the idea of weighing each model by its goodness of fit. However, R^2 isn't a good choice for doing this because its value can be negative. Instead we should weigh each model by 1/MSE.

We can report weighted medians instead of just medians.

We could also do weighted std's but that's just a choice -- probably keep them unweighted.