Solver scalability

[debasish83]

@sryza I am using cross sectional time series for a problem (similar to ARX models) where I got into problems of size > 1024 and looks like the gram solver is not scaling (assuming OLS uses that)...Are you ok if I move the spark-ts solvers to use BFGS (for smooth opt) / OWLQN (for trend filter) ?

[debasish83]

val regression = new OLSMultipleLinearRegression()
regression.setNoIntercept(true)
regression.newSampleData(Y, X)
val params = regression.estimateRegressionParameters()
This solver does not scale beyond 1024 kind features...Do you use nonlinear CG somewhere ? I have a variant in Breeze QuadraticMinimizer that scales to 4096 and beyond that we can the nonlinear CG / BFGS...Let me know

[sryza]

@debasish83 that's fine with me.