[NOMRG] debug script for pdcdws & pinball

[mathurinm]

@Badr-MOUFAD there is this comment

    # Despite violating the conditions mentioned in [1]

    # this choice of steps yield in practice a convergent algorithm
    # with better speed of convergence

Do you think the issue happens because of it? Or it could be something else?

[Badr-MOUFAD]

Yes indeed @mathurinm, you are right.
After investigation, the algorithm converges when using stepsizes that verifies the inequality.

However, this comes with a drawback of making the convergence speed very very slow.
The algorithm fails to get the last remaining 1e-1 to the optimal object (Extrapolation might help if we manage to spot a auto regressive behavior of the iterates 🤷‍♂️)

I'm afraid that fixing this problem will result in performance degradation in other problems/setups.

I tried a small hack by multiplying/dividing by n_samples/n_feautres to take bigger steps in primal/dual spaces, but it doesn’t not work all the times.

[mathurinm]

However, this comes with a drawback of making the convergence speed very very slow.

I think a good enough solution would be to add an unsafe_stepsize=True parameter, that uses the current stepsize. If this argument is False, we fall back to the safe, smaller stepsizes.
This would deserve a comment pointing to the adequate equation in the Fercoq & Richtarik paper, because it is not trivial

WDYT?

Also, I'm not understanding the problem 100%, because if we change y a bit in @tpanum's example, convergence is back to normal 🤔