Add Accuracy in time

[Vincent-Maladiere]

Address #74

Need to document and add test

[antoinebaker]

I think the current implementation may be misleading when the time_grid does not have a good resolution (for instance small n_time_grid).

There are two tau values:

• one used in y_test_class, it's the specified value (by quantiles or taus) and returned value as well
• one used in y_pred_class, it's the least greater value in time_grid, given by time_grid[tau_idx]

I see two options:

1. raise a warning if these two are too "different" (but difficult to tell what too different means as it will depend on the CIF rate of change)
2. actually redefine and return tau = time_grid[tau_idx] used for both y_pred_class and y_test_class, in other words tau is forced to be in time_grid.

WDYT ?

[antoinebaker]

I think I saw an earlier remark of @judithabk6 about this, but you should update this to the aistats ref and congrats btw :)

[Vincent-Maladiere]

I added the HAL link, for AISTATS I guess we should wait for the proceeding link? Because the PDF is not available on OpenReview anymore. WDYT?

[antoinebaker]

I agree that's a good reason to wait

[antoinebaker]

time_grid needs to be sorted.

[Vincent-Maladiere]

Yes, although that's a common assumption, it might be good to sort it and sort the columns of y_pred? If so, would you raise a warning? Or should we throw an error when time_grid is not sorted? 🤔

[Vincent-Maladiere]

I added a warning, and a reordering with a copy

[Vincent-Maladiere]

I think the current implementation may be misleading when the time_grid does not have a good resolution (for instance small n_time_grid).

You're right. To extend your suggestion 2, would you agree with dropping the taus argument entirely? So that, if quantiles are not defined, we use the grid itself, and if they are, we are assured to have tau in time_grid by doing the following:

time_grid = np.array([5, 10])
quantiles = np.array([.0, .2, .5, .8])

taus = np.quantile(time_grid, quantiles)
tau_indices = np.searchsorted(time_grid, taus)
taus = np.unique(time_grid[tau_indices])
taus
>> [5, 10]

[antoinebaker]

To extend your suggestion 2, would you agree with dropping the taus argument entirely? So that, if quantiles are not defined, we use the grid itself, and if they are, we are assured to have tau in time_grid by doing the following

Yes, it seems a good solution to me ! And if I remember correctly

taus = np.quantile(time_grid, quantiles, method='inverted_cdf')

does just that.