Impute Transform with frame parameter

[qnzhou]

I am reading the most recent doc and find one example usage of impute transform a bit confusing.

In the last example shown in the impute transform doc, a frame of [-1, 1] is used to limit the aggregation (mean in this case) to data points right before and after the missing value. However, the resulting plot does not match this understanding. i.e. I will expect the imputed value for x at t=3 to be 4.5 instead of 4, and the imputed value for y at t=1 to be 2.5 instead of 3. Any insights on what is going on here?

Here is the minimal example to reproduce the chart in question:

import numpy as np
import pandas as pd
import altair as alt

data = pd.DataFrame({
    't': range(5),
    'x': [2, np.nan, 3, 1, 3],
    'y': [5, 7, 5, np.nan, 4]
}).melt('t').dropna()

background = alt.Chart(data).mark_line(point=True).encode(
    x=alt.X('t:Q'),
    y='value:Q',
    color='variable:N',
    opacity=alt.value(0.2)
)

chart = alt.Chart(data).transform_impute(
    impute='value',
    key='t',
    method='min',
    frame=[-1, 1],
    groupby=['variable']
).mark_line(point=True).encode(
    x='t:Q',
    y='value:Q',
    color='variable:N'
)
background + chart

[ChristopherDavisUCI]

This also looks a little strange to me. (Did you change from 'mean' to 'min' intentionally? I agree that it is 'mean' in the docs.) Could it just be some issue with numerical precision? Or maybe right endpoints are included but not left endpoints? Do you get the expected behavior if you use frame=[-1.001, 1.001]?

In many cases the Vega-Lite documentation is more thorough than the Altair documentation, so you could also look there: https://vega.github.io/vega-lite/docs/impute.html

[qnzhou]

Yeah, I was playing with the aggregate setting (using mean, min, max, etc.) in order to understand this behavior. However, none produced the expected result. Changing frame=[-1.001, 1.001] does not produce a valid chart either (there is no errors, but the chart is just blank). It seems the frame range must be integral values as they are indicating the number of data points before and after the missing value.

[ChristopherDavisUCI]

I'm still having some trouble wrapping my head around what's happening, but if you get rid of the .dropna() from the definition of data, does it give the expected behavior? As far as I can tell, Altair and Vega-Lite are pretending the data point is at the end of the DataFrame, so for example in your frame list [a,b], the b is playing no role (because nothing comes after the point).

Definitely the documentation here could use some reworking, but does this seem to be working correctly? Here is the code I used, where I commented out the dropna method.

import numpy as np
import pandas as pd
import altair as alt

data = pd.DataFrame({
    't': range(5),
    'x': [2, np.nan, 3, 1, 3],
    'y': [5, 7, 5, np.nan, 4]
}).melt('t')#.dropna()

chart = alt.Chart(data).transform_impute(
    impute='value',
    key='t',
    method='mean',
    frame=[1, 1],
    groupby=['variable']
).mark_line(point=True).encode(
    x='t:Q',
    y='value:Q',
    color='variable:N'
)

chart

[qnzhou]

Thanks @ChristopherDavisUCI for exploring this issue! Dropping .dropna() does seem to produce the right output. However, I am still confused about why. It seems impute transform works properly when missing data points are labeled with NAN, but not when they are actually missing? This sounds like a bug to me.

For the frame parameter, my understanding of frame=[-a, b] is to simply take a data points before and b data points after the current missing data point to perform the aggregation. If I understand correctly, you are indicating Altair is taking b-1 data points instead after the current missing data point? i.e. the bth data point plays no role in the aggregation. But the asymmetry between a and b is kind of unintuitive...

[ChristopherDavisUCI]

I agree it's confusing @qnzhou. It's hard to say if it's a bug or just some misunderstanding on our part. If it's a bug, I think it's a bug on the Vega-Lite side rather than the Altair side (Altair shouldn't be doing anything more than producing a specification that can be used by Vega-Lite). Here's an example of Vega-Lite code that I believe behaves the same way: Open the Chart in the Vega Editor

I'd suggest asking in the Vega-Lite repo if you'd like more insight into how the "before and after" data points are being handled.

[qnzhou]

Sounds good! Here is the discussion thread on vega-lite: vega/vega-lite#8766

Seems to be related to the following issues:

• Impute "frame" argument is not working as advertised. vega-lite#4895
• impute seems difficult to use correctly vega-lite#8519