SmartDataframe vs SmartDatalake

[adityamiskin]

I dont understand why is there a need to have two types of Dataframes. Arent both doing the same thing? SmartDataframe works only on 1 dataframe whereas SmartDatalake works on multiple. Why cant we use SmartDatalake for everything as 1 dataframe can be considered as multiple dataframes with length 1.

// For example : We have 2 dataframes, df1, df2.
dfs = [df1,df2]

// both are same
data = SmartDataframe(df1)
data = SmartDatalake(dfs[0])

// we can generalize it
data = SmartDatalake(dfs)   // assert dfs should be a list. Doesnt matter the length.

So why can we just generalize it to have only SmartDatalake and take care of length of dfs in the backend?

[gventuri]

Hi @adityamiskin, there are some differences.

For example, you can use a SmartDataframe as a normal pandas dataframe, which better integrates in the workflow.
Additionally, you can use shortcuts with SmartDataframes, which are not available on the SmartDatalake.