You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Pydantic enforces strict types. In the current implementation all Spark related logic (readers, writers, transforms, integrations) expect DataFrame (pyspark.sql.DataFrame) class as input or output. However in Spark Connect and subsequently in the Serverless compute the DataFrame class is pyspark.sql.connect.DataFrame, which causes errors in pydantic model validations.
Describe the solution you'd like
Model should except both native and connect DataFrames as a valid input / output
Describe alternatives you've considered
...
Additional context
...
The text was updated successfully, but these errors were encountered:
@mikita-sakalouski to be honest, I would prefer if we treat this a separate issue and merge to main as part of the #59 .
Reason being is that this small change only addresses the strict validation of the DF types, nothing else. It unlocks the next steps for the serverless compute poc.
The feature you are referring is much bigger and will probably require more testing and time.
Is your feature request related to a problem? Please describe.
Pydantic enforces strict types. In the current implementation all Spark related logic (readers, writers, transforms, integrations) expect DataFrame (pyspark.sql.DataFrame) class as input or output. However in Spark Connect and subsequently in the Serverless compute the DataFrame class is pyspark.sql.connect.DataFrame, which causes errors in pydantic model validations.
Describe the solution you'd like
Model should except both native and connect DataFrames as a valid input / output
Describe alternatives you've considered
...
Additional context
...
The text was updated successfully, but these errors were encountered: