[Flyte][3][flytepropeller][Attribute Access][flytectl] Binary IDL With MessagePack #5763
Conversation
Signed-off-by: Future-Outlier <[email protected]>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5763 +/- ##
==========================================
+ Coverage 36.31% 36.35% +0.03%
==========================================
Files 1304 1304
Lines 110072 110147 +75
==========================================
+ Hits 39974 40044 +70
Misses 65936 65936
- Partials 4162 4167 +5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Future-Outlier
changed the title
| t.Run("Generic", func(t *testing.T) { | ||
| literalVal := map[string]interface{}{ | ||
| "x": 1, | ||
| "y": "ystringvalue", | ||
| } | ||
| var literalType = &core.LiteralType{Type: &core.LiteralType_Simple{Simple: core.SimpleType_STRUCT}} | ||
| lit, err := MakeLiteralForType(literalType, literalVal) |
There was a problem hiding this comment.
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
| @@ -559,12 +562,35 @@ func MakeLiteralForType(t *core.LiteralType, v interface{}) (*core.Literal, erro | |||
| strValue = fmt.Sprintf("%.0f", math.Trunc(f)) | |||
| } | |||
| if newT.Simple == core.SimpleType_STRUCT { | |||
| // If the type is a STRUCT, we expect the input to be a complex object | |||
There was a problem hiding this comment.
will this not break if the clients or otherside have the old view of struct?
i.e, this seems backwards incompatible?
cc @EngHabu
There was a problem hiding this comment.
this will not break JSON string case, since this is not a string type.
Signed-off-by: Future-Outlier <[email protected]>
|
Here is how I tested backward compatibility. (mock the process user update propeller)
shut down flyte cluster and change to the new flyte cluster 
python example: import os
import tempfile
from dataclasses import dataclass
import pandas as pd
from flytekit import task, workflow, ImageSpec, dynamic
from flytekit.types.directory import FlyteDirectory
from flytekit.types.file import FlyteFile
from flytekit.types.structured import StructuredDataset
from mashumaro.mixins.json import DataClassJSONMixin
image = ImageSpec(
base_image="ghcr.io/flyteorg/flytekit:py3.9-1.11.0",
packages=["pandas", "pyarrow"],
registry="localhost:30000"
)
@dataclass
class Datum(DataClassJSONMixin):
x: int
y: str
z: dict[int, str]
@task(container_image=image)
def stringify(s: int) -> Datum:
"""
A dataclass return will be treated as a single complex JSON return.
"""
return Datum(x=s, y=str(s), z={s: str(s)})
@task(container_image=image)
def add(x: Datum, y: Datum) -> Datum:
"""
Flytekit automatically converts the provided JSON into a data class.
If the structures don't match, it triggers a runtime failure.
"""
x.z.update(y.z)
return Datum(x=x.x + y.x, y=x.y + y.y, z=x.z)
@dataclass
class FlyteTypes(DataClassJSONMixin):
dataframe: StructuredDataset
file: FlyteFile
directory: FlyteDirectory
@task(container_image=image)
def upload_data() -> FlyteTypes:
"""
Flytekit will upload FlyteFile, FlyteDirectory and StructuredDataset to the blob store,
such as GCP or S3.
"""
# 1. StructuredDataset
df = pd.DataFrame({"Name": ["Tom", "Joseph"], "Age": [20, 22]})
# 2. FlyteDirectory
temp_dir = tempfile.mkdtemp(prefix="flyte-")
df.to_parquet(temp_dir + "/df.parquet")
# 3. FlyteFile
file_path = tempfile.NamedTemporaryFile(delete=False)
file_path.write(b"Hello, World!")
fs = FlyteTypes(
dataframe=StructuredDataset(dataframe=df),
file=FlyteFile(file_path.name),
directory=FlyteDirectory(temp_dir),
)
return fs
@task(container_image=image)
def download_data(res: FlyteTypes):
assert pd.DataFrame({"Name": ["Tom", "Joseph"], "Age": [20, 22]}).equals(res.dataframe.open(pd.DataFrame).all())
f = open(res.file, "r")
assert f.read() == "Hello, World!"
assert os.listdir(res.directory) == ["df.parquet"]
@dynamic(container_image=image)
def dataclass_wf(x: int, y: int) -> (Datum, FlyteTypes):
for _ in range(3):
o1 = add(x=stringify(s=x), y=stringify(s=y))
o2 = upload_data()
download_data(res=o2)
return o1, o2
if __name__ == "__main__":
from flytekit.clis.sdk_in_container import pyflyte
from click.testing import CliRunner
import os
runner = CliRunner()
path = os.path.realpath(__file__)
# result = runner.invoke(pyflyte.main,
# ["run", path, "dataclass_wf", "--x", "-1", "--y", "2"])
# print("Local Execution: ", result.output)
# #
result = runner.invoke(pyflyte.main,
["run", "--remote", path, "dataclass_wf", "--x", "-1", "--y", "2"])
print("Remote Execution: ", result.output)
|
|
use the old flytekit with a new propeller env: 
|
There was a problem hiding this comment.
Hey @Future-Outlier, what happens if a new flytekit produced a msgpack and old flytekit tried to consume it?
| @@ -11,14 +12,16 @@ import ( | |||
| "github.com/flyteorg/flyte/flytestdlib/storage" | |||
| ) | |||
|
|
|||
| const messagepack = "msgpack" | |||
There was a problem hiding this comment.
Please move this const to one place and reference it everywhere...
| switch resolvedVal := currVal.(type) { | ||
| // map | ||
| case map[interface{}]interface{}: | ||
| tmpVal, exist = resolvedVal[attr.GetStringValue()] |
There was a problem hiding this comment.
What will GetStringValue() return if if the attr is foo[0]? shouldn't it fail in that case and say "you can't index into a struct" or something ?
There was a problem hiding this comment.
| currVal = tmpVal | ||
| // list | ||
| case []interface{}: | ||
| if int(attr.GetIntValue()) >= len(resolvedVal) { |
There was a problem hiding this comment.
ditto here. If it's foo.bar GetIntValue() will return 0 I presume? it should fail here too
There was a problem hiding this comment.
yes, GetIntValue() will return 0 , but foo.bar will go to the up map case, so it will called attr.GetStringValue to get the bar attribute
| // Unsupported serialization format | ||
| return nil, errors.Errorf(errors.PromiseAttributeResolveError, nodeID, | ||
| "Unsupported format '%v' found for literal value.\n"+ | ||
| "Please ensure the serialization format is supported.", serializationFormat) |
There was a problem hiding this comment.
This case can never happen, right? since you checked for that early on
There was a problem hiding this comment.
Yes, I can remove it
upstream output (msgpack) -> downstream input (msgpack) This will fail when executing the pod plugin, but not crashed the propeller. |
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Details about 1: @dataclass
class DC2:
a: float
b: DC
c: List[DC]
d: Dict[int, DC]
o = DC2()
x=o.d[1].a # This should be `int` type, but will return `Struct` type when using `get literal type` function. |
…h MessagePack (flyteorg#5763) Signed-off-by: Siddhant Rai <[email protected]>
Tracking Issue
Issue #5318
Why are these changes needed?
flytectlto create and get executions.Note: This PR includes handling of primitive types, Flyte types, and dataclasses.
What changes are proposed in this pull request?
Attribute Access
msgpackbytes inBinary Literal.Binary Literal.flytectl
flytectl, serialize it tomsgpackbytes and create a binary scalar directly.This version is more concise, avoids redundant words, and clarifies the structure a bit.
How was this patch tested?
Attribute Access
unit tests and remote execution
flytectl
command line
get_task.yamlSetup process
single binary
Screenshots
Local Execution
Remote Execution
Check all the applicable boxes