better set_body #42
Merged
better set_body #42
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
set_body wasn't doing the right thing when it comes to determining the destination datset structure for writing a body. We now inherit structure from existing dataset data, and only fall back to json if no structure exists. The general problem here is a separation of concerns between data in the starlark runtime and dataset serialization formats. Those should always be seperate. closes qri-io/qri#756
the mental model behind 'raw' and 'data_format' is off. 'data_format' implies that we're changing the ultimate data format, which is not the case. Now set_body has only one optional argument: 'parse_as', which defaults to the empty string. By default qri assumes the data value provided to set_body is an iterable starlark data structure (tuple, set, list, dict). When parse_as is set, set_body assumes the provided body value will be a string of serialized structured data in the given format. valid parse_as values are "json", "csv", "cbor", "xlsx". We'll need more tests to confirm each format works properly, but all this is doing under the hood is overriding all parsing and setting body bytes directly, which is analogous to "qri save --body file.[parse_as] me/dataset". BREAKING CHANGE: ds.set_body raw and data_format params have been replaced by parse_as
ghost
added
the
dustmop
approved these changes
May 16, 2019
ds/testdata/test.star
Outdated
|
|
||
| # csv_ds is a global variable provided by dataset_test.go | ||
| # "cycling" csv data through starlark shouldn't have significant effects on the |
There was a problem hiding this comment.
how about "round-tripping" instead of "cycling"
ds/dataset.go
Outdated
| } | ||
|
|
||
| return starlark.None, fmt.Errorf("expected raw data for body to be a string") | ||
| d.write.SetBodyFile(qfs.NewMemfileBytes(fmt.Sprintf("data.%s", df), []byte(string(str)))) |
There was a problem hiding this comment.
Use "body.%s" instead of "data.%s".
There was a problem hiding this comment.
totally. makes me want this more: qri-io/dataset#188
ds/dataset_test.go
Outdated
| }, | ||
| }, | ||
| } | ||
| ds.SetBodyFile(qfs.NewMemfileBytes("data.csv", []byte(text))) |
There was a problem hiding this comment.
Use "body.csv" instead of "data.csv"
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
set_body wasn't doing the right thing when it comes to determining the destination datset structure for writing a body. We now inherit structure from existing dataset data, and only fall back to json if no structure exists.
The general problem here is a separation of concerns between data in the starlark runtime and dataset serialization formats. Those should always be seperate, with set_body giving qri the data it wants serialized, or skipping serialization with
ds.set_body(data, parse_as='format'