feat(pass-style,marshal): ByteArray, a binary Passable #2414
+245
−13
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
erights
force-pushed
the
markm-binary-direct
branch
from
eed07e2
to
1df60e9
Compare
erights
force-pushed
the
markm-binary-direct
branch
from
835d70e
to
74007b3
Compare
erights
changed the base branch from
master
to
markm-repair-arraybuffer-transfer
erights
force-pushed
the
markm-repair-arraybuffer-transfer
branch
from
4dd9bf8
to
d19e509
Compare
erights
force-pushed
the
markm-binary-direct
branch
from
74007b3
to
6af60d7
Compare
erights
force-pushed
the
markm-repair-arraybuffer-transfer
branch
from
d19e509
to
585042f
Compare
erights
force-pushed
the
markm-binary-direct
branch
from
6af60d7
to
affa6b3
Compare
|
Converting to draft until more of the stubbed out cases are implemented, and until more of both are tested. |
erights
force-pushed
the
markm-repair-arraybuffer-transfer
branch
3 times, most recently
from
8dce401
to
06e15fe
Compare
erights
force-pushed
the
markm-binary-direct
branch
2 times, most recently
from
53ec91b
to
ce0fd14
Compare
erights
added a commit
that referenced
this pull request
Sep 4, 2024
Staged on #2419 Closes: #XXXX Refs: #2414 #2418 #2419 ## Description #2414 by itself does not work on Node 18 and Node 20 because - those platforms do not have `Array.prototype.transfer`, so #2414 must use `structuredClone` instead - `structuredClone` does exist on Node >= 18, so it should be on supported platforms (though see #2418 ). However, `structuredClone` itself is dangerous and so must not be added to the shared intrinsics. As a result, in #2414 , when `@endo/pass-style` is initialized in a created compartment, it fails to find either `Array.prototype.transfer` and `structuredClone To solve this, @kriskowal suggested that we also shim `Array.prototype.transfer` if needed during `lockdown`, along with other repairs. We are avoiding similarly shimming `Array.prototype.transferToImmutable` because it is not yet standard. But `Array.prototype.transfer` is standard, and so `lockdown` can globally shim it before hardening the shared intrinsics. This PR implements @kriskowal 's suggestion. ### Security Considerations none ### Scaling Considerations by itself, none ### Documentation Considerations nothing signicant. ### Testing Considerations See #2418 . Aside from that, none ### Compatibility and Upgrade Considerations On platforms with neither `Array.prototype.transfer` nor a global `structuredClone`, the ses-shim will *currently* not install an emulation of `Array.prototype.transfer`. However, once we verify that endo is not intended to support platforms without both, we may change lockdown to throw, failing to lock down.
erights
force-pushed
the
markm-binary-direct
branch
from
ce0fd14
to
066637e
Compare
erights
force-pushed
the
markm-binary-direct
branch
from
066637e
to
e696f45
Compare
erights
commented
Oct 14, 2024
| // then according to their lengths. | ||
| // Thus, if the data of ByteArray X is a prefix of | ||
| // the data of ByteArray Y, then X is smaller than Y. | ||
| return comparator(left.byteLength, right.byteLength); |
There was a problem hiding this comment.
Should switch to shortlex. See https://en.wikipedia.org/wiki/Shortlex_order
That way, encodePassable could encode as prefix followed by the unescaped content.
erights
commented
Oct 14, 2024
| // Thus, if the data of ByteArray X is a prefix of | ||
| // the data of ByteArray Y, then X is smaller than Y. | ||
| // @ts-expect-error narrowed | ||
| return compareRank(left.byteLength, right.byteLength); |
There was a problem hiding this comment.
Should delegate the entire compare to rankOrder, which, btw, will switch to shortlex order.
erights
force-pushed
the
markm-binary-direct
branch
from
e696f45
to
6312dd7
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Staged on #2414
Closes: #1331
Refs: #1538 ocapn/ocapn#5 (comment)
Description
Use the
@endo/immutable-arraybufferponyfill directly, rather than the shim@endo/immutable-arraybuffer/shim.js, to provide objects that act like the immutable ArrayBuffers that would result from thetransferToImmutableproposal.Recognize these objects, when also frozen, as the new pass-style
byteArray. Add a branch for each of the dispatches on pass-style that must now understand byte-arrays as well. As of this PR, many of those additional branches are stubbed out with "not yet implemented" errors as placeholders.Security Considerations
Does correctly enforce immutability. But by depending only on the ponyfill rather than the shim, this PR inherits the eval twin problem of the ponyfill. Each instantiation of
passStyleOfwill only recognize the byte-arrays from the instantiation of@endo/pass-stylethat made that byte-array. For environment that, for example, use thepassStyleOfendowed to them by liveslots, liveslots will also need to endow them with the corresponding byte-array maker.In theory, this is not much worse than the need to co-endow
passStyleOfandProxy, so thatpassStyleOfcan reject apparent copy structures that are also proxies.Scaling Considerations
The underlying ponyfill implementation helps us handle bulk binary data in a largely no-copy or minimal-copy manner.
Documentation Considerations
Will add a new pass-style, so everywhere we enumerate all the pass-styles, we'll need to add byte-array.
Testing Considerations
test.failingtest.Compatibility Considerations
Old code that dispatched only on the pass-styles it knew about will not correctly handle passables that contain byte-arrays.
Upgrade Considerations