fix handling of null values in arrow vectors #2195
Conversation
There was a problem hiding this comment.
To me this suggests that we can’t/shouldn’t use vector.toArray at all, and instead we should use the “slow” version vector.toJSON to let Arrow handle the conversion into usable JavaScript values. Maybe we could optimize it to allow vector.toArray if vector.nullCount is zero, though. But the type-specific handling of invalid values here seems likely to be wrong and not something that we want to replicate outside of Arrow.
src/options.js
Outdated
| const values = vector.toArray(); | ||
| if (values.nullCount) { | ||
| const NULL = vector.type?.typeId === 5 /* Utf8 */ ? null : undefined; | ||
| for (let i = 0; i < values.length; ++i) if (!values.isValid(i)) values[i] = NULL; |
There was a problem hiding this comment.
We can’t use assignment (mutation) here because this might not be a copy. According to the docs:
If this Vector contains a single Data chunk and the Vector's type is a primitive numeric type corresponding to one of the JavaScript TypedArrays, this method returns a zero-copy slice of the underlying TypedArray values. If there's more than one chunk, the resulting TypedArray will be a copy of the data from each chunk's underlying TypedArray values.
|
ok that's simpler 😓 |
Call vector.toJSON instead of vector.toArray when there are null values.
closes #2194