Spec discussion: BATCH message type, first iteration #859
Replies: 5 comments 11 replies
-
|
Proposed "batch_config": {
"encoding": {
"format": "jsonl",
"compression": "gz"
},
"storage": {
...
}
}For CSV: "batch_config": {
"encoding": {
"format": "csv",
"compression": "gz",
"include_header": false,
"delimiter": "|"
},
"storage": {
...
}
} |
Beta Was this translation helpful? Give feedback.
-
|
I like this a lot! 🙌 Thanks, @edgarrmondragon for proposing. |
Beta Was this translation helpful? Give feedback.
-
|
I've updated the spec proposal to use e.g. |
Beta Was this translation helpful? Give feedback.
-
|
FYI, since record-level optimizations keep coming up in discussion, I've logged this new discussion to allow further discussion on that topic: |
Beta Was this translation helpful? Give feedback.
-
|
@edgarrmondragon and team, for the storage layer at least, what about potentially using the
Filesystems supported: My only hesitations are that:
Another possibility is that we completely insulate the users and the spec itself from the In a first iteration we might support The contract between tap and target would be that a target (or the SDK itself) supports one or more protocol prefixes for files in the |
Beta Was this translation helpful? Give feedback.
-
Note that depending on the flexibility of the underlying library ( I'd suggest using, for example, |
Beta Was this translation helpful? Give feedback.
-
Fantastic! I meant this as an example, in case the Important to call out that sometimes we'll parse this file ourselves in the SDK, and other times we'll pass the pointer (perhaps with some adaptation) on to the target library to ingest natively. However, even if we are passing on the file to be ingested natively, there is still value in us being able to parse the file ourselves in Python - especially for use cases around pre-validation.
Good point! Maybe that was a red herring then... 🙄 🐟
I am not particularly attached to And presumably, a similar evaluation would occur with Wdyt? Does this sound right to you? |
Beta Was this translation helpful? Give feedback.
-
@aaronsteers Yeah, that'd be fine. Though, should we plan how we're going to eventually support parameters? Some of the implementations support a root argument that carries both the root directory of the filesystem, as well as connection params like For example, a local root: "batch_config": {
"encoding": {
...
},
"storage": {
"protocol": "local",
"root": "/path/to/dir",
"output_dir": "./.output/",
"file_prefix": "mytap-tempfile-"
}
}would then be processed as from fs import open_fs
mapped_protocol = "file://" if protocol == "local" else ...
with open_fs(f"{mapped_protocol}://{root}") as fs:
with fs.open(output_dir, "w") as file:
file.write(...)So the What do you think of merging the |
Beta Was this translation helpful? Give feedback.
-
|
@edgarrmondragon - I like something like We do then need to parse that or use pattern matching to confirm it's viable but this is a pretty trivial lift with some basic assumptions on the format. If we move in the direction for the base storage location as a string including protocol, we probably don't also need a root directory part also, since we can have the use include the base dir in that root. Do you agree? (I don't feel strongly either way, tbh.) We could eventually expose something like |
Beta Was this translation helpful? Give feedback.
-
|
New thread for the basic BATCH message spec Multiple-files per messages vs one message for every file?The current spec for BATCH messages requires the following sample structure: {
"type": "BATCH",
"stream": "users",
"encoding": {
...
},
"manifest": [
"path/to/batch/file/1",
"path/to/batch/file/2"
]
}Would this be functionally equivalent to: (pretty-printed for readability) {
"type": "BATCH",
"stream": "users",
"encoding": {
...
},
"filepath": "path/to/batch/file/1"
}
{
"type": "BATCH",
"stream": "users",
"encoding": {
...
},
"filepath": "path/to/batch/file/2"
}I prefer the second and I ask because I'm struggling to define where the number of files in the @aaronsteers wdyt? cc @meltano/engineering in case you have thoughts |
Beta Was this translation helpful? Give feedback.
-
|
I guess targets could benefit from bulk-loading multiple files at once with some glob expression, and thus from a single message with multiple batches. Yet, where do we declare this in the tap? |
Beta Was this translation helpful? Give feedback.
-
|
@edgarrmondragon - I see your points and it's something definitely worth thinking about. What about our default handler just having 'list of one element' behavior? This keeps the default implementation simple, while also leaving the spec open for multi-file support per message when it is appropriate. For the default implementation, I agree, there probably isn't much benefit to batching multiple files together and how/why to configure it that way seems unclear to me as well. Logical multi-file batches versus a logical sequence of single-file batchesNow that you raise this, and now that I've had a bit more time to think of it, I think a helpful context would be to think of how it will be overridden by native implementations. The primary use case for multi-file-manifest-per-message is when handling databases that are fastest when creating multi-file outputs.
And then, passing all of the files along in one manifest is a signal that they are part of the same logical batch, versus being sequential logical batches. For streams which are ordered and also using multi-file manifests in the BATCH record type, sort order would be expected across messages but not across files in the manifest. There's no target-specific applications I know of that care about sort order, but some targets do perform better when loaded in order of the sort key. So, a multi-file manifest may behave differently in the target versus a sequence of single-file manifests, according to the target implementation. Another example, a target that supports loading from multi files at once will almost certainly send all files in the manifest together in the same command, while the same target receiving a sequence of files would more likely try to sequence multiple SQL commands, one per BATCH message. Wdyt? |
Beta Was this translation helpful? Give feedback.
-
|
For simplicity, I like the one-file-per-message implementation. It helps enforce ordering (even if order isn't generally important for targets) and it also gives more flexibility to the target to decide how it wants to handle batches. In the "target that supports loading from multi files at once" case (e.g. Redshift), it would be perfectly reasonable for that target to do its own batching. I.e. it can wait for |
Beta Was this translation helpful? Give feedback.
-
|
@kgpayne and @edgarrmondragon re:
My point above is that these are not functionally equivalent, and there's a benefit to expressiveness even if most implementations only have a single file per BATCH message. The target can decide to ignore the logical nature of the batch, but at least we have the ability to communicate the distinction between these very different operations: UNLOAD fact_mytable TO s3://path-to-output-prefix/
> 512,234 records output over 26 files
COPY INTO mydest FROM s3://path-to-output-prefix/*
> 512,234 records affected.Which is very different from: UNLOAD fact_mytable TO s3://....
> 512,234 records output over 26 files
COPY INTO mydest FROM s3://.....part1
> 20,000 records affected.
COPY INTO mydest FROM s3://.....part2
> 20,000 records affected.
COPY INTO mydest FROM s3://.....part3
> 20,000 records affected.
COPY INTO mydest FROM s3://.....part4
> 20,000 records affected.
...
COPY INTO mydest FROM s3://.....part26
> 234 records affected.And, to the idea of including record counts, it's very likely that we don't know how many records are written per file, but we do know how many records total are in the batch. I'm not arguing that we need a record count in the first iteration, just that in order to be able to express that there are |
Beta Was this translation helpful? Give feedback.
-
This is a great point. Thanks @aaronsteers! |
Beta Was this translation helpful? Give feedback.
-
|
@edgarrmondragon @aaronsteers this is excellent 👏 Two other things that I found useful when implementing
Something like: {
"type": "BATCH",
"stream": "users",
"encoding": {
...
},
"filepath": "path/to/batch/file/1",
"metadata": {
"last_batch": false,
"batch_size": 10000
}
}
{
"type": "BATCH",
"stream": "users",
"encoding": {
...
},
"filepath": "path/to/batch/file/2",
"metadata": {
"last_batch": true,
"batch_size": 367
}
} |
Beta Was this translation helpful? Give feedback.
-
Opening this discussion to review and finalize the BATCH message type spec.
Related to:
BATCHmessage type (akaFAST_SYNCspec) #9Tap config
Starting with a first supported file type of
.jsonl.gz:Target config
No changes needed. Details are provided within the
BATCHmessage type.Message type sample
Benefit of compression
Most modern data systems can read and write gzipped sources faster than raw/uncompressed files, due to the relatively low cost of CPU cycles versus the relative high savings of reduced IO when reading/writing compressed files. These savings are more pronounced when network upload/download speeds are considered and dealing with uploading files to remote systems.
Stream isolation in batch files
To streamline load operations, each batch file will only contain data from a single stream. This allows faster loads on the target side, since data validation and other logic can be performed on a per-file basis instead of per-record.
Negotiated capabilities
First iteration
In first iteration, the user would need to confirm capabilities of tap and target manually, and would need to set the tap config manually. The target would throw an error if any of these are true:
BATCHmessage type is not supported by the targetformatprovided from the tapmanifestLong-term path to automate tap/target batch negotiation
In the long-term, the tap and target would both print information about supported batch capabilities when CLI
<plugin-name> --about --format=jsonis executed. An orchestrator would be able to run--abouton both the tap and target, and would then be able to negotiate the most optimal batching strategy.Note, per the below, that batching may still be performed by the orchestrator even if one or zero of the plugins support batching natively.
Batch compatibility with legacy taps and targets
Batch capabilities will be available for legacy non-SDK taps and targets via the
meltano-map-transformplugin, which itself inherits from SDK tap and target classes, and which would be able to serialize and deserialized batch files, as needed.Sample invocation with legacy tap and legacy target:
Sample invocation with SDK tap and legacy target:
Sample invocation with legacy tap and SDK target:
All of the above hybrid examples optimize the workload by keeping record messages out of the memory buffer between tap and target. The tap is able to write to disk as quickly as records are available, and the target only reads data from disk as quickly as it is able to load that data into the destination.
Beta Was this translation helpful? Give feedback.
All reactions