When targets emit state

[pnadolny13]

Related to a slack thread https://meltano.slack.com/archives/CMN8HELB0/p1688411149324679

My expectation of a singer target is that it will emit state messages received from the tap whenever it's confident that it was able to load those records to the destination successfully. The current implementation of the SDK seems to wait basically until the full sync is complete before it emits the state messages. When users have large syncs that might only make partial progress it would be very helpful to save checkpoints for the records that were processed so the next sync can complete quicker.

My understanding of whats happening today:

• the target is iterating jsonl input via _process_lines and calls the appropriate message type method, in this case the two important messages are RECORD and STATE
• RECORD
  • the target _process_record_messages is sending records to the sink implementation until the sink tells it that its full
  • once its full drain_one is called to execute the sink implementation's batch process step (i.e. write batch to destination).
  • then it continues processing records
• SCHEMA
  • the target receives state messages and just stores them in a dict. Overwriting the state message when new state messages arrive, so its always the most recent state message.
  • the only place that I see the schema messages being emitted is in the drain_all method
  • drain_all is called by _handle_max_record_age when an amount of time has passed and the target decides it should drain. I think this is defaulted to every 5 mins
  • and its called at the end of the pipe in _process_endofpipe meaning the full sync is completed

After reviewing this the arbitrary 5 mins drain should handle emitting state frequently for large syncs to get sufficient checkpoints.