vector halts after an error from the buffer

[canopenerda]

Hi folks,

I have a vector pipeline sourcing data from kafka and persisting data to clickhouse, with hyrid buffer setting. The pipelines halts processing after following error. Restarting vector resumes the processing.

error

{"log":{"error":"The reader detected that a data file contains a partially-written record.","error_code":"partial_write","error_type":"reader_failed","host":"96e4807a14f8","internal_log_rate_limit":true,"message":"Error encountered during buffer read.","metadata":{"kind":"event","level":"ERROR","module_path":"vector_buffers::internal_events","target":"vector_buffers::internal_events"},"pid":1,"source_type":"internal_logs","stage":"processing","timestamp":"2024-12-14T22:11:23.461122537Z","vector":{"component_id":"xx_sink_clickhouse","component_kind":"sink","component_type":"clickhouse"}}}

buffer setting

{
      "buffer": [
        {
          "type": "memory",
          "max_events": 250000,
          "when_full": "overflow"
        },
        {
          "type": "disk",
          "max_size": 1073741824
        }
      ]
{

[jszwedko]

I think this issue has been reported here: #19379

[canopenerda]

@jszwedko Thanks! The symptom is similar. I'm sure I have ample storage for disk buffer and to ensure overall stability of the box, I put a limit on the buffer size. I guess when the disk buffer approaches the limit and finally filled or breached, it should block instead of I/O exception in this case.

[canopenerda]

@jszwedko Sorry to bother you again. Today we faced data loss when using hybrid buffer. I checked the issue you sent. It seems we haven't found the root cause yet. I have one question, if I stop vector, will vector drain the buffer before it exits if enough time is allowed for it? I'm thinking of switching back to pure memory/disk buffer, preferrably disk.

[jszwedko]

It should try to drain the buffer when shutting down, yes. By default it will wait 60s but this is configurable via https://vector.dev/docs/reference/cli/#vector_graceful_shutdown_limit_secs

[canopenerda]

That's cool to know this flag. I ran vector in docker and when I stop/restart I used --timeout from docker command which is cap by this flag. Thanks for your prompt replies during the holiday season.

[canopenerda]

@jszwedko Sincerely sorry to DM you again after such a short time. After perusing through all relevant issues with disk buffer, it seems it's not ready for large volume deployment. As I mentioned above, I just switched back to pure disk buffer. Now I feel the only safe option is to use memory buffer. However, if disk buffer is not drained when stopping vector or consumed when vector restarts with pure memory buffer settings, it means data loss. Would you please kindly advice the correct way to swtich from disk buffer to memory buffer? Thank you!

[jszwedko]

I think you would need to start Vector with the disk buffer configuration and then shut it down. As mentioned in #22034 (reply in thread), if I remember right, Vector should try to drain the buffers before shutting down.

[canopenerda]

some data files created before were not acknowledged and it seems they will never get acknowledged, which also indicate data loss.

[canopenerda]

@jszwedko would you mind elaborate on in what circumstances data would be no longer acknowledged and is it possible to rescue the data at your convenience?

[jszwedko]

would be no longer acknowledged

I'm not sure I understand this question. Could you rephrase?

possible to rescue the data

As mentioned in the above threads, you can start Vector and then tell it to shutdown to have it drain the disk buffers.

[canopenerda]

@jszwedko When client reported data loss, and the only change I made was switching from pure memory buffer to a hybrid one and adjusting the disk buffer sizes, I checked buffer data files for the pipeline. As you can see in the screeshot I attached in the thread above, the data files are generated in different days, even in Novermber. It's not likely those files created in Novermber would ever be processed since there were several restarts to adjust the buffer settings.

[jszwedko]

Ah, I see. It is expected that the files stick around until all data from them has been flushed (after which point they should be "garbage collected").

[canopenerda]

@jszwedko I just took another screenshot of the same directory. As you can see, some of the old data files are still there which means there were data loss if they were not properly processed. As you mentioned above, buffer should be drained when vector restarts, and it means these data files were not properly handled during restarts and it's not likely that they would be processed in the future.

The pipeline stuck again, and I checked the buffer directory before and after restart and found one file highlighted in the screenshot was always present.