page_counter indexes up when a time_file event is triggered

[lremurphy]

This is in regards to the logstash-output-s3-2.0.7

When dynamically creating dated subfolders in the s3 bucket (yyyy/mm/dd)

It would be observed that the size_file part# would increment when a time_file is triggered:

example with 10 minute time_file rotation s3 file names:

...
ls.s3.vm.test.local.2015-08-25T15.33.part34.txt
ls.s3.vm.test.local.2015-08-25T15.43.part35.txt
ls.s3.vm.test.local.2015-08-25T15.53.part36.txt
...

It appears the fix for this (on my end) is to change the "configure_periodic_rotation" function and replace "next_page" with "reset_page_counter"

 private
  def configure_periodic_rotation
    @periodic_rotation_thread = Stud::Task.new do
      LogStash::Util::set_thread_name("<S3 periodic uploader")

      Stud.interval(periodic_interval, :sleep_then_run => true) do
        @logger.debug("S3: time_file triggered, bucketing the file", :filename => @tempfile.path)

        move_file_to_bucket_async(@tempfile.path)

        reset_page_counter
        create_temporary_file
      end
    end
  end

[ph]

Yes, it currently increment the page on both time_file or size_file, it's currently does that for backward compatibility with previous version of the plugins.

I have started work on #4, which would allow dynamic prefix on the file.

[lremurphy]

No worries, I can change it manually on my end; just wasn't certain this was expected behavior and as an observer it didn't appear to be the expected behavior according to the documentation. Thank you for updating this and I am very interested to see what you come up with for dynamic prefixing!

[ph]

Just to clarify this issue, this is the intended behavior every rotation will trigger a page increment.

[FlorinAndrei]

@ph That should be clarified in the documentation. We ran into some nasty issues here with scripts importing files from S3 because, based on the documentation, we thought the part segment is not appended to the name when only using time_file.