Dynamic field values in prefix is not supported

[nigoel]

Creation of different file paths in s3 for different kind of events or tracking log file is not supported.

For example, here folder prefixes based on fields "app" and "type" are not supported.
input {
file {
path => "//data//executors//logs/"
}
}

filter {
grok { match => ["path","/%{GREEDYDATA}/executors/%{GREEDYDATA:app}[.]%{GREEDYDATA}/logs/%{GREEDYDATA:type}"]
}
}
output {
s3{
.....................
.....................
prefix => "applogs/%{app}/%{type}"
}
}

[nigoel]

Fix has been provided in https://github.com/nigoel/logstash-output-s3

[Onalu]

I'd like to see dynamic values on prefix too. I want to collect my logs in daily folders and there is no option for it now.

@nigoel maybe you should make a pull request since you already implemented a solution.

[nigoel]

@Onalu Here it is #17

[shaharmor]

+1

[vaijab]

Any plans on implementing this?

[BDuelz]

+1

[obazoud]

+1

[zot420]

+1

[NoumanSaleem]

Would love this for all other string values like bucket

[cero-t]

+1

[beeva-luisgonzalez]

Please merge it!

[ghost]

Please merge it!

[ghost]

@nigoel would you pull the latest version and merge your changes? So your repo is up-to-date. That would be awesome!! I've tried, but got conflicts I couldn't solve.

[ghost]

@nigoel I am using your version of the plugin as I needed dynamic prefixes. Workers stop after they've uploaded once. It seems need_cleanup? throws an error if @empty_downloads[prefix] is nil.
I couldn't create a pull request for your repository hence posting it here. The fix should look like this (or similar, I am not a ruby programmer):

  public
  def need_cleanup?(prefix)
    if @empty_uploads[prefix].nil?
      return false
    end
    return @empty_uploads[prefix] > @no_event_wait
  end

[asemt]

+1

[s4nch3z]

+1

[dgeebs]

+1

[donovankim]

+1

[wagner]

+1

[vaijab]

Adding +1 does not do anything, just spams us all.

[ph]

I am closing this issue in favor of #4, so all the discussion can happen in the same place.
I have started on working on that feature.