Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can hdfs path use ${tag} or ${record} ? just like: /data/hdfs/${tag}.#{Socket.gethostname}.log #64

Open
monkeypiggy opened this issue Sep 25, 2017 · 1 comment

Comments

@monkeypiggy
Copy link

monkeypiggy commented Sep 25, 2017

Hello,
I use 'fluent-plugin-webhdfs' version '1.2.2', fluentd version: 1.4
My log file looks like:
interface=login&time=20170925&...others fields...
In my td-agent config file, I named my log as a tag:interface.login.hdfs

<source>
  @type tail
  format none
  path /data/td-agent/logs/interface.log
  pos_file /etc/td-agent/pos/interface.pos
  read_from_head true
  tag interface.**login**.hdfs
</source>
<match interface.**>
  @type webhdfs
  host *.*.*.*
  port 50070
  path /data/gamelogs/hdfs/date=${record['time']}/${tag[2]}.#{Socket.gethostname}.log
  <format>
    @type json
  </format>
</match>

and I need to get "login" from tag, and get time from my log record, so if there has any method to get these two values?

In some other plugins, I can use $tag[2] or ${record['interface']}, ${record['time']} to get these values,but in this plugin, it doesn't work.
Thanks...

@monkeypiggy monkeypiggy changed the title Can hdfs path use tag or record? just like: /data/hdfs/${tag}.#{Socket.gethostname}.log Can hdfs path use ${tag} or ${record} ? just like: /data/hdfs/${tag}.#{Socket.gethostname}.log Sep 25, 2017
@JiHyunSong
Copy link
Contributor

@monkeypiggy hello, I met this problem, too.

I solved it add some configuration.. like below

<match **>
  @type webhdfs
  ----
  path   "/data/${tag}/%Y%m%d/%H/#{Socket.gethostname}.${chunk_id}.log"
  <buffer tag,time>
  </buffer>
</match>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants