No visuals on Grafna dashboard for Firewall Logs

[panks21]

Hi
I have set this up on a Ubuntu 18.04 and graylog is receiving the logs from the pfsense. However, I don't see anything on the Grafana dashboard. I also tested the data source pfsensefw (http://elasticsearch:9200) and it doesn't complain.

Following is the log output from the CLI and screenshots from Grafana.

tcpdump -i ens160 not port 22 | grep 5442
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
23:18:26.196359 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:26.313221 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local0.info, length: 195
23:18:26.338964 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 276
23:18:27.190880 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:28.193809 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:29.196005 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:29.240726 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 311
23:18:29.332991 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local0.info, length: 183
23:18:30.195597 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:31.197824 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:31.242791 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 301
23:18:32.195049 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:32.374036 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local0.info, length: 183
23:18:33.193469 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:34.193956 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:34.252609 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 299
23:18:35.190906 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275

[jakegibbonsdev]

I am curious as to whether it is a timezone issue. Did you try changing the timerange to the difference in UTC? For me, I need to specify "last 6 hours" to include UTC - 5.

[panks21]

I did.. except pfsense.. the browser, host OS and graylog docker.. all in UTC timezone

[wintrmte]

Are you using PF Sense 2.5 by chance? I upgraded mine to 2.5 and am having the same exact issue. I haven't dug into the problem yet.

[panks21]

Yes.. I was testing it
It was a fresh install. Moved to OPNSense recently

[jakegibbonsdev]

It seems like it may be a different issue, but most of my PFSense 2.5/Graylog issues were solved by setting everything to a common timezone.

graylog.env:
GRAYLOG_ROOT_TIMEZONE=America/Toronto
GRAYLOG_TIMEZONE=America/Toronto

graylog/Dockerfile:
ENV TZ=America/Toronto
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

[nandonov]

I am fighting with the same issue, so far the DPI portion is working fine.
In graylog I can see the messages, the elastic search seems to work fine too, the issue is something on Grafana, unless this is what I can see for now. I am not sure timezone can be related to it, I also adjusted the timezone on Grafana to make sure I am on the right timezone and I cannot see data yet. Will let you know if I find something.

[nandonov]

I think I find the issue, seems like a permission problem on the dashboard. I exported the dashboard and imported it to change the permissions to the following:

Once I did, I was able to see the data:

Hope this helps!

[panks21]

Thanks will try this later this week

[nandonov]

I was probably too quick with the assumptions. For some reason, I had a small quantity of logs I was able to see, maybe during my tests I had a batch of logs which got to the Grafana and I was able to see it. I am unable to see new logs though, I will keep digging and let you know if I find something, this time I will test it for few days before share it :)

[t-jones14]

I am running pfSense 2.5 and having this same issue. I haven't had much luck finding the issue.

I can see messages coming into the stream in Graylog, but can't see them for some reason. See attached.

[nandonov]

I have reviewed the graylog configuration and details. I am new to graylog, but I used Kabana with logstash few years ago and there is some simlarities. I have found that there is no fields extracted from the logs, I am not sure if it is the same for you, but witout this portion Grafana wont work for sure. Can someone confirm there fields you have on the left while you are on the search window:

This is how it should be:

Thanks!

[t-jones14]

@nandonov This is what I see.

In addition, you're further along than I am... I don't even see my messages populating in Graylog even though Graylog IS receiving them...

[nandonov]

Ok, it seems to work for you, the issue should be on my side then.

[nandonov]

About the logs, you need to select "Search in all messages" on the filter, see the screenshot:

I found the issue, it was the content pack, I used the one provided by devopstales, it has all fields:

https://github.com/devopstales/pfsense-graylog/tree/master/pfsense_content_pack/graylog3

[nandonov]

Ok, I was able to make it work but followed 3 different guides. In this repo there is a lots of effort to automate some of the boring stuff but it makes it complex to troubleshoot after that. I have deleted everything and followed the devopstales guides. There is a tricky part there too, as he is implementing the components on 3 different giudes and you need to pay attention as you can miss some of the key components (I did forgot to install some of the requested plugins for Grafana, the instruction of which is not on the repo). So here is the giudes I followed:

Infrastructure part:
https://devopstales.github.io/home/graylog4-install/

Cerebro installation, Graylog setup, Grafana setup:
https://devopstales.github.io/home/graylog4-pfsense/

Finally, I have followed the devopstales repo bellow:

https://github.com/devopstales/pfsense-graylog

Everything is working now, I hope this will help someone.

Have a nice one!

[nandonov]

Oh, yeah, you need to edit one of the panels as the preconfigured value is not supported, so the panel is not showing anything. The name of the panel is "Real time logs by iface:$iface", the bottom one. Just edit the panel and lower the value to 10 000, like this:

[t-jones14]

Ok, I was able to make it work but followed 3 different guides. In this repo there is a lots of effort to automate some of the boring stuff but it makes it complex to troubleshoot after that. I have deleted everything and followed the devopstales guides. There is a tricky part there too, as he is implementing the components on 3 different giudes and you need to pay attention as you can miss some of the key components (I did forgot to install some of the requested plugins for Grafana, the instruction of which is not on the repo). So here is the giudes I followed:

Infrastructure part:
https://devopstales.github.io/home/graylog4-install/

Cerebro installation, Graylog setup, Grafana setup:
https://devopstales.github.io/home/graylog4-pfsense/

Finally, I have followed the devopstales repo bellow:

https://github.com/devopstales/pfsense-graylog

Everything is working now, I hope this will help someone.

Have a nice one!

Thank you! @nandonov

I will follow along with these now and report back.

[t-jones14]

Nevermind.

Though my fix below managed to get the logs to pass into Grafana, there aren't any timestamps now...

~~ #### Fix Time Parse Error ~~
I was able to fix below issue by editing the pipeline rule as follows:

~~ ###### Original Pipeline Rule ~~

rule "timestamp_pfsense_for_grafana"
     when
     has_field("timestamp")
    then
    // the following date format assumes there's no time zone in the string
     let source_timestamp = parse_date(substring(to_string(now("America/Habana")),0,23), "yyyy-MM-dd'T'HH:mm:ss.SSS");
     let dest_timestamp = format_date(source_timestamp,"yyyy-MM-dd HH:mm:ss");
     set_field("real_timestamp", dest_timestamp);
    end

~~ ###### Change pipeline rule to: ~~

rule "timestamp_pfsense_for_grafana"
     when
     has_field("timestamp")
    then
    // the following date format assumes there's no time zone in the string
     let source_timestamp = parse_date(substring(to_string(now("America/Habana")),0,23), "yyyy-MM-dd'T'HH:mm:ss");
     let dest_timestamp = format_date(source_timestamp,"yyyy-MM-dd HH:mm:ss");
     set_field("real_timestamp", dest_timestamp);
    end

@nandonov I followed the devopstales guides to a T and am now receiving the following:

The timestamp format is correct, not sure why ELK can't parse it. Any ideas?