Importing into 6.6.2 ElasticSearch

[mikeguidry]

I'll admit I am new to ElasticSearch... and I'm willing to remove/close the issue by stating I'm wrong...

I am trying to setup connbeat on ElasticSearch 6.6.2... I am using docker-elk, and getting to know the framework(s). I am trying to use connbeat for per connection status rather than metricbeat, or packetbeat's alternatives. I am not familiar however with dashboard schema, etc. I tried import---dashboards but am getting to this error message:

Initialize the Elasticsearch 6.6.2 loader
Elasticsearch URL http://127.0.0.1:9200
For Elasticsearch version >= 6.0.0, the Kibana dashboards need to be imported via the Kibana API.

It sort of feels like learning MySQL again, and I understand that the software cannot handle all known/future cases but I figured I'd open up a ticket to show that in comparison to packetbeat, and metricbeat that it is an issue to get going...

[raboof]

For Elasticsearch version >= 6.0.0, the Kibana dashboards need to be imported via the Kibana API

I'm not familiar with ES's dashboard import feature, so I don't think I can help you there

I figured I'd open up a ticket to show that in comparison to packetbeat, and metricbeat that it is an issue to get going

Is this different from packetbeat and metricbeat? Or do those have dashboards 'builtin' somehow?

[mikeguidry]

Alright.. So. It was my first few days playing with elastic search. I am 90+ hours of playing with the framework + a ton of other utilities. I am happy to say I understand the majority of it now. I also had to learn some go since I never had a reason to play with it before.

Ok.. so doing connbeat -setup had some issues regarding older 5.9.2, or other elastic search directly in the JSONs.. I had sed replaced those. The log file had some issues regarding insertions. The JSON after I changed to 6.6.2 was attempting to use a mapping whereas the insertion was sending a string. I am unsure (I didn't check git history etc) whether it was some change that happened randomly, or if you stopped caring about the template/index all together. either way, I still wanted to take advantage of all of the other code :)..

I went into the source where it says connbeat.go: processAsMap. It was being called from exportFullConnection (for local_process which was being mapped improperly), and I just changed to string. I hoped it would work although I wasn't getting any decent results.

I realized how this processes_linux.go uses /proc/pid/fd and socket: symlinks to determine which process is using the specific socket from /proc/net/tcp... anyways, it wasn't filling the "Binary" portion of the structure so the exe wasn't getting returned.

I made a super hackish situation:
func getexe(pid int) string {
path := fmt.Sprintf("%s/%d/%s", "/proc", pid, "exe")
exe, err := os.Readlink(path)
if os.IsNotExist(err) {
return ""
}

return exe

}

func retprocname(process *processes.UnixProcess) string {
exe := getexe(process.Pid())
return exe
}

and used:
"local_process": retprocname(s.Process),
in exportServerConnection, and exportFullConnection... It differs from the map but I am not familiar enough with elastic search to yet manipulate indexes, or templates to that degree.. maybe next week ;)

I noticed something in the go vendor dir as as procfs/proc.go has Executable() but the strings related to the library I was already playing with too closely to quickly follow through.. I just decided to use the hack above to get some stats going so I can determine if I liked, or wanted to keep the information.

In the end: I'd love to send a patch, or pull request however I am day #1 on GO, so definitely not efficient enough to submit decent code for pulling... I just figured I'd write what happened afterwards since I didn't respond. If I end up tidying up, or using the code further with template, or index modifications then by then I'd probably be familiar enough with libbeats, and go to submit a pull.

I was unhappy that libbeats doesn't have a C/C++ version but I decided to create a sysdig libbeats importer to handle some other information I need so if I end up using this stuff long term then I'll hop back over here to give a patch, or update. If not, then assume I had to move on, but I hope the information helps anyone following through...

have a great day, and thanks for responding

[raboof]

I'd love to send a patch, or pull request however I am day #1 on GO, so definitely not efficient enough to submit decent code for pulling... I just figured I'd write what happened afterwards since I didn't respond.

Totally understood, thanks for sharing!

If I end up tidying up, or using the code further with template, or index modifications then by then I'd probably be familiar enough with libbeats, and go to submit a pull.

Looking forward to it 😄