Falcosidekick

Description

A simple daemon for enhancing available outputs for Falco. It takes a falco's event and forwards it to different outputs.

It works as a single endpoint for as many as you want falco instances :

Outputs

Currently available outputs are :

• Slack
• Rocketchat
• Mattermost
• Teams
• Datadog
• Discord
• AlertManager
• Elasticsearch
• Loki
• NATS
• Influxdb
• AWS Lambda
• AWS SQS
• AWS SNS
• SMTP (email)
• Opsgenie
• StatsD (for monitoring of falcosidekick)
• DogStatsD (for monitoring of falcosidekick)
• Webhook
• Azure Event Hubs

Usage

Run the daemon as any other daemon in your architecture (systemd, k8s daemonset, swarm service, ...)

With docker

docker run -d -p 2801:2801 -e SLACK_WEBHOOKURL=XXXX -e DATADOG_APIKEY=XXXX falcosecurity/falcosidekick

With Helm

git clone https://github.com/falcosecurity/falcosidekick.git
cd ./falcosidekick/deploy/helm/falcosidekick/
helm install --name falcosidekick .

Falco's config

If installing falco with helm, set this (adapted to your environment) in your values.yaml :

jsonOutput: true
jsonIncludeOutputProperty: true
httpOutput:
  enabled: true
  url: "http://localhost:2801/"

or

jsonOutput: true
jsonIncludeOutputProperty: true
programOutput:
  enabled: true
  keepAlive: false
  program: "curl -d @- localhost:2801/"

If managing falco.yaml manually, set this:

json_output: true
json_include_output_property: true
http_output:
  enabled: true
  url: "http://localhost:2801/"

Configuration

Configuration is made by file (yaml) and env vars, both can be used but env vars override values from file.

YAML File

See config_example.yaml :

#listenport: 2801 # port to listen for daemon (default: 2801)
debug: false # if true all outputs will print in stdout the payload they send (default: false)
customfields: # custom fields are added to falco events
  Akey: "AValue"
  Bkey: "BValue"
  Ckey: "CValue"
checkCert: true # check if ssl certificate of the output is valid (default: true)

slack:
  webhookurl: "" # Slack WebhookURL (ex: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ), if not empty, Slack output is enabled
  #footer: "" # Slack footer
  #icon: "" # Slack icon (avatar)
  #username: "" # Slack username (default: Falcosidekick)
  outputformat: "all" # all (default), text, fields
  minimumpriority: "debug" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
  messageformat: "Alert : rule *{{ .Rule }}* triggered by user *{{ index .OutputFields \"user.name\" }}*" # a Go template to format Slack Text above Attachment, displayed in addition to the output from `SLACK_OUTPUTFORMAT`, see [Slack Message Formatting](#slack-message-formatting) in the README for details. If empty, no Text is displayed before Attachment.

rocketchat:
  webhookurl: "" # Rocketchat WebhookURL (ex: http://XXXX/hooks/YYYY), if not empty, Rocketchat output is enabled
  #icon: "" # Rocketchat icon (avatar)
  #username: "" # Rocketchat username (default: Falcosidekick)
  outputformat: "all" # all (default), text, fields
  minimumpriority: "debug" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
  # messageformat: "Alert : rule *{{ .Rule }}* triggered by user *{{ index .OutputFields \"user.name\" }}*" # a Go template to format Rockatchat Text above Attachment, displayed in addition to the output from `ROCKETCHAT_OUTPUTFORMAT`, see [Slack Message Formatting](#slack-message-formatting) in the README for details. If empty, no Text is displayed before Attachment.

mattermost:
  webhookurl: "" # Mattermost WebhookURL (ex: http://XXXX/hooks/YYYY), if not empty, Mattermost output is enabled
  #footer: "" # Mattermost footer
  #icon: "" # Mattermost icon (avatar)
  #username: "" # Mattermost username (default: Falcosidekick)
  outputformat: "all" # all (default), text, fields
  minimumpriority: "debug" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
  # messageformat: "Alert : rule **{{ .Rule }}** triggered by user **{{ index .OutputFields \"user.name\" }}**" # a Go template to format Mattermost Text above Attachment, displayed in addition to the output from `MATTERMOST_OUTPUTFORMAT`, see [Slack Message Formatting](#slack-message-formatting) in the README for details. If empty, no Text is displayed before Attachment.

teams:
  webhookurl: "" # Teams WebhookURL (ex: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ), if not empty, Teams output is enabled
  #activityimage: "" # Image for message section
  outputformat: "text" # all (default), text, facts
  minimumpriority: "debug" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

datadog:
  # apikey: "" # Datadog API Key, if not empty, Datadog output is enabled
  # host: "" # Datadog host. Override if you are on the Datadog EU site. Defaults to american site with "https://api.datadoghq.com"
  # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

alertmanager:
  # hostport: "" # http://{domain or ip}:{port}, if not empty, Alertmanager output is enabled
  # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

elasticsearch:
  # hostport: "" # http://{domain or ip}:{port}, if not empty, Elasticsearch output is enabled
  # index: "falco" # index (default: falco)
  # type: "event"
  # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
  # suffix: "daily" # date suffix for index rotation : daily (default), monthly, annually, none

influxdb:
  # hostport: "" # http://{domain or ip}:{port}, if not empty, Influxdb output is enabled
  # database: "falco" # Influxdb database (default: falco)
  # user: "" # user to use if auth is enabled in Influxdb
  # password: "" # pasword to use if auth is enabled in Influxdb
  # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

loki:
  # hostport: "" # http://{domain or ip}:{port}, if not empty, Loki output is enabled
  # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

nats:
  # hostport: "" # nats://{domain or ip}:{port}, if not empty, NATS output is enabled
  # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

aws:
  # accesskeyid: "" # aws access key (optionnal if you use EC2 Instance Profile)
  # secretaccesskey: "" # aws secret access key (optionnal if you use EC2 Instance Profile)
  # region : "" # aws region (optionnal if you use EC2 Instance Profile)
  lambda:
    # functionname : "" # Lambda function name, if not empty, AWS Lambda output is enabled
    # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
  sqs:
    # url : "" # SQS Queue URL, if not empty, AWS SQS output is enabled
    # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
  sns:
    # topicarn : "" # SNS TopicArn, if not empty, AWS SNS output is enabled
    rawjson: false # Send Raw JSON or parse it (default: false)
    # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

smtp:
  # hostport: "" # host:port address of SMTP server, if not empty, SMTP output is enabled
  # user: "" # user to access SMTP server
  # password: "" # password to access SMTP server
  # from: "" # Sender address (mandatory if SMTP output is enabled)
  # to: "" # comma-separated list of Recipident addresses, can't be empty (mandatory if SMTP output is enabled)
  # outputformat: "" # html (default), text
  # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

statsd:
  forwarder: "" # The address for the StatsD forwarder, in the form "host:port", if not empty StatsD is enabled
  namespace: "falcosidekick." # A prefix for all metrics (default: "falcosidekick.")

dogstatsd:
  forwarder: "" # The address for the DogStatsD forwarder, in the form "host:port", if not empty DogStatsD is enabled
  namespace: "falcosidekick." # A prefix for all metrics (default: "falcosidekick.")
  # tag :
  #   key: "value"

opsgenie:
  # apikey: "2c771471-e2af-4dc6-bd35-e7f6ff479b64" # Opsgenie API Key, if not empty, Opsgenie output is enabled
  region: "eu" # (us|eu) region of your domain
  # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

webhook:
  # address: "" # Webhook address, if not empty, Webhook output is enabled
  # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

azure:
  # eventHub:
    # name: "" # The name of the Hub, if not empty, EventHub output is enabled
    # namespace: "" # The name of the space the Hub is part of
    # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

discord:
  webhookurl: "" # discord WebhookURL (ex: https://discord.com/api/webhooks/xxxxxxxxxx...), if not empty, Discord output is enabled
  # icon: "" # Discord icon (avatar)
  # minimumpriority: "debug" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

Usage :

usage: falcosidekick [<flags>]

Flags:
      --help                     Show context-sensitive help (also try --help-long and --help-man).
  -c, --config-file=CONFIG-FILE  config file

Env vars

Configuration of the daemon can be made also by env vars, these values override these from yaml file.

The env vars "match" field names in *yaml file with this structure (take care of lower/uppercases) : yaml: a.b --> envvar: A_B :

• LISTENPORT : port to listen for daemon (default: 2801)
• DEBUG : if true all outputs will print in stdout the payload they send (default: false)
• CUSTOMFIELDS : a list of comma separated custom fields to add to falco events, syntax is "key:value,key:value"
• CHECKCERT: check if ssl certificate of the output is valid (default: true)
• SLACK_WEBHOOKURL : Slack Webhook URL (ex: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ), if not empty, Slack output is enabled
• SLACK_FOOTER : Slack footer
• SLACK_ICON : Slack icon (avatar)
• SLACK_USERNAME : Slack username (default: Falcosidekick)
• SLACK_OUTPUTFORMAT : all (default), text (only text is displayed in Slack), fields (only fields are displayed in Slack)
• SLACK_MINIMUMPRIORITY : minimum priority of event for using use this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• SLACK_MESSAGEFORMAT : a Go template to format Slack Text above Attachment, displayed in addition to the output from SLACK_OUTPUTFORMAT, see Slack Message Formatting in the README for details. If empty, no Text is displayed before Attachment.
• ROCKETCHAT_WEBHOOKURL : Rocketchat Webhook URL (ex: https://XXXX/hooks/YYYY), if not empty, Rocketchat output is enabled
• ROCKETCHAT_ICON : Rocketchat icon (avatar)
• ROCKETCHAT_USERNAME : Rocketchat username (default: Falcosidekick)
• ROCKETCHAT_OUTPUTFORMAT : all (default), text (only text is displayed in Rocketchat), fields (only fields are displayed in Rocketchat)
• ROCKETCHAT_MINIMUMPRIORITY : minimum priority of event for using use this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• ROCKETCHAT_MESSAGEFORMAT : a Go template to format Rocketchat Text above Attachment, displayed in addition to the output from ROCKETCHAT_OUTPUTFORMAT, see Slack Message Formatting in the README for details. If empty, no Text is displayed before Attachment.
• MATTERMOST_WEBHOOKURL : Mattermost Webhook URL (ex: https://XXXX/hooks/YYYY), if not empty, Mattermost output is enabled
• MATTERMOST_FOOTER : Mattermost footer
• MATTERMOST_ICON : Mattermost icon (avatar)
• MATTERMOST_USERNAME : Mattermost username (default: Falcosidekick)
• MATTERMOST_OUTPUTFORMAT : all (default), text (only text is displayed in Mattermost), fields (only fields are displayed in Mattermost)
• MATTERMOST_MINIMUMPRIORITY : minimum priority of event for using use this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• MATTERMOST_MESSAGEFORMAT : a Go template to format Mattermost Text above Attachment, displayed in addition to the output from MATTERMOST_OUTPUTFORMAT, see Mattermost Message Formatting in the README for details. If empty, no Text is displayed before Attachment.
• TEAMS_WEBHOOKURL : Teams Webhook URL (ex: https://outlook.office.com/webhook/XXXXXX/IncomingWebhook/YYYYYY"), if not empty, Teams output is enabled
• TEAMS_ACTIVITYIMAGE : Teams section image
• TEAMS_OUTPUTFORMAT : all (default), text (only text is displayed in Teams), facts (only facts are displayed in Teams)
• TEAMS_MINIMUMPRIORITY : minimum priority of event for using use this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• DATADOG_APIKEY : Datadog API Key, if not empty, Datadog output is enabled
• DATADOG_HOST : Datadog host. Override if you are on the Datadog EU site. Defaults to american site with "https://api.datadoghq.com"
• DATADOG_MINIMUMPRIORITY : minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• DISCORD_WEBHOOKURL : Discord WebhookURL (ex: https://discord.com/api/webhooks/xxxxxxxxxx...), if not empty, Discord output is enabled
• DISCORD_ICON : Discord icon (avatar)
• DISCORD_MINIMUMPRIORITY : minimum priority of event for using use this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• ALERTMANAGER_HOSTPORT : AlertManager http://host:port, if not empty, AlertManager is enabled
• ALERTMANAGER_MINIMUMPRIORITY : minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• ELASTICSEARCH_HOSTPORT : Elasticsearch http://host:port, if not empty, Elasticsearch is enabled
• ELASTICSEARCH_INDEX : Elasticsearch index (default: falco)
• ELASTICSEARCH_TYPE : Elasticsearch document type (default: event)
• ELASTICSEARCH_MINIMUMPRIORITY : minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• ELASTICSEARCH_SUFFIX : date suffix for index rotation : daily (default), monthly, annually, none
• INFLUXDB_HOSTPORT : Influxdb http://host:port, if not empty, Influxdb is enabled
• INFLUXDB_DATABASE : Influxdb database (default: falco)
• INFLUXDB_USER : user to use if auth is enabled in Influxdb
• INFLUXDB_PASSWORD : user to use if auth is enabled in Influxdb
• INFLUXDB_MINIMUMPRIORITY : minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• LOKI_HOSTPORT : Loki http://host:port, if not empty, Loki is enabled
• LOKI_MINIMUMPRIORITY : minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• NATS_HOSTPORT : NATS "nats://host:port", if not empty, NATS is enabled
• NATS_MINIMUMPRIORITY : minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• AWS_ACCESSKEYID : AWS Access Key Id (optionnal if you use EC2 Instance Profile)
• AWS_SECRETACCESSKEY : AWS Secret Access Key (optionnal if you use EC2 Instance Profile)
• AWS_REGION : AWS Region (optionnal if you use EC2 Instance Profile)
• AWS_LAMBDA_FUNCTIONNAME : AWS Lambda Function Name, if not empty, AWS Lambda output is enabled
• AWS_LAMBDA_MINIMUMPRIORITY : minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• AWS_SQS_URL : AWS SQS Queue URL, if not empty, AWS SQS output is enabled
• AWS_SQS_MINIMUMPRIORITY : minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• AWS_SNS_TOPICARN : AWS SNS TopicARN, if not empty, AWS SNS output is enabled
• AWS_SNS_RAWJSON : Send Raw JSON or parse it (default: false)
• AWS_SNS_MINIMUMPRIORITY : minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• SMTP_HOSTPORT : "host:port" address of SMTP server, if not empty, SMTP output is enabled
• SMTP_USER : user to access SMTP server
• SMTP_PASSWORD : password to access SMTP server
• SMTP_FROM : Sender address (mandatory if SMTP output is enabled)
• SMTP_TO : comma-separated list of Recipident addresses, can't be empty (mandatory if SMTP output is enabled)
• SMTP_OUTPUTFORMAT : "" # html (default), text
• SMTP_MINIMUMPRIORITY : minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• OPSGENIE_APIKEY : Opsgenie API Key, if not empty, Opsgenie output is enabled
• OPSGENIE_REGION : "" # (us|eu) region of your domain (default is 'us')
• OPSGENIE_MINIMUMPRIORITY : minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• STATSD_FORWARDER: The address for the StatsD forwarder, in the form http://host:port, if not empty StatsD is enabled
• STATSD_NAMESPACE: A prefix for all metrics (default: "falcosidekick.")
• DOGSTATSD_FORWARDER: The address for the DogStatsD forwarder, in the form http://host:port, if not empty DogStatsD is enabled
• DOGSTATSD_NAMESPACE: A prefix for all metrics (default: falcosidekick."")
• DOGSTATSD_TAGS: A comma-separated list of tags to add to all metrics
• WEBHOOK_ADDRESS : "" # Webhook address, if not empty, Webhook output is enabled
• WEBHOOK_MINIMUMPRIORITY : minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
• AZURE_EVENTHUB_NAME: Name of the Hub, if not empty, EventHub is enabled
• AZURE_EVENTHUB_NAMESPACE: Name of the space the Hub is in
• AZURE_EVENTHUB_MINIMUMPRIORITY: minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)

Slack/Rocketchat/Mattermost Message Formatting

The SLACK_MESSAGEFORMAT environment variable and slack.messageformat YAML value accept a Go template which can be used to format the text of a slack alert. These templates are evaluated on the JSON data from each Falco event - the following fields are available:

Template Syntax	Description
{{ .Output }}	A formatted string from Falco describing the event.
{{ .Priority }}	The priority of the event, as a string.
{{ .Rule }}	The name of the rule that generated the event.
{{ .Time }}	The timestamp when the event occurred.
{{ index .OutputFields \"<field name>\" }}	A map of additional optional fields emitted depending on the event. These may not be present for every event, in which case they expand to the string <no value>

Go templates also support some basic methods for text manipulation which can be used to improve the clarity of alerts - see the documentation for details.

Handlers

Different URI (handlers) are available :

• / : main and default handler, your falco config must be configured to use it
• /ping : you will get a pong as answer, useful to test if falcosidekick is running and its port is opened (for healthcheck purpose for example)
• /test : (for debug only) send a test event to all enabled outputs.
• /debug/vars : get statistics from daemon (in JSON format), it uses classic expvar package and some custom values are added

Logs

All logs are sent to stdout.

2019/05/10 14:32:06 [INFO] : Enabled Outputs : Slack Datadog

Metrics

Golang ExpVar

The daemon exposes the common Golang metrics and some custom values in JSON format. It's useful for monitoring purpose.

StatsD

The daemon is able to push its metrics to a StatsD server. See Configuration section for how-to.

Examples

Run you daemon and try (from falco's documentation) :

curl "http://localhost:2801/" -d'{"output":"16:31:56.746609046: Error File below a known binary directory opened for writing (user=root command=touch /bin/hack file=/bin/hack)","priority":"Error","rule":"Write below binary dir","time":"2019-05-17T15:31:56.746609046Z", "output_fields": {"evt.time":1507591916746609046,"fd.name":"/bin/hack","proc.cmdline":"touch /bin/hack","user.name":"root"}}'

You should get :

Slack

(SLACK_OUTPUTFORMAT="all")

(SLACK_OUTPUTFORMAT="text")

(SLACK_OUTPUTFORMAT="fields" and SLACK_MESSAGEFORMAT="Alert : rule *{{ .Rule }}* triggered by user *{{ index .OutputFields "user.name" }}*")

Mattermost

Teams

(TEAMS_OUTPUTFORMAT="all")

(TEAMS_OUTPUTFORMAT="text")

Datadog

(Tip: filter on sources: falco)

AlertManager

Elasticsearch (with Kibana)

Influxdb

> use falco
Using database falco
> show series
key
---
events,akey=AValue,bkey=BValue,ckey=CValue,priority=Debug,rule=Testrule
events,akey=A_Value,bkey=B_Value,ckey=C_Value,priority=Debug,rule=Test_rule
> select * from events
name: events
time                akey    bkey    ckey    priority rule      value
----                ----    ----    ----    -------- ----      -----
1560433816893368400 AValue  BValue  CValue  Debug    Testrule  This is a test from falcosidekick
1560441359119741800 A_Value B_Value C_Value Debug    Test_rule This is a test from falcosidekick

Loki (with Grafana)

AWS SQS

SMTP

(SMTP_OUTPUTFORMAT="html")

(SMTP_OUTPUTFORMAT="text")

Opsgenie

Discord

Development

Build

go build

Quicktest

Create a debug event

curl -H "Content-Type: application/json" -H "Accept: application/json" localhost:2801/test

Test & Coverage

go test ./outputs -count=1 -cover -v

Author

Thomas Labarussias (https://github.com/Issif)