input: kafka: Add initial documentation (fluent#1091)

* input: kafka: Add initial documetation

Signed-off-by: Thiago Padilha <[email protected]>

* input: kafka: Address review suggestion

Co-authored-by: Pat <[email protected]>
Signed-off-by: Thiago Padilha <[email protected]>

* input: kafka: Put each sentence in a separate line (review request).

Signed-off-by: Thiago Padilha <[email protected]>

* kafka: Fix some typos/errors found during in_kafka review

Signed-off-by: Thiago Padilha <[email protected]>

---------

Signed-off-by: Thiago Padilha <[email protected]>
Signed-off-by: Thiago Padilha <[email protected]>
Co-authored-by: Pat <[email protected]>

Author: tchrono

pipeline/inputs/kafka.md

@@ -0,0 +1,71 @@
+# Kafka
+
+The Kafka input plugin allows subscribing to one or more Kafka topics to collect messages from an [Apache Kafka](https://kafka.apache.org/) service.
+This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka) \(built-in dependency\).
+
+## Configuration Parameters
+
+| Key | Description | default |
+| :--- | :--- | :--- |
+| brokers | Single or multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092. |  |
+| topics | Single entry or list of topics separated by comma \(,\) that Fluent Bit will subscribe to. |  |
+| client\_id | Client id passed to librdkafka. | |
+| group\_id | Group id passed to librdkafka. | fluent-bit |
+| poll\_ms | Kafka brokers polling interval in milliseconds. | 500 |
+| rdkafka.{property} | `{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) |  |
+
+## Getting Started
+
+In order to subscribe/collect messages from Apache Kafka, you can run the plugin from the command line or through the configuration file:
+
+### Command Line
+
+The **kafka** plugin can read parameters through the **-p** argument \(property\), e.g:
+
+```text
+$ fluent-bit -i kafka -o stdout -p brokers=192.168.1.3:9092 -p topics=some-topic
+```
+
+### Configuration File
+
+In your main configuration file append the following _Input_ & _Output_ sections:
+
+```text
+[INPUT]
+    Name        kafka
+    Brokers     192.168.1.3:9092
+    Topics      some-topic
+    poll_ms     100
+
+[OUTPUT]
+    Name        stdout
+```
+
+#### Example of using kafka input/output plugins
+
+The fluent-bit source repository contains a full example of using fluent-bit to process kafka records: 
+
+```text
+[INPUT]
+    Name kafka
+    brokers kafka-broker:9092
+    topics fb-source
+    poll_ms 100
+
+[FILTER]
+    Name    lua
+    Match   *
+    script  kafka.lua
+    call    modify_kafka_message
+
+[OUTPUT]
+    Name kafka
+    brokers kafka-broker:9092
+    topics fb-sink
+```
+
+The above will connect to the broker listening on `kafka-broker:9092` and subscribe to the `fb-source` topic, polling for new messages every 100 milliseconds.
+
+Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker.
+
+The example can be executed locally with `make start` in the `examples/kafka_filter` directory (docker/compose is used).

pipeline/outputs/kafka.md

@@ -11,7 +11,7 @@ Kafka output plugin allows to ingest your records into an [Apache Kafka](https:/
 | message\_key\_field | If set, the value of Message\_Key\_Field in the record will indicate the message key. If not set nor found in the record, Message\_Key will be used \(if set\). |  |
 | timestamp\_key | Set the key to store the record timestamp | @timestamp |
 | timestamp\_format | Specify timestamp format, should be 'double', '[iso8601](https://en.wikipedia.org/wiki/ISO_8601)' (seconds precision) or 'iso8601_ns' (fractional seconds precision) | double |
-| brokers | Single of multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092. |  |
+| brokers | Single or multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092. |  |
 | topics | Single entry or list of topics separated by comma \(,\) that Fluent Bit will use to send messages to Kafka. If only one topic is set, that one will be used for all records. Instead if multiple topics exists, the one set in the record by Topic\_Key will be used. | fluent-bit |
 | topic\_key | If multiple Topics exists, the value of Topic\_Key in the record will indicate the topic to use. E.g: if Topic\_Key is _router_ and the record is {"key1": 123, "router": "route\_2"}, Fluent Bit will use topic _route\_2_. Note that if the value of Topic\_Key is not present in Topics, then by default the first topic in the Topics list will indicate the topic to be used. |  |
 | dynamic\_topic | adds unknown topics \(found in Topic\_Key\) to Topics. So in Topics only a default topic needs to be configured | Off |
@@ -26,7 +26,7 @@ In order to insert records into Apache Kafka, you can run the plugin from the co
 
 ### Command Line
 
-The **kafka** plugin, can read the parameters from the command line in two ways, through the **-p** argument \(property\), e.g:
+The **kafka** plugin can read parameters through the **-p** argument \(property\), e.g:
 
 ```text
 $ fluent-bit -i cpu -o kafka -p brokers=192.168.1.3:9092 -p topics=test