Skip to content

Commit

Permalink
docs: Edge Processor docs
Browse files Browse the repository at this point in the history
  • Loading branch information
ikheifets-splunk committed Apr 5, 2024
1 parent e55ac8b commit bad9d4f
Show file tree
Hide file tree
Showing 2 changed files with 116 additions and 0 deletions.
110 changes: 110 additions & 0 deletions docs/edge_processor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Edge Processor integration guide (Experimental)

## Intro

`Edge Processor` can be used on that usecases:

* Enrich log message extra data (for example add some field or override index) using `SPL2`
* Filter log message using `SPL2`
* Send log messages to alternative destanations (like `AWS S3`, `Apache Kafka`, etc.)

## How it's working

```mermaid
stateDiagram
direction LR
SC4S: SC4S
EP: Edge Processor
Dest: Another destination
Device: Your device
S3: AWS S3
Instance: Instance
Pipeline: Pipeline with SPL2
Device --> SC4S: Syslog protocol
SC4S --> EP: HEC
state EP {
direction LR
Instance --> Pipeline
}
EP --> Splunk
EP --> S3
EP --> Dest
```

## Basic Setup

### Docker / Podman

Setup on your `env_file` HEC URL as IP of EP worker EC2 instance.
Token you can find in EP "global settings" page.

```
SC4S_DEST_SPLUNK_HEC_{EP1}_URL=http://x.x.x.x:8088
SC4S_DEST_SPLUNK_HEC_{EP1}_TOKEN=secret
```

### Kubernetes

Setup on your `values.yaml` HEC URL as IP of EP worker EC2 instance.
Token you can find in EP "global settings"page.

```
splunk:
hec_url: "https://x.x.x.x:8088"
hec_token: "secret"
```

## mTLS encryption

### Preparing certs

Before setup you need to [generate mTLS certificates](https://docs.splunk.com/Documentation/SplunkCloud/9.1.2308/EdgeProcessor/SecureForwarders). Server mTLS certificates should be uploaded to `Edge Processor` and client certifcates should be used with `SC4S`.

Please rename your files, we expcting such filenames for client mTLS cerificates:

* `key.pem` - client certificate key
* `cert.pem` - client certificate
* `ca_cert.pem` - certificate authority

### Docker / Podman

1. Use HTTPS in HEC url: `SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://x.x.x.x:8088`
2. Move your clients mTLS certificates to `/opt/sc4s/tls/hec`
3. Mount `/opt/sc4s/tls/hec` to `/etc/syslog-ng/tls/hec` using docker/podman volumes.
4. Define mounting mTLS point for HEC: `SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_MOUNT=/etc/syslog-ng/tls/hec`
5. Start/Restart SC4S

### Kubernetes

1. Add secret name of mTLS certs at `values.yaml`:

```
splunk:
hec_url: "https://x.x.x.x:8088"
hec_token: "secret"
hec_tls: "secret-name"
```

2. Add your mtls certs at `secrets.yaml`:

```
hec_tls:
key: |
-----BEGIN PRIVATE KEY-----
Exmaple key
-----END PRIVATE KEY-----
cert: |
-----BEGIN CERTIFICATE-----
Exmaple cert
-----END CERTIFICATE-----
ca: |
-----BEGIN CERTIFICATE-----
Example ca
-----END CERTIFICATE-----
```

3. Encrypt your `secrets.yaml` using `ansible-vault`.
4. Add IP of cluster nodes to inventory file `ansible/inventory/inventory_microk8s_ha.yaml`
5. Deploy ansible playbook `ansible/playbooks/microk8s_ha.yml`
6 changes: 6 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@ markdown_extensions:
- sane_lists
- codehilite
- pymdownx.snippets
- pymdownx.superfences:
custom_fences:
- name: mermaid
class: mermaid
format: !!python/name:pymdownx.superfences.fence_code_format


theme:
Expand Down Expand Up @@ -57,6 +62,7 @@ nav:
- SC4S Lite (Experimental):
- Intro: "lite.md"
- Pluggable modules: "pluggable_modules.md"
- Edge Processor: "edge_processor.md"
- Troubleshooting:
- SC4S Startup and Validation: "troubleshooting/troubleshoot_SC4S_server.md"
- SC4S Logging and Troubleshooting Resources: "troubleshooting/troubleshoot_resources.md"
Expand Down

0 comments on commit bad9d4f

Please sign in to comment.