diff --git a/docs/edge_processor.md b/docs/edge_processor.md new file mode 100644 index 0000000000..4d493dab5e --- /dev/null +++ b/docs/edge_processor.md @@ -0,0 +1,110 @@ +# Edge Processor integration guide + +## Intro + +`Edge Processor` can be used on that usecases: + +* Enrich log message extra data (for example add some field or override index) using `SPL2` +* Filter log message using `SPL2` +* Send log messages to alternative destanations (like `AWS S3`, `Apache Kafka`, etc.) + +## How it's working + +```mermaid +stateDiagram + direction LR + + SC4S: SC4S + EP: Edge Processor + Dest: Another destination + Device: Your device + S3: AWS S3 + Instance: Instance + Pipeline: Pipeline with SPL2 + + Device --> SC4S: Syslog protocol + SC4S --> EP: HEC + state EP { + direction LR + Instance --> Pipeline + } + EP --> Splunk + EP --> S3 + EP --> Dest +``` + +## Basic Setup + +### Docker / Podman + +Setup on your `env_file` HEC URL as IP of EP worker EC2 instance. +Token you can find in EP "global settings" page. + +``` +SC4S_DEST_SPLUNK_HEC_{EP1}_URL=http://x.x.x.x:8088 +SC4S_DEST_SPLUNK_HEC_{EP1}_TOKEN=secret +``` + +### Kubernetes + +Setup on your `values.yaml` HEC URL as IP of EP worker EC2 instance. +Token you can find in EP "global settings"page. + +``` +splunk: + hec_url: "https://x.x.x.x:8088" + hec_token: "secret" +``` + +## mTLS encryption + +### Preparing certs + +Before setup you need to [generate mTLS certificates](https://docs.splunk.com/Documentation/SplunkCloud/9.1.2308/EdgeProcessor/SecureForwarders). Server mTLS certificates should be uploaded to `Edge Processor` and client certifcates should be used with `SC4S`. + +Please rename your files, we expcting such filenames for client mTLS cerificates: + + * `key.pem` - client certificate key + * `cert.pem` - client certificate + * `ca_cert.pem` - certificate authority + +### Docker / Podman + + 1. Use HTTPS in HEC url: `SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=https://x.x.x.x:8088` + 2. Move your clients mTLS certificates to `/opt/sc4s/tls/hec` + 3. Mount `/opt/sc4s/tls/hec` to `/etc/syslog-ng/tls/hec` using docker/podman volumes. + 4. Define mounting mTLS point for HEC: `SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_MOUNT=/etc/syslog-ng/tls/hec` + 5. Start/Restart SC4S + +### Kubernetes + + 1. Add secret name of mTLS certs at `values.yaml`: + +``` +splunk: + hec_url: "https://x.x.x.x:8088" + hec_token: "secret" + hec_tls: "secret-name" +``` + + 2. Add your mtls certs at `secrets.yaml`: + +``` +hec_tls: + key: | + -----BEGIN PRIVATE KEY----- + Exmaple key + -----END PRIVATE KEY----- + cert: | + -----BEGIN CERTIFICATE----- + Exmaple cert + -----END CERTIFICATE----- + ca: | + -----BEGIN CERTIFICATE----- + Example ca + -----END CERTIFICATE----- +``` + + 3. Encrypt your `secrets.yaml` using `ansible-vault`. + 4. Add IP of cluster nodes to inventory file `ansible/inventory/inventory_microk8s_ha.yaml` + 5. Deploy ansible playbook `ansible/playbooks/microk8s_ha.yml` \ No newline at end of file diff --git a/docs/gettingstarted/ansible-docker-podman.md b/docs/gettingstarted/ansible-docker-podman.md index a0368a2550..c99e66f77f 100644 --- a/docs/gettingstarted/ansible-docker-podman.md +++ b/docs/gettingstarted/ansible-docker-podman.md @@ -1,29 +1,33 @@ -## Notice -SC4S installation can now be automated with Ansible. All you need to do now is provide a host on which you want to run SC4S and basic configuration (Splunk endpoint, HEC token, TLS configuration, etc.). +## About automating SC4S installation with Ansible +You can use Ansible to automate SC4S installation. To do this you provide the host on which you want to run SC4S and some basic configuration, including: +* The Splunk endpoint +* The HEC token +* TLS configuration # Initial Configuration -All you need to do before running sc4s with Ansible is providing `env_file`. In the env file provide at least proper Splunk endpoint and HEC token. -Create a file in `ansible/resources` catalog or edit [example file](/ansible/resources/env_file). +Before you run SC4S with Ansible provide the `env_file` with the Splunk endpoint and HEC token. + +1. Create a file in `ansible/resources` catalog or edit [example file](/ansible/resources/env_file). ``` dotenv --8<---- "ansible/resources/env_file" ``` -Next provide a host on which you want to run Docker Swarm cluster and host application in inventory file: +2. Provide a host on which you want to run Docker Swarm cluster and host application in the inventory file: ``` yaml --8<---- "ansible/inventory/inventory.yaml" ``` ## Deploy SC4S -Now you can run ansible playbook to deploy the application if you have ansible installed on your host -or use docker ansible image provided in the package: +1. If you have ansible installed on your host, run Ansible playbook to deploy the application. Otherwise you +can use the Docker Ansible image provided in the following package: ```bash # From repository root docker-compose -f ansible/docker-compose.yml build docker-compose -f ansible/docker-compose.yml up -d docker exec -it ansible_sc4s /bin/bash ``` -Once you are in containers remote shell you can run Docker Swam ansible playbook. -If you are authenticating via username/password: +2. Once you are in the container's remote shell, run the Docker Swam Ansible playbook. +To authenticate using username/password: ``` bash ansible-playbook -i path/to/inventory.yaml -u --ask-pass path/to/playbooks/docker.yml or @@ -39,9 +43,9 @@ ansible-playbook -i path/to/inventory.yaml -u --key-file p # Verify Proper Operation -SC4S has a number of "preflight" checks to ensure that the container starts properly and that the syntax of the underlying syslog-ng -configuration is correct. After this step completes, to verify SC4S is properly communicating with Splunk, -execute the following search in Splunk: +SC4S performs a number of "preflight" checks to ensure that the container starts properly and that the syntax of the underlying syslog-ng configuration is correct. + +After the checks are complete, execute the following search in Splunk to verify that SC4S is properly communicating with Splunk: ```ini index=* sourcetype=sc4s:events "starting up" @@ -52,19 +56,18 @@ This should yield an event similar to the following: ```ini syslog-ng starting up; version='3.28.1' ``` -You can verify if all services in swarm cluster are working by checking ```sc4s_container``` field in splunk- each service should be recognized by different container id. All other fields should be the same. +To verify that all services in a Swarm cluster are working, check the ```sc4s_container``` field in Splunk. Each service should be recognized by a different container ID. All other fields should be the same. -When the startup process proceeds normally (without syntax errors). If you do not see this, -follow the steps below before proceeding to deeper-level troubleshooting: +The startup process then proceeds normally without syntax errors. If it does not, try the following: * Check to see that the URL, token, and TLS/SSL settings are correct, and that the appropriate firewall ports are open (8088 or 443). -* Check to see that the proper indexes are created in Splunk, and that the token has access to them. +* Check to see that the proper indexes are created in Splunk, and that the token has access to the indexes. * Ensure the proper operation of the load balancer if used. -* Lastly, execute the following command to check the sc4s startup process running in the container (on the node that is hosting sc4s service). +* Execute the following command to check the SC4S startup process running in the container on the node that is hosting the SC4S service: ```bash sudo docker ps ``` -You will get an ID and , next: +You will get an ID and . Next, execute: ```bash docker logs @@ -82,4 +85,4 @@ syslog-ng checking config sc4s version=v1.36.0 starting goss starting syslog-ng -``` \ No newline at end of file +``` diff --git a/mkdocs.yml b/mkdocs.yml index 9f26aae99d..62c1c0c207 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -14,6 +14,11 @@ markdown_extensions: - sane_lists - codehilite - pymdownx.snippets + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format theme: @@ -57,6 +62,7 @@ nav: - SC4S Lite (Experimental): - Intro: "lite.md" - Pluggable modules: "pluggable_modules.md" + - Edge Processor: "edge_processor.md" - Troubleshooting: - SC4S Startup and Validation: "troubleshooting/troubleshoot_SC4S_server.md" - SC4S Logging and Troubleshooting Resources: "troubleshooting/troubleshoot_resources.md"