Monitoring guide (#6545)

Co-authored-by: Olivier Ruas <[email protected]>
GitOrigin-RevId: 7de83b723231fee45ee1768ff02449a8543042a8

docs/2.developers/4.user-guide/080.deployment/70.pathway-monitoring.md

@@ -0,0 +1,158 @@
+---
+title: 'Monitoring Pathway instance'
+description: 'Tutorial on how to set up monitoring of Pathway instance'
+date: '2024-05-27'
+thumbnail: '/assets/content/blog/th-prometheus.png'
+tags: ['tutorial', 'engineering']
+keywords: ['monitoring', 'OpenTelemetry', 'Grafana', 'metrics']
+---
+
+# Monitoring Pathway Instance
+
+This article will guide you through setting up monitoring for your Pathway application.
+It will allow you to reliably oversee the execution of your streaming pipeline.
+
+⚠️ Monitoring features require a Pathway for Business license.\
+To obtain a free license, please visit the [Get License](/get-license) page.
+
+Proper monitoring of Pathway application is inevitable for business-critical production deployment.
+One has to ensure that all the current and previous bottlenecks, like drops in latency and throughput or any anomalous behaviors
+are easily identifiable, so that all the issues can be immediately resolved.
+
+Pathway leverages the OpenTelemetry protocol (OTLP) for gathering and sending traces, metrics, 
+and logs which enables connecting seamlessly with various monitoring destinations, including the popular OpenTelemetry Collector.
+
+This tutorial will show you how to set up the monitoring of a Pathway application using an OpenTelemetry Collector. The logs will be accessed using Grafana Cloud and [Grafana Loki](https://grafana.com/oss/loki/).
+
+To follow along, you will need an OpenTelemetry Collector set up with the gRPC protocol enabled for its OTLP receiver. 
+If you already have your OpenTelemetry Collector configured correctly, you can skip the next section, otherwise please make sure
+you have [Docker installed](https://www.docker.com/get-started/).
+
+## Configuring OpenTelemetry Collector
+
+The OpenTelemetry Collector offers various deployment options. 
+This tutorial focuses on an approach using Docker and the official `otel/opentelemetry-collector-contrib` image. 
+However, keep in mind there are other methods available. 
+You can for example use the [official Helm Chart](https://opentelemetry.io/docs/kubernetes/helm/collector/) to run it on Kubernetes,
+or reach for solutions like [Grafana Alloy](https://grafana.com/docs/alloy/latest/get-started/install/).
+
+First, create a configuration file named `config.yaml` for the OpenTelemetry Collector with OTLP receiver and debug exporter.
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      grpc:
+
+exporters:
+  debug:
+    verbosity: detailed
+```
+
+Now specify pipelines for traces, metrics and logs, adding the following snippet in the same file:
+
+```yaml
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      exporters: [debug]
+    metrics:
+      receivers: [otlp]
+      exporters: [debug]
+    logs:
+      receivers: [otlp]
+      exporters: [debug]
+```
+
+You can now launch the server, using command below.
+
+```bash
+docker run -v config.yaml:/etc/otelcol-contrib/config.yaml -p <PORT>:4317 otel/opentelemetry-collector-contrib:latest
+```
+
+Make sure to replace `<PORT>` with the chosen port.
+
+## Enabling Pathway monitoring
+
+Once you have your license key ready, setting up monitoring in Pathway is easy. Just copy these lines at the beginning of your pipeline.
+Don't forget, you can obtain your free license [here](/get-license).
+
+```python
+import pathway as pw
+
+pw.set_license_key(key="YOUR-KEY")
+pw.set_monitoring_config(server_endpoint="http://localhost:<PORT>")
+
+# your pipeline here...
+
+pw.run()
+```
+
+That's it! Run your pipeline now and observe the OpenTelemetry Collector logs for collected data. You should see the first entries appear shortly.
+
+## Exploring the Data
+
+The OpenTelemetry Collector lets you export collected data to various observability backends. 
+Explore the OpenTelemetry documentation and the Collector's [repository](https://github.com/open-telemetry/opentelemetry-collector) for more details.
+
+This tutorial focuses on using [Grafana Loki](https://grafana.com/oss/loki/) as your log destination. Grafana Cloud offers a comprehensive package that covers your monitoring needs. 
+Once you've completed this tutorial, configuring Grafana Prometheus for metrics and Grafana Tempo for traces should be straightforward.
+Complete configuration example can be found in [our repository](https://github.com/pathwaycom/pathway/tree/main/examples/projects/monitoring).
+
+### Configuring Grafana Cloud
+
+Create your account in the Grafana Cloud, by going to [https://grafana.com/](https://grafana.com/). 
+In the organization panel, click on "Send Logs" button under Loki section.
+It should direct you to Loki configuration page. 
+
+Modify your `config.yaml` by adding configuration below. Replace `<USER>`, `<TOKEN>`, and `<URL>` with the information found under the "Sending Logs to Grafana Cloud" section. 
+Ensure that your token is generated with metrics write permissions.
+
+```yaml
+# set up authorization extension  
+extensions:
+  basicauth/grafana_cloud_loki:
+    client_auth:
+      username:  <USER>
+      password:  <TOKEN>
+
+# add tempo exporter with authenticator
+ exporters:
+  debug:
+  	verbosity: detailed
+  loki/grafana_cloud_logs:
+    endpoint: <URL>
+    auth:
+      authenticator: basicauth/grafana_cloud_loki
+
+
+# enabled authorization extension and add exporter to the logs pipeline
+service:
+   extensions: [basicauth/grafana_cloud_loki]
+   pipelines:
+    logs:
+      receivers: [otlp]
+      exporters: [loki/grafana_cloud_logs, debug]
+	...
+```
+
+You can now restart your OpenTelemetry Collector and check Grafana Loki.
+
+1. Go to your Grafana organization portal and launch Grafana.
+2. Select "Explore" in the secontion on the left.
+3. Select `grafanacloud-<your-org>-loki` source.
+4. Filter by label `service_name=~".*pathway"`
+
+<img src="/assets/content/tutorials/grafana-cloud-loki.png" alt="Explore loki" class="mx-auto" />
+
+You should see the logs appearing below.
+
+This tutorial only scratches the surface of monitoring setup but should provide a good starting point.
+Don't hesitate to check out [our examples](https://github.com/pathwaycom/pathway/tree/main/examples/projects/monitoring) as well as [OpenTelemetry documentation](https://opentelemetry.io/docs/) to learn more.
+
+
+
+
+
+

python/pathway/internals/run.py

@@ -28,8 +28,9 @@ def run(
             pathway.MonitoringLevel.NONE, pathway.MonitoringLevel.IN_OUT,
             pathway.MonitoringLevel.ALL. If unset, pathway will choose between
             NONE and IN_OUT based on output interactivity.
-        with_http_server: whether to start a http server with runtime metrics. Learn
-            more in a `tutorial </developers/user-guide/deployment/prometheus-monitoring/>`_ .
+        with_http_server: whether to start a http server with runtime metrics. [will be deprecated soon]
+            Learn more about Pathway monitoring in a
+            `tutorial </developers/user-guide/deployment/pathway-monitoring/>`_ .
         default_logging: whether to allow pathway to set its own logging handler. Set
             it to False if you want to set your own logging handler.
         persistence_config: the config for persisting the state in case this
@@ -71,8 +72,9 @@ def run_all(
             pathway.MonitoringLevel.NONE, pathway.MonitoringLevel.IN_OUT,
             pathway.MonitoringLevel.ALL. If unset, pathway will choose between
             NONE and IN_OUT based on output interactivity.
-        with_http_server: whether to start a http server with runtime metrics. Learn
-            more in a `tutorial </developers/user-guide/deployment/prometheus-monitoring/>`_ .
+        with_http_server: whether to start a http server with runtime metrics. [will be deprecated soon]
+            Learn more about Pathway monitoring in a
+            `tutorial </developers/user-guide/deployment/pathway-monitoring/>`_ .
         default_logging: whether to allow pathway to set its own logging handler. Set
             it to False if you want to set your own logging handler.
         persistence_config: the config for persisting the state in case this