Switch to Loki logging

src/docs/app-monitoring/best-practices-for-application-logging-in-openshift.md

@@ -3,9 +3,9 @@ title: Best practices for application logging in OpenShift
 
 slug: best-practices-for-application-logging-in-openshift 
 
-description: document explains best practices for logging within the OpenShift environment and the use of console logs in Kibana as well as log long term retention and best security practices 
+description: document explains best practices for logging within the OpenShift environment and the use of console logs in Loki as well as log long term retention and best security practices 
 
-keywords: logging, app logging, openshift logging, kibana logging, logs, console logs
+keywords: logging, app logging, openshift logging, loki logging, logs, console logs
 
 page_purpose: help developers consider best practices for logging in OpenShift 
 
@@ -17,51 +17,52 @@ content_owner: Olena Mitovska
 
 sort_order: 9
 ---
-# Best practices for application logging in OpenShift 
+# Best practices for application logging in OpenShift
+
 Last updated: **November 28, 2023**
 
-This guide delves into accessing and querying logs in Kibana, highlighting best practices for efficient log storage. You will also be able to know  how to secure your container logs and gain insights into the intricacies of using logfiles for long-term retention. 
+This guide delves into accessing and querying logs in Loki, highlighting best practices for efficient log storage. You will also be able to know how to secure your container logs and gain insights into the intricacies of using logfiles for long-term retention.
 
-It emphasizes key considerations, including the secure use of shared resources, proactive measures against potential attacks, and the importance of tailored log retention strategies. Whether you're accessing Kibana with OpenShift credentials or fine-tuning log security, this document provides actionable insights for effective log management in your OpenShift environment.
+It emphasizes key considerations, including the secure use of shared resources, proactive measures against potential attacks, and the importance of tailored log retention strategies. Whether you're accessing Loki with OpenShift credentials or fine-tuning log security, this document provides actionable insights for effective log management in your OpenShift environment.
 
 ## On this page
+
 * **[How to include effective logging practices in your code](#how-to-include-effective-logging-practices-in-your-code)**
-* **[Console logs in Kibana](#console-logs-in-kibana)**
+* **[Console logs in Loki](#console-logs-in-loki)**
 * **[Managing logfiles and long-term log retention](#managing-logfiles-and-long-term-retention)**
 * **[Logging and security](#logging-and-security)**
 * **[Related pages](#related-pages)**
 
 ---
+
 ## How to include effective logging practices in your code
 
 Most of this document will be about logging in Openshift. If you’re looking for general logging best practices while developing your application, here are some helpful resources:
 
-
 * [Heroku Dev Center - Writing best practices for application logs](https://devcenter.heroku.com/articles/writing-best-practices-for-application-logs)
 * [NewRelic - logging best practices](https://newrelic.com/blog/best-practices/best-log-management-practices)
 * [DataSet - the 10 commandments of logging](https://www.dataset.com/blog/the-10-commandments-of-logging/)
 * [Splunk - logging best practices](https://dev.splunk.com/enterprise/docs/developapps/addsupport/logging/loggingbestpractices/)
 
+## Console logs in Loki
 
-## Console logs in Kibana
 In OpenShift, you can easily access application console logs through the web console or the OpenShift CLI. These logs are accessible as long as the container is running, but keep in mind that restarting the container erases logs from its previous instance.
 
 Pods might restart to relocate from one node to another, causing logs to vanish from OpenShift unexpectedly. Stay aware that logs are tied to the container's lifecycle, and a container restart means a fresh log slate.
 
-Kibana serves as a centralized tool that gathers log messages, known as "console logs," from the standard output and standard error streams of your application. It covers all pods running on the OpenShift platform, preserving these logs for approximately 14 days. While the retention period may vary slightly, any adjustment will not exceed 24 hours.
+Loki serves as a centralized tool that gathers log messages, known as "console logs," from the standard output and standard error streams of your application. It covers all pods running on the OpenShift platform, preserving these logs for 14 days.
 
 Should your team require log storage beyond the 14-day window, it's your responsibility to implement a solution. The Platform Team recommends leveraging the [OCIO Object Storage service](../platform-architecture-reference/platform-storage.md) for efficient log storage. For additional details, refer to the next section on logs and long-term log retention.
 
-### Accessing Kibana with OpenShift credentials
-To access Kibana, use your OpenShift credentials. If you can view a pod's console logs in the OpenShift web console, you can seamlessly access the same pod's logs in Kibana. A minimal setup in Kibana is required for log viewing and querying, especially if you're new to Kibana.
+### Accessing Loki
 
-For detailed instructions on this setup process, refer to our OCP101 lab section [covering logging and visualizations](https://github.com/BCDevOps/devops-platform-workshops/blob/master/101-lab/content/12_logging_and_visualizations.md). This will guide you through the necessary steps to make the most of Kibana's features.
+Loki is built into the OpenShift web console. You can access it under the side bar section Observe and then selecting Logs.
 
-### Querying logs effectively on Kibana
+### Querying logs effectively on Loki
 
-Kibana empowers you to perform targeted log queries across various fields such as log message, time, and container name. This flexibility extends to querying logs across multiple containers and pods, simplifying the search for specific messages within a deployment or stateful set.
+Loki empowers you to perform targeted log queries across various fields such as log message, time, and container name. This flexibility extends to querying logs across multiple containers and pods, simplifying the search for specific messages within a deployment or stateful set.
 
-It's crucial to remember that Kibana operates as a shared service, and the storage it utilizes is a communal resource for all platform users. Exercise caution when utilizing log levels like DEBUG in production environments.
+It's crucial to remember that Loki operates as a shared service, and the storage it utilizes is a communal resource for all platform users. Exercise caution when utilizing log levels like DEBUG in production environments.
 
  In instances where your application generates an exceptionally high volume of logs, the Platform Team may reach out to address and resolve the issue. Efficient use of shared resources ensures a smoother experience for everyone on the platform.
 
@@ -73,22 +74,25 @@ Typically, logfiles are the go-to for storing logs in specific cases rather than
 
 Once your logs find a home in a logfile on disk, they could stick around indefinitely. But, beware! Holding onto logfiles forever could lead to a significant space crunch over time. To avoid this, it's crucial to trim your logfiles, bidding farewell to the oldest ones when they're no longer needed. When deciding on a timeframe for logfile retention, consider what logs your team might require for troubleshooting or auditing purposes, and adhere to any policy or legal requirements for log retention.
 
-The Platform Team suggests using a cronjob to automatically clear out old logfiles on a regular schedule. 
+The Platform Team suggests using a cronjob to automatically clear out old logfiles on a regular schedule.
 
-Now, if you're wondering about retaining logs sent to standard output, in short: Keep sending them there. It makes logs easy to read and ensures compatibility with Kibana's querying functionality. However, if you want to retain this data beyond Kibana's realm or use your own Kibana instance with custom retention rules, a team on our OpenShift Platform has got your back. Check out their documentation for a [handy template to deploy your own EFK stack](https://github.com/bcgov/elmsd-nodejs/tree/main/packages/openshift/templates/efk-stack) in your namespace. It's a great resource for diving deeper into log management.
+Now, if you're wondering about retaining logs sent to standard output, in short: Keep sending them there. It makes logs easy to read and ensures compatibility with Loki's querying functionality. However, if you want to retain this data beyond Loki's realm or use your own Kibana instance with custom retention rules, a team on our OpenShift Platform has got your back. Check out their documentation for a [handy template to deploy your own EFK stack](https://github.com/bcgov/elmsd-nodejs/tree/main/packages/openshift/templates/efk-stack) in your namespace. It's a great resource for diving deeper into log management.
 
 ## Logging and security
-When dealing with container logs, remember that anyone with View permissions in your namespace can access them. It's crucial to exercise caution in revealing too much detail or sensitive information in these logs. Additionally, keep in mind that logs are transmitted in plaintext to both Kibana and SIEM systems. Avoid including confidential data like passwords, private keys, or Personally Identifiable Information (PII). 
 
-For added security, log the connections to and from your application. This aids in auditing potential compromises down the line. Since web connections into your pods display the source IP of the Router pods, ensure you also extract the X-Forwarded-For header and log that IP. 
+When dealing with container logs, remember that anyone with View permissions in your namespace can access them. It's crucial to exercise caution in revealing too much detail or sensitive information in these logs. Additionally, keep in mind that logs are transmitted in plaintext to both Loki and SIEM systems. Avoid including confidential data like passwords, private keys, or Personally Identifiable Information (PII).
 
-OpenShift adds an X-Forwarded-For header to your app in the HTTP protocol, containing the client IP connected to your web server. Logging this IP is preferred over the default source IP in your logs, as the source IP is just the OpenShift router.
+For added security, log the connections to and from your application. This aids in auditing potential compromises down the line. Since web connections into your pods display the source IP of the Router pods, ensure you also extract the `X-Forwarded-For` header and log that IP.
+
+OpenShift adds an `X-Forwarded-For` header to your app in the HTTP protocol, containing the client IP connected to your web server. Logging this IP is preferred over the default source IP in your logs, as the source IP is just the OpenShift router.
 
 If you're maintaining log files, establish proper permissions to restrict access to only authorized individuals. Additionally, regularly back up these files to the OCIO Object Store. This practice helps preserve them in case of an attack attempting to delete local logs. It's a proactive step to ensure the integrity and availability of your logs.
 
 ---
+
 ## Related pages
-* [Logging and visualizations - OpenShift 101 ](https://github.com/BCDevOps/devops-platform-workshops/blob/master/101-lab/content/12_logging_and_visualizations.md)
+
+* [Logging and visualizations - OpenShift 101](https://github.com/BCDevOps/devops-platform-workshops/blob/master/101-lab/content/12_logging_and_visualizations.md)
 * [Template to deploy your own EFK stack](https://github.com/bcgov/elmsd-nodejs/tree/main/packages/openshift/templates/efk-stack)
 * [Heroku Dev Center - Writing best practices for application logs](https://devcenter.heroku.com/articles/writing-best-practices-for-application-logs)
 * [NewRelic - logging best practices](https://newrelic.com/blog/best-practices/best-log-management-practices)

src/docs/app-monitoring/guidelines-for-sli-and-monitoring.md

@@ -25,7 +25,7 @@ Last updated: **April 17, 2024**
 
 **Performance Monitoring Tools:**  You can use tools like Prometheus, Grafana to continuously watch and display system performance metrics, including response times
 
-**Logging**: Make sure that the Application system records the time it takes for each data retrieval request. Periodically analyze these logs or use log aggregation tools like the ELK Stack (Kibana, Logstash, etc.) to gain insights
+**Logging**: Make sure that the Application system records the time it takes for each data retrieval request. Periodically analyze these logs to gain insights
 
 **Threshold Alerts**: Create alerts to inform system administrators or engineers when response times exceed the defined threshold
 

src/docs/build-deploy-and-maintain-apps/deploy-an-application.md

@@ -207,9 +207,9 @@ OpenShift provides robust monitoring capabilities to ensure your application is
 
 - **Monitoring Dashboard**: Sysdig allows you to set up a monitoring dashboard for your application. Check out this [video tutorial](https://www.youtube.com/watch?v=K4rkSCSq3C4&list=PL9CV_8JBQHiorxwU-2nA8aqM4KTzdCnfg) on how to set up your Sysdig monitoring dashboard
 
-### 2. Configuring Application Logs with Kibana
+### 2. Configuring Application Logs with Loki
 
-Logs are critical for debugging and understanding the behaviour of your application. OpenShift integrates with Kibana for log aggregation. This allows you to collect, index, and visualize logs in a centralized location. For a detailed guide on [how to configure and use Kibana](https://stackoverflow.developer.gov.bc.ca/questions/906).
+Logs are critical for debugging and understanding the behaviour of your application. OpenShift integrates with Loki for log aggregation. This allows you to collect, index, and visualize logs in a centralized location. For a detailed guide on [how to configure and use Loki](TBD).
 
 ---
 

src/docs/build-deploy-and-maintain-apps/maintain-an-application.md

@@ -374,7 +374,7 @@ By setting up comprehensive monitoring and alert systems for your application, y
 
 1. Refer to [this guide](../app-monitoring/check-application-health-after-outage.md) to identify the cause of the outage,  whether it's a platform-wide problem or specific to your application.  This guide will also provide guidance on initiating the troubleshooting process for your application's issues. Moreover, consider creating a checklist for your application. This checklist should outline the essential items to review in order to confirm that everything is functioning properly following an outage.
 
-2. Utilize application logs as a valuable resource for troubleshooting. OpenShift is seamlessly integrated with Kibana, which facilitates log aggregation. Through this integration, you can gather, categorize, and visualize container logs in a centralized hub. For comprehensive training on application logging with Kibana refer to the [provided training materials](https://github.com/bcgov/devops-platform-workshops/blob/master/openshift-201/logging.md)
+2. Utilize application logs as a valuable resource for troubleshooting. OpenShift is seamlessly integrated with Loki, which facilitates log aggregation. Through this integration, you can gather, categorize, and visualize container logs in a centralized hub. For comprehensive training on application logging with Loki refer to the [provided training materials](https://github.com/BCDevOps/devops-platform-workshops/blob/master/openshift-201/logging.md)
 
 3. It's essential for your team to have a strong understanding of your application and its relevant components.  This knowledge base is crucial for effective troubleshooting and minimizing the impact of any downtime
 
@@ -503,7 +503,7 @@ It is also important to keep the Platform Services [Product Registry](https://re
 - [Restoring Backup Volumes on OpenShift](../automation-and-resiliency/netapp-backup-restore.md)
 - [Grant user access in OpenShift](../openshift-projects-and-access/grant-user-access-openshift.md)
 - [Vault secrets management](../secrets-management/vault-secrets-management-service.md)
-- [Kibana training application logging](https://github.com/bcgov/devops-platform-workshops/blob/master/openshift-201/logging.md)
+- [Loki training application logging](https://github.com/BCDevOps/devops-platform-workshops/blob/master/openshift-201/logging.md)
 - [Check application health after outage](../app-monitoring/check-application-health-after-outage.md)
 - [Building secure and reliable systems e-book](https://sre.google/books/building-secure-reliable-systems/)
 - [Deploying to OpenShift e-book](https://cloud.redhat.com/hubfs/pdfs/Deploying_to_OpenShift.pdf)

src/docs/security-and-privacy-compliance/platform-security-compliance.md

@@ -98,7 +98,7 @@ The following services were designed with the associated [IMIT standards](https:
 |                                          | [6.10 Cryptographic Standard](https://www2.gov.bc.ca/assets/download/A834831A594245CD81D9BF99DDF39FCD) |
 | Asset Management (Artifactory, Registry)        | [6.23 Asset Management Security Standard](https://www2.gov.bc.ca/assets/download/F9BA3AFD52B34727BA261F052ADEAA0B) |
 | Networking (SDN, KNPs)                   | 6.13 Network Security Zone Standard - available by request to [email protected] |
-| Logging/Monitoring (Kibana, SIEM)        | [6.27 Operations Security Standard](https://www2.gov.bc.ca/assets/download/0F4DF4FAC5214C6387B6B51DD538FF6E) |
+| Logging/Monitoring (Loki, SIEM)        | [6.27 Operations Security Standard](https://www2.gov.bc.ca/assets/download/0F4DF4FAC5214C6387B6B51DD538FF6E) |
 | App Security (SAST,DAST,SCA)             | [6.14 Application & Web Development & Deployment Standard](https://www2.gov.bc.ca/assets/download/29237A3033824CCBAC0465939BFB2CEF) |
 |                                          | [6.34 Vulnerability Management Scanning Standrard](https://www2.gov.bc.ca/assets/download/3DFC2361BC334BFDBDF907B6B16C5358) |
 

src/docs/security-and-privacy-compliance/platform-security-tools.md

@@ -26,7 +26,7 @@ Here you will find details on security tooling used by platform administrators a
 * [**Container image scanning (ACS, Xray)**](#container-image-scanning-acs-xray)
 * [**Container runtime security**](#container-runtime-security)
 * [**Secrets management**](#secrets-management)
-* [**Logging and monitoring (EKS, Kibana, Graphana, Sysdig Monitor, SIEM, Uptime, Status)**](#logging-and-monitoring-eks-kibana-graphana-sysdig-monitor-siem-uptime-status)
+* [**Logging and monitoring (Loki, Sysdig Monitor, SIEM, Uptime, Status)**](#logging-and-monitoring-loki-sysdig-monitor-siem-uptime-status)
 * [**Related pages**](#other-important-considerations)
 
 <!-- ### End of "On this page" --> 
@@ -107,7 +107,7 @@ Vault is the preferred secrets management tool to use on OpenShift.
 
 Integrated as the GitOps Operator within OpenShift, Argo CD empowers a GitOps capability to synchronize a Git repository with OpenShift configurations, whether they pertain to the platform or specific applications. 
 
-## Logging and monitoring (ElasticSearch, Kibana, Graphana, Sysdig Monitor, SIEM, Uptime, Status)
+## Logging and monitoring (Loki, Sysdig Monitor, SIEM, Uptime, Status)
 
 The Platform Services team offers a range of tools designed to ensure the expected behavior of our platform and applications. These tools not only help in monitoring normal operations but also enable thorough investigations into any anomalies that may arise.
 
@@ -116,12 +116,10 @@ The Platform Services team offers a range of tools designed to ensure the expect
 Within the OpenShift interface, project teams can view logs associated with a given pod through the Logs tab.  
 ![OpenShift Pod details screen Logs tab example](../../images/openshift-pod-details-logs-tab-example.jpg)
 
-**Kibana:**
+**Loki:**
 
 This tool provides a more wholistic view of logs for an application or at the platform level, as well as providing visualization and alerting capability.
 
-* [More on Kibana](https://kibana-openshift-logging.apps.silver.devops.gov.bc.ca/)
-
 **Sysdig Monitor:**
 
 This tool allows our platform admins and platform teams to build monitoring dashboards.