diff --git a/docusaurus/docs/guides/ec2-monitoring.md b/docusaurus/docs/guides/ec2-monitoring.md index 26fdbc232..4dca81fa3 100644 --- a/docusaurus/docs/guides/ec2-monitoring.md +++ b/docusaurus/docs/guides/ec2-monitoring.md @@ -34,9 +34,7 @@ The Unified CloudWatch Agent is an open-source software under the MIT license wh The CloudWatch Agent can be installed through the [command line](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/installing-cloudwatch-agent-commandline.html). The required package for various architectures and various operating systems are available for [download](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/download-cloudwatch-agent-commandline.html). Create the necessary [IAM role](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent-commandline.html) which provides permissions for CloudWatch agent to read information from the Amazon EC2 instance and write it to CloudWatch. Once the required IAM role is created, you can [install and run](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-commandline-fleet.html) the CloudWatch agent on the required Amazon EC2 Instance. -:::note - "References" - +:::info Documentation: [Installing the CloudWatch agent using the command line](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/installing-cloudwatch-agent-commandline.html) AWS Observability Workshop: [Setup and install CloudWatch agent](https://catalog.workshops.aws/observability/en-US/aws-native/ec2-monitoring/install-ec2) @@ -46,8 +44,7 @@ The CloudWatch Agent can be installed through the [command line](https://docs.aw The CloudWatch Agent can also be installed through [AWS Systems Manager](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/installing-cloudwatch-agent-ssm.html). Create the necessary IAM role which provides permissions for CloudWatch agent to read information from the Amazon EC2 instance and write it to CloudWatch & communicate with AWS Systems Manager. Before installing the CloudWatch agent on the EC2 instances, [install or update](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/download-CloudWatch-Agent-on-EC2-Instance-SSM-first.html#update-SSM-Agent-EC2instance-first) the SSM agent on the required EC2 instances. The CloudWatch agent can be downloaded through the AWS Systems Manager. JSON Configuration file can be created to specify what metrics (including custom metrics), logs are to be collected. Once the required IAM role is created & the configuration file is created, you can install and run the CloudWatch agent on the required Amazon EC2 Instances. -:::note - References: +:::info Documentation: [Installing the CloudWatch agent using AWS Systems Manager](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/installing-cloudwatch-agent-ssm.html) AWS Observability Workshop: [Install CloudWatch agent using AWS Systems Manager Quick Setup](https://catalog.workshops.aws/observability/en-US/aws-native/ec2-monitoring/install-ec2/ssm-quicksetup) @@ -85,13 +82,13 @@ AWS provides two types of tools, automated and manual that help customers monito Metrics are the fundamental concept in CloudWatch. A metric represents a time-ordered set of data points that are published to CloudWatch. Think of a metric as a variable to monitor, and the data points as representing the values of that variable over time. For example, the CPU usage of a particular EC2 instance is one metric provided by Amazon EC2. -![cw-metrics]((../imagescw-metrics.png) +![cw-metrics](../images/cw-metrics.png) #### Default Metrics using CloudWatch Agent Amazon CloudWatch collects metrics from Amazon EC2 instance which can be viewed through AWS Management Console, AWS CLI, or an API. The available metrics are data points which are covered for 5 minute interval through Basic Monitoring or at a 1 minute interval for detailed monitoring (if turned on). -![default-metrics]((../imagesdefault-metrics.png) +![default-metrics](../images/default-metrics.png) #### Custom Metrics using CloudWatch Agent @@ -113,11 +110,11 @@ The CloudWatch agent supports collecting custom metrics from your EC2 instances. Amazon CloudWatch Logs helps customers monitor and troubleshoot systems and applications in near real time using existing system, application and custom log files. To collect logs from Amazon EC2 Instances and on-premise servers to CloudWatch, the unified CloudWatch Agent needs to be installed. The latest unified CloudWatch Agent is recommended, since it can collect both logs and advanced metrics. It also supports a variety of operating systems. If the instance uses Instance Metadata Service Version 2 (IMDSv2) then the unified agent is required. -![cw-logs]((../imagescw-logs.png) +![cw-logs](../images/cw-logs.png) The logs collected by the unified CloudWatch agent are processed and stored in Amazon CloudWatch Logs. Logs can be collected from Windows or Linux Servers and from both Amazon EC2 and on-premise servers. The CloudWatch agent configuration wizard can be used to setup the config JSON file which defines the setup of the CloudWatch agent. -![logs-view]((../imageslogs-view.png) +![logs-view](../images/logs-view.png) :::note AWS Observability Workshop: [Logs](https://catalog.workshops.aws/observability/en-US/aws-native/logs) @@ -142,11 +139,11 @@ Actions can be, Invoke an AWS Lambda function, Invoke Amazon EC2 Run Command, Re Amazon [CloudWatch alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html) can watch a metric over a time period you specify, and perform one or more actions based on the value of the metric relative to a given threshold over a number of time periods. An alarm invokes actions only when the alarmchanges state. The action can be a notification sent to an Amazon Simple Notification Service (Amazon SNS) topic or Amazon EC2 Auto Scaling or take other appropriate actions like [stop, terminate, reboot, or recover an EC2 instance.](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/UsingAlarmActions.html) -![CloudWatch Alarm]((../imagescw-alarm.png) +![CloudWatch Alarm](../images/cw-alarm.png) Once the alarm is triggered an Email notification is sent to an SNS Topic as an action. -![sns-alert]((../imagessns-alert.png) +![sns-alert](../images/sns-alert.png) #### Monitoring for Auto Scaling Instances @@ -160,19 +157,19 @@ Getting to know the inventory details of resources in AWS accounts, the resource Automatic Dashboards are available in all AWS public regions which provides an aggregated view of the health and performance of all AWS resources including Amazon EC2 instances under CloudWatch. This helps customers quickly get started with monitoring, resource-based view of metrics and alarms, and easily drill-down to understand the root cause of performance issues. Automatic Dashboards are pre-built with AWS service recommended [best practices](https://docs.aws.amazon.com/prescriptive-guidance/latest/implementing-logging-monitoring-cloudwatch/cloudwatch-dashboards-visualizations.html), remain resource aware, and dynamically update to reflect the latest state of important performance metrics. -![ec2 dashboard]((../imagesec2-auto-dashboard.png) +![ec2 dashboard](../images/ec2-auto-dashboard.png) #### Custom Dashboards in CloudWatch With [Custom Dashboards](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create_dashboard.html) Customers can create as many additional dashboards as they want with different widgets and customize it accordingly . Dashboards can be configured for cross-region and cross account view and can be added to a favorites list. -![ec2 custom dashboard]((../imagesec2-custom-dashboard.png) +![ec2 custom dashboard](../images/ec2-custom-dashboard.png) #### Resource Health Dashboards in CloudWatch Resource Health in CloudWatch ServiceLens is a fully managed solution that customers can use to automatically discover, manage, and visualize the [health and performance of Amazon EC2 hosts](https://aws.amazon.com/blogs/mt/introducing-cloudwatch-resource-health-monitor-ec2-hosts/) across their applications. Customers can visualize the health of their hosts by performance dimension such as CPU or memory, and slice and dice hundreds of hosts in a single view using filters such as instance type, instance state, or security groups. It enables a side-by-side comparison of a group of EC2 hosts and provides granular insights into an individual host. -![ec2 resource health]((../imagesec2-resource-health.png) +![ec2 resource health](../images/ec2-resource-health.png) ## Monitoring And Observability using Open Source Tools @@ -180,7 +177,7 @@ Resource Health in CloudWatch ServiceLens is a fully managed solution that custo [AWS Distro for OpenTelemetry (ADOT)](https://aws.amazon.com/otel) is a secure, production-ready, AWS-supported distribution of the OpenTelemetry project. Part of the Cloud Native Computing Foundation, OpenTelemetry provides open source APIs, libraries, and agents to collect distributed traces and metrics for application monitoring. With AWS Distro for OpenTelemetry, customers can instrument applications just once to send correlated metrics and traces to multiple AWS and Partner monitoring solutions. -![AWS Distro for Open Telemetry Overview]((../imagesadot.png) +![AWS Distro for Open Telemetry Overview](../images/adot.png) AWS Distro for OpenTelemetry (ADOT) provides a distributed monitoring framework that enables correlating data for monitoring application performance and health in an easy way which is critical for greater service visibility and maintenance. @@ -192,7 +189,7 @@ The key components of ADOT are SDKs, auto-instrumentation agents, collectors and [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector): The collector in the distribution is built using the upstream OpenTelemetry collector. Added AWS-specific exporters to the upstream collector to send data to AWS services including AWS X-Ray, Amazon CloudWatch and Amazon Managed Service for Prometheus. -![adot architecture]((../imagesadot-arch.png) +![adot architecture](../images/adot-arch.png) #### Metrics & Traces through ADOT Collector & Amazon CloudWatch @@ -200,7 +197,7 @@ AWS Distro for OpenTelemetry (ADOT) Collector along with the CloudWatch agent ca To support OpenTelemetry metrics in Amazon CloudWatch, [AWS EMF Exporter for OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/awsemfexporter) converts OpenTelemetry format metrics to CloudWatch Embedded Metric Format(EMF) which enables applications integrated in OpenTelemetry metrics to be able to send high-cardinality application metrics to CloudWatch. [The X-Ray exporter](https://aws-otel.github.io/docs/getting-started/x-ray#configuring-the-aws-x-ray-exporter) allows traces collected in an OTLP format to be exported to [AWS X-ray](https://aws.amazon.com/xray/). -![adot emf architecture]((../imagesadot-emf.png) +![adot emf architecture](../images/adot-emf.png) ADOT Collector on Amazon EC2 can be installed through AWS CloudFormation or using [AWS Systems Manager Distributor](https://catalog.workshops.aws/observability/en-US/aws-managed-oss/ec2-monitoring/configure-adot-collector) to collect application metrics. @@ -208,7 +205,7 @@ ADOT Collector on Amazon EC2 can be installed through AWS CloudFormation or usin [Prometheus](https://prometheus.io/) is a standalone open-source project and maintained independently for systems monitoring and alerting. Prometheus collects and stores metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels. -![Prometheus Architecture]((../imagesPrometheus.png) +![Prometheus Architecture](../images/Prometheus.png) Prometheus is configured via command line flags and all the configuration details are maintained in the prometheus.yaml file. The 'scrape_config' section within the configuration file specifies the targets and parameters specifying how to scrape them. [Prometheus Service Discovery](https://github.com/prometheus/prometheus/tree/main/discovery) (SD) is a methodology of finding endpoints to scrape for metrics. Amazon EC2 service discovery configurations allow retrieving scrape targets from AWS EC2 instances are configured in the `ec2_sd_config`. @@ -221,23 +218,23 @@ The CloudWatch agent on EC2 instances can be installed & configured with Prometh Customers can choose to have an all open-source setup for their observability needs. For which, AWS Distro for OpenTelemetry (ADOT) Collector can be configured to scrape from a Prometheus-instrumented application and send the metrics to Prometheus Server. There are three OpenTelemetry components involved in this flow, that are the Prometheus Receiver, the Prometheus Remote Write Exporter, and the Sigv4 Authentication Extension. Prometheus Receiver receives metric data in Prometheus format. Prometheus Exporter exports data in Prometheus format. Sigv4 Authenticator extension provides Sigv4 authentication for making requests to AWS services. -![adot prometheus architecture]((../imagesadot-prom-arch.png) +![adot prometheus architecture](../images/adot-prom-arch.png) #### Prometheus Node Exporter [Prometheus Node Exporter](https://github.com/prometheus/node_exporter) is an open-source time series monitoring and alerting system for cloud environments. Amazon EC2 Instances can be instrumented with Node Exporter to collect and store node-level metrics as time-series data, recording information with a timestamp. Node exporter is a Prometheus exporter which can expose variety of host metrics via URL http://localhost:9100/metrics. -![prometheus metrics screenshot]((../imagesprom-metrics.png) +![prometheus metrics screenshot](../images/prom-metrics.png) Once the metrics are created, they can be sent to [Amazon Managed Prometheus](https://aws.amazon.com/prometheus/). -![amp overview]((../imagesamp-overview.png) +![amp overview](../images/amp-overview.png) ### Streaming Logs from Amazon EC2 Instances using Fluent Bit Plugin [Fluent Bit](https://fluentbit.io/) is an open source and multi-platform log processor tool for handling data collection at scale, collecting & aggregating diverse data that deal with various sources of information, variety of data formats, data reliability, security, flexible routing and multiple destinations. -![fluent architecture]((../imagesfluent-arch.png) +![fluent architecture](../images/fluent-arch.png) Fluent Bit helps create an easy extension point for streaming logs from Amazon EC2 to AWS services including Amazon CloudWatch for log retention and analytics. The newly-launched [Fluent Bit plugin](https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit#new-higher-performance-core-fluent-bit-plugin) can route logs to Amazon CloudWatch. @@ -245,13 +242,13 @@ Fluent Bit helps create an easy extension point for streaming logs from Amazon E [Amazon Managed Grafana](https://aws.amazon.com/grafana/) is a fully managed service based on the open source Grafana project, with rich, interactive & secure data visualizations to help customers instantly query, correlate, analyze, monitor, and alarm on metrics, logs, and traces across multiple data sources. Customers can create interactive dashboards and share them with anyone in their organization with an automatically scaled, highly available, and enterprise-secure service. With Amazon Managed Grafana, customers can manage user and team access to dashboards across AWS accounts, AWS regions, and data sources. -![grafana overview]((../imagesgrafana-overview.png) +![grafana overview](../images/grafana-overview.png) Amazon Managed Grafana can be added with Amazon CloudWatch as a data source by using the AWS data source configuration option in the Grafana workspace console. This feature simplifies adding CloudWatch as a data source by discovering existing CloudWatch accounts and manage the configuration of the authentication credentials that are required to access CloudWatch. Amazon Managed Grafana also supports [Prometheus data sources](https://docs.aws.amazon.com/grafana/latest/userguide/prometheus-data-source.html), i.e. both self-managed Prometheus servers and Amazon Managed Service for Prometheus workspaces as data sources. Amazon Managed Grafana comes with a variety of panels, makes it easy to construct the right queries and customize the display properties allowing customers to create the dashboards they need. -![grafana dashboard]((../imagesgrafana-dashboard.png) +![grafana dashboard](../images/grafana-dashboard.png) ## Conclusion