Merge pull request #19470 from newrelic/daily-release/12-09-24-morning

Daily release/12 09 24 morning

src/content/docs/infrastructure/amazon-integrations/get-started/introduction-aws-integrations.mdx

@@ -23,10 +23,11 @@ redirects:
 freshnessValidatedDate: 2024-09-20
 ---
 
-New Relic offers several methods to collect performance data about your Amazon Web Services (AWS) environment. Once you've connected New Relic with AWS, you can access your system's AWS telemetry, letting you closely monitor the health of your AWS stack. This doc describes how to get started with AWS monitoring and provides an overview about AWS costs. 
+New Relic offers several methods to collect performance data about your Amazon Web Services (AWS) environment. After you've connected New Relic with AWS, you can access your system's AWS telemetry, letting you closely monitor the health of your AWS stack. Our AWS observability solution supports [Amazon CloudWatch Metric Streams integration](/docs/infrastructure/amazon-integrations/get-started/introduction-aws-integrations/#monitor-aws) to get metrics, and [Forward AWS service logs to New Relic](/install/aws-logs/) to get logs into New Relic. This document describes how to get started with AWS monitoring and provides an overview about AWS costs. 
 
 ## Start with the CloudWatch Metric Streams integration [#monitor-aws]
 
+
 Our AWS observability solution is the Amazon CloudWatch Metric Streams integration, which collects data about your entire AWS stack with [Amazon CloudWatch Metric Streams](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Metric-Streams.html). For the majority of use cases, you begin monitoring your AWS stack by [installing our Amazon CloudWatch Metric Streams integration](/install/aws-cloudwatch).
 
 Our CloudWatch Metric Streams integration provides the best data about your AWS stack, but if you use [integrations not supported by CloudWatch Metric Streams](/docs/infrastructure/amazon-integrations/connect/connect-aws-new-relic-infrastructure-monitoring/#integrations-not-replaced-streams), you may need to use our legacy API polling solution. If you're currently using API polling for supported integrations, we recommend you [migrate your data to Amazon CloudWatch Metric Streams](/install/aws-cloudwatch/?task=migrate).

src/content/docs/logs/forward-logs/guidelines-to-upgrade-to-new-lambda.mdx

@@ -0,0 +1,16 @@
+---
+title: Guidelines for upgrading to the new Lambda
+tags:
+  - Logs
+  - Upgrading guidelines
+  - Upgrade to new Lambda
+metaDescription: 'Upgrade to the new Lambda by removing old deployments and avoiding duplicate triggers for consistent data flow to New Relic.'
+freshnessValidatedDate: never
+---
+If you want to upgrade from the old Lambda to the new one, we recommend deleting the previously deployed New Relic S3 Lambda and CloudWatch Lambda. Alternatively, you can delete the previously created triggers before deploying the new one.
+
+If you choose to use both solutions, ensure there are no duplicate triggers to avoid sending duplicate logs from the same resource. We do not recommend this, as it may cause reaching S3 and CloudWatch limits on reading logs, which can lead to inconsistencies in the data reaching New Relic.
+
+<Callout variant="tip">
+  To return to AWS integration for log forwarding, [click here](/install/aws-logs/?service=aws_services).
+</Callout>

src/content/docs/logs/forward-logs/manage-triggers-and-filters-with-cloudformation.mdx

@@ -0,0 +1,18 @@
+---
+title: Manage triggers and filters with CloudFormation
+tags:
+  - Manage triggers with CloudFormation
+  - Manage filters with CloudFormation
+  - Manage triggers and filters with CloudFormation
+metaDescription: 'Manage S3 triggers and CloudWatch filters with CloudFormation for effective resource management and proper cleanup upon stack deletion.'
+freshnessValidatedDate: never
+---
+When you deploy a CloudFormation stack, it creates triggers on an S3 bucket and filters on a CloudWatch log group. Deleting the stack will remove these triggers and filters.
+
+If you manually create triggers in an S3 bucket or filters on CloudWatch log groups and set the destination as a Lambda function created by New Relic, these are also supported. However, we do not recommend this approach. Deleting the stack will not remove these manually created triggers or filters, as the stack does not have permission to delete them.
+
+This guidance is important to ensure effective resource management and to prevent unintended triggers or filters from remaining after a stack deletion.
+
+<Callout variant="tip">
+  To return to AWS integration for log forwarding, [click here](/install/aws-logs/?service=aws_services).
+</Callout>

src/content/docs/logs/troubleshooting/aws-log-integration-issues/custom-attributes-not-appearing.mdx

@@ -0,0 +1,22 @@
+---
+title: Custom attributes not appearing in logs
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Troubleshooting
+metaDescription: 'Troubleshooting tips if the custom attributes not appearing in the logs.'
+freshnessValidatedDate: never
+---
+
+## Problem
+
+Logs are not appearing in New Relic for a CloudWatch via Firehose integration setup.
+
+## Solution
+
+Possible reasons and their solutions:
+
+* For S3 via Lambda or CloudWatch via Lambda integrations, the `CUSTOM_META_DATA` environment variable may not be set, or it may not have a valid JSON value. Verify it.
+* For CloudWatch via Firehose, the key-value pairs may not be set correctly. Ensure your key-value pairs are present under <DNT>**Parameters**</DNT>.
+* Your key names may overlap with forbidden keywords.

src/content/docs/logs/troubleshooting/aws-log-integration-issues/log-forwarding-restrictions.mdx

@@ -0,0 +1,24 @@
+---
+title: AWS Log forwarding restrictions
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Log forwarding restrictions
+metaDescription: 'Restrictions for log forwarding with New Relic.'
+freshnessValidatedDate: never
+---
+
+
+
+New Relic has the following restrictions with AWS integration for log forwarding when ingesting logs via S3 or Firehose:
+
+### For ingesting AWS logs from S3 [#s3-restrictions]
+* The log file size should be less that 80 MB.
+* The size of a single log line should be less than 8 MB.
+* Log lines between 1mb and 8mb will be split into multiple messages and forwarded to New Relic.
+* New Relic currently doesn't support Client Json or log line parsing.
+* New Relic currently supports only `Gzip` and `Bzip2` compression formats for log files. It processes files in other compressed formats as uncompressed files.
+### For ingesting logs from CloudWatch [#cloudwatch-restrictions]
+* New Relic currently doesn't support client Json or log line parsing.
+* New Relic currently supports only [Lambda layers](https://github.com/newrelic/newrelic-lambda-layers) for parsing logic.

src/content/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-cloudwatch-firehose.mdx

@@ -0,0 +1,22 @@
+---
+title: Logs not appearing from CloudWatch via Firehose
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Troubleshooting
+metaDescription: 'Troubleshooting tips if the Logs are not appearing from CloudWatch via Firehose.'
+freshnessValidatedDate: never
+---
+
+## Problem
+
+Logs are not appearing in New Relic for a CloudWatch via Firehose integration setup.
+
+## Solution
+
+Possible reasons and their solutions:
+
+* The API key may be incorrect. Navigate to <DNT>**Amazon Data Firehose > Firehose streams > your-stream > Edit destination settings**</DNT>, and update the API key with the correct ingest license key.
+* The provided endpoint may be wrong. Verify the endpoint.
+* Check the S3 logs or Cloudwatch error logs for any issues.

src/content/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-cloudwatch-lambda.mdx

@@ -0,0 +1,26 @@
+---
+title: Logs not appearing from CloudWatch via Lambda
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Troubleshooting
+metaDescription: 'Troubleshooting tips if the Logs are not appearing from CloudWatch via Lambda.'
+freshnessValidatedDate: never
+---
+
+## Problem
+
+Logs are not appearing in New Relic for a CloudWatch via Lambda integration setup.
+
+## Solution
+
+Possible reasons and their solutions:
+
+* The New Relic ingest license key may not be valid. In your AWS account, find the resource with Logical Id `NewRelicLogsLicenseKeySecret` from <DNT>**CloudFormation stack > Resources**</DNT>, and verify the key.
+* To debug further, navigate to <DNT>**CloudFormation Stack > Resources > NewRelicServerlessLogForwarder (Lambda Function) > Configuration > Environment Variables**</DNT> and set the `DEBUG_MODE` parameter to true. Then, check the CloudWatch logs for the Lambda function to get the debug logs.
+* The event may have been a genuine failure. To check, navigate to <DNT>**CloudFormation Stack > Resources > NewRelicLogForwarderDLQ**</DNT>, or search for your DLQ in AWS SQS, and examine the DLQ resource.
+* The New Relic region and the ingest license key provided to the Lambda function or Firehose may not match. Verify and match it.
+
+  For example, if the New Relic region is EU, provide a New Relic ingest license key from the EU region.
+* The `NewRelicServerlessLogForwarder` may not have the CloudWatch trigger. Examine the output of the nested CloudFormation stacks.

src/content/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-s3-lambda.mdx

@@ -0,0 +1,22 @@
+---
+title: Logs not appearing from S3 via Lambda
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Troubleshooting
+metaDescription: 'Troubleshooting tips if the Logs are not appearing from S3 via Lambda.'
+freshnessValidatedDate: never
+---
+
+## Problem
+
+Logs are not appearing in New Relic for a S3 via Lambda integration setup.
+
+## Solution
+
+Possible reasons and their solutions:
+
+* Verify the possible reasons in [Logs not appearing from CloudWatch vai Lambda](/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-cloudwatch-lambda).
+* S3 files may not adhere to the [Log forwarding restrictions](/docs/logs/troubleshooting/aws-log-integration-issues/log-forwarding-restrictions) in New Relic.
+* The `NewRelicServerlessLogForwarder` may not have the S3 trigger. Check the output of the nested CloudFormation stacks, and add it.

src/content/docs/logs/troubleshooting/aws-log-integration-issues/stack-creation-fails.mdx

@@ -0,0 +1,28 @@
+---
+title: Failing to create a Stack
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Troubleshooting
+metaDescription: 'Troubleshooting tips if the stack creation fails.'
+freshnessValidatedDate: never
+---
+
+## Problem
+
+Creating a CloudFormation stack may fail due to various reasons.
+
+## Solution
+
+Possible reasons and their solutions:
+
+* The IAM role you have selected during the CloudFormation stack creation may not have sufficient [permissions](https://docs.aws.amazon.com/ARG/latest/userguide/gettingstarted-prereqs-permissions.html).
+* If you did not select an IAM role, make sure your account has the required [permissions](/docs/accounts/accounts-billing/new-relic-one-user-management/user-permissions/#logs).
+* If the failure message appears as following, delete or rename the existing resources with that name, and retry.
+
+  `Resource type {resource-name} with identifier {resource-identifier-name} already exists.`
+
+* Check the output of the nested CloudFormation stacks to identify potential errors.
+* Ensure the selected S3 buckets don’t have event notifications on any object creation events.
+* Ensure the selected CloudWatch log groups don’t have similar subscription filters.

src/content/docs/logs/troubleshooting/aws-log-integration-issues/stack-deletion-fails.mdx

@@ -0,0 +1,20 @@
+---
+title: Failing to delete a Stack
+type: troubleshooting
+tags:
+  - Logs
+  - Log management
+  - Troubleshooting
+metaDescription: 'Troubleshooting tips if the stack deletion fails.'
+freshnessValidatedDate: never
+---
+
+## Problem
+
+When deleting a CloudFormation stack, it may fail due to various reasons.
+
+## Solution
+
+Possible reason and its solution:
+
+* If the S3 buckets created by the stack (e.g., for Firehose error logs) is not empty, then it cannot be deleted. archive the contents and then retry deleting the stack.

src/install/aws-logs/appInfo_service.mdx

@@ -1,5 +1,6 @@
 ---
 componentType: appInfoConfigOption
 optionType: service
-headingText: What service are you forwarding from?
+headingText: Select an AWS service to instrument
 ---
+You can send AWS service logs to New Relic using various methods. Select an AWS service to learn how you can collect and forward logs to New Relic.

src/install/aws-logs/aws_services/before-you-begin.mdx

@@ -0,0 +1,13 @@
+---
+headingText: Before you begin
+componentType: default
+---
+
+To set up AWS integration, ensure you have the following:
+* A New Relic [license key](https://one.newrelic.com/api-keys). Don't have one? [Sign up for free!](https://newrelic.com/signup) No credit card required.
+* An AWS account with [permissions](https://docs.aws.amazon.com/ARG/latest/userguide/gettingstarted-prereqs-permissions.html) to deploy new AWS resources and [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html).
+* A New Relic account with [permissions](/docs/accounts/accounts-billing/new-relic-one-user-management/user-permissions/#logs) to set up log forwarding.
+
+<Callout variant="important">
+If you are already using Lambda for S3 or CloudWatch, refer the [Guidelines for upgrading to the new Lambda](/docs/logs/forward-logs/guidelines-to-upgrade-to-new-lambda).
+</Callout>

src/install/aws-logs/aws_services/configure-log-forwarding.mdx

@@ -0,0 +1,43 @@
+---
+headingText: Configure logs forwarding
+componentType: default
+---
+
+There are multiple ways to send New Relic logs from your AWS services. Choose the path that best suits your needs. Both the integration instructions and the CloudFormation stack will adapt based on your selections.
+
+### Select AWS sink(s) [#select-aws-sinks]
+
+Most AWS service logs are generated in S3 buckets or CloudWatch log groups. Select one or both AWS sinks.
+<img
+  title="Select AWS sinks"
+  alt="Select AWS sinks"
+  src="/images/select-logs-intgration-path-1.webp"
+/>
+### Select log integration path(s) [#select-log-integration-paths]
+
+Consider using the <DNT>**Amazon Kinesis Data Firehose**</DNT> for direct and high-volume data streaming, and the <DNT>**New Relic Lambda Function**</DNT> forwarder if you need high control over data transformation before forwarding logs.
+<img
+  title="Select logs integration path"
+  alt="Select logs integration path"
+  src="/images/select-logs-intgration-path-2.webp"
+/>
+
+### Set up triggers(s) [#set-up-triggers]
+Setting up triggers ensures that new logs are sent to New Relic every time an AWS service logs to an S3 bucket or a CloudWatch log group. With a single CloudFormation stack, you can configure up to 40 triggers (S3 buckets and/or CloudWatch log groups).
+
+* For AWS service logs stored in S3 buckets, specify the **S3 bucket name** and **Prefix** (optional).
+* For AWS service logs stored in CloudWatch log groups, specify the **Log Group** name and **Filter** (optional).
+
+<img
+  title="Set up triggers"
+  alt="Set up triggers"
+  src="/images/set-up-trigger.webp"
+/>
+
+### Select New Relic logs API endpoint [#select-api-endpoint]
+The system automatically sets the endpoint based on the region of your New Relic account.
+* For US account, logs are sent to `https://log-api.newrelic.com/log/v1`
+* For UE account, logs are sent to `https://log-api.eu.newrelic.com/log/v1`
+
+### Add custom attributes [#add-custom-attributes]
+Add custom attributes to organize your logs and make it easier to search, filter, analyze, and parse.

src/install/aws-logs/aws_services/deploy-cloudformation-stack.mdx

@@ -0,0 +1,8 @@
+---
+headingText: Deploy CloudFormation stack
+componentType: default
+---
+
+After you customize the CloudFormation stack parameters to meet your requirements, click <DNT>**Create stack**</DNT>.
+Creating the stack's resources may take several minutes and may generate multiple stacks, depending on how you configure log and metric forwarding. With a single deployment command, you can create a new stack to forward logs from multiple AWS resources.
+If stack creation fails, refer to the [troubleshooting procedure](/docs/logs/troubleshooting/aws-log-integration-issues/stack-creation-fails).

src/install/aws-logs/aws_services/launch-cloudformation-template.mdx

@@ -0,0 +1,9 @@
+---
+headingText: Launch CloudFormation template
+componentType: default
+---
+
+1. Click <DNT>**Launch your CloudFormation in AWS**</DNT> to navigate to your AWS Console. The CloudFormation stack includes the configured parameters based on your selections in New Relic One, so no additional editing is necessary unless you want to make changes.
+2. Copy and paste your license key into your AWS console.
+
+We recommend you store your New Relic license key in <DNT>**[AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html)**</DNT> by keeping the environment variable set to `true` (default value).

src/install/aws-logs/aws_services/see-your-data.mdx

@@ -0,0 +1,31 @@
+---
+headingText: See your data
+componentType: default
+---
+
+After the new stack is created, it forwards your collected logs to New Relic One. To see your data:
+
+* Navigate to the [logs UI](https://one.newrelic.com/launcher/logger.log-launcher).
+* OR, navigate to the [New query explorer](/docs/chart-builder/use-chart-builder/choose-data/use-advanced-nrql-mode-specify-data) to query your data.
+
+For logs forwarded via the New Relic Lambda forwarder:
+   ```
+   SELECT * FROM Log WHERE instrumentation.provider = 'aws' AND instrumentation.name = 'lambda'
+   ```
+
+For logs forwarded via the Kinesis Data Firehose integration:
+   ```
+   SELECT * FROM Log WHERE instrumentation.provider = 'aws' AND instrumentation.name = 'firehose'
+   ```
+
+If no data appears after you enable the log management capabilities, or the custom attributes are not appearing in the log, refer to the following troubleshooing procedures:
+* [Logs not appearing from CloudWatch via Firehose](/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-cloudwatch-firehose)
+* [Logs not appearing from CloudWatch via Lambda](/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-cloudwatch-lambda)
+* [Logs not appearing from S3 via Lambda](/docs/logs/troubleshooting/aws-log-integration-issues/logs-not-appearing-from-s3-lambda)
+* [Custom attributes not appearing in logs](/docs/logs/troubleshooting/aws-log-integration-issues/custom-attributes-not-appearing)
+* [Failing to delete a Stack](/docs/logs/troubleshooting/aws-log-integration-issues/stack-deletion-fails)
+* [Log forwarding restrictions](/docs/logs/troubleshooting/aws-log-integration-issues/log-forwarding-restrictions)
+
+<Callout variant="important">
+When configuring your CloudFormation integration, it's important to understand how triggers and filters are managed. For guidance on handling S3 triggers and CloudWatch filters, refer to [Manage triggers and filters with CloudFormation](/docs/logs/forward-logs/manage-triggers-and-filters-with-cloudformation).
+</Callout>

src/install/aws-logs/aws_services/set-up-aws-integration-with-log-forwarding.mdx

@@ -0,0 +1,12 @@
+---
+headingText: Set up your AWS integration with log forwarding
+componentType: default
+---
+
+Using a single CloudFormation template, you can easily set up your AWS integration and forward New Relic AWS services [metrics](/install/aws-cloudwatch/), logs, or both.
+1. Log in to your [New Relic](https://one.newrelic.com/) account.
+2. Navigate to <DNT>** + Integrations & Agents**</DNT>.
+3. Search and select <DNT>**Integrate your AWS account**</DNT> from the <DNT>**All**</DNT> tab.
+3. Select a New Relic account and continue.
+4. Select the telemetry type you want to send to your New Relic account.
+5. Select <DNT>**Automate AWS with CloudFormation**</DNT> and continue. This is currently the only setup method available for log integration.