Skip to content

Commit

Permalink
Merge pull request #17339 from newrelic/sd-log-live-archives-clean-up
Browse files Browse the repository at this point in the history
fix(logs): making changes to live archives documentation for clarity
  • Loading branch information
homelessbirds authored May 20, 2024
2 parents f99ff1f + 6592abb commit 52897c7
Show file tree
Hide file tree
Showing 2 changed files with 45 additions and 58 deletions.
9 changes: 9 additions & 0 deletions src/content/docs/logs/get-started/live-archives-billing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,15 @@ For the purposes of this example, in month 9 we would:

Adding up all the days' measurements, each of 1 GB Month, gives a total of 30 GB Months by Oct 1. Therefore, this customer has used 30 GB Months of storage for the month of September.

## When will you see your data?

Live archive data will become accessible only after the following conditions are met:

* **Live archive feature activation**: You have explicitly enabled the live archive feature within your New Relic account.
* **Logging retention period met**: The duration specified by your account's live archive retention period has elapsed since the data was initially logged.

For example, if you enable live archive feature on January 1st and set your live archive retention to 30 days, live archive data will become available on or after February 1st. If your logging retention is set to 60, 90, or 120 days, you will need to wait until your respective retention period has passed to use this feature.

## Other calculation details [#other-calc-details]

Here are some additional aspects of our calculation:
Expand Down
94 changes: 36 additions & 58 deletions src/content/docs/logs/get-started/live-archives.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,76 +17,55 @@ signupBanner:
freshnessValidatedDate: never
---


import logsEditDataRetentionLiveArchives from 'images/logs_screenshot-crop_edit-data-retention-live-archives.webp'


import logsCreateaPartitionUsingLiveArchives from 'images/logs_screenshot-crop_create-a-partition-using-live-archives.webp'


import logsQueryLiveArchivesNrql from 'images/logs_screenshot-crop_query-live-archives-nrql.webp'


import logsLuceneLiveArchives from 'images/logs_screenshot-crop_lucene-live-archives.webp'


import logsEditDataPartitionUsingLiveArchives from 'images/logs_screenshot-crop_edit-data-partition-using-live-archives.webp'


Live archives allows you to store logs for as long as you need. New Relic has a thirty-day retention for logs by default, but with live archives, you can extend that period for up to seven years. Like old tax documents, sometimes you don't need your data close at hand, but it's safe to keep the information available for reference if needed.

Is live archives for you? Here are some use cases where you might benefit from this feature:

Is live archives for you? Here are some example use cases:


* <DoNotTranslate>**Internal retention requirements**</DoNotTranslate>: Have an internal retention period where you would like to keep logs and be able to query them only when necessary? Live archives can help with extended storage and querying capability.
* <DoNotTranslate>**Trends and long-term debugging**</DoNotTranslate>: Long term log storage can help improve year-over-year performance. For example, if you manage a retail company, comparing logs from one holiday season to the next can improve performance.

* <DoNotTranslate>**Internal retention requirements**</DoNotTranslate>: Does your company have a policy that requires you to keep logs for a certain period? The extended storage periods of live archives may help you meet your requirements.
* <DoNotTranslate>**Trends and long-term debugging**</DoNotTranslate>: Long-term log storage can help improve year-over-year performance. For example, if you manage a retail company, you can compare logs from one holiday season to the next to see how you can improve performance.

## How live archives works [#how-it-works]


The live archives feature offers long-term log data storage that you intend to query sparingly for a lower storage cost. With live archives, you can have peace of mind knowing your logs are available for a longer period for potential audits or long-term debugging, but you will only be charged for querying when you _query old data_.

The live archives feature offers long-term log data storage that you intend to query sparingly for a lower storage cost. With live archives, you can have peace of mind knowing your logs are available for a longer period for potential audits or long-term debugging, but you will be charged separately for storage and querying your old data.

## Get started
<Steps>
<Step>
### Enable live archives [#enable]


First, you need to request live archives capabilities.


1. On the <DoNotTranslate>**Data retention**</DoNotTranslate> UI page, located in the [data management hub](https://one.newrelic.com/data-management-hub).
2. Scroll down to <DoNotTranslate>**Logging live archives**</DoNotTranslate>.
3. Hit <DoNotTranslate>**Request**</DoNotTranslate>. A member of our team will reach out to you.
</Step>


<Step>
Once you've enabled live archives, you need to edit your data retention.


1. On the <DoNotTranslate>**Data retention**</DoNotTranslate> UI page, located in the [data management hub](https://one.newrelic.com/data-management-hub) select <DoNotTranslate>**Edit data retention**</DoNotTranslate>.
2. Scroll down to <DoNotTranslate>**Logging live archives**</DoNotTranslate> and enter the total number of days you'd like your data stored. Keep in mind that this includes the standard retention days.
3. Click <DoNotTranslate>**Apply changes**</DoNotTranslate>.


Keep in mind that you will need [Admin capabilities](/docs/accounts/accounts-billing/new-relic-one-user-management/user-management-concepts/#roles) to edit data retention.


<img
title= "edit data retention"
alt= "A screenshot showing users how to edit their data retention."
src={logsEditDataRetentionLiveArchives}
/>


<figcaption>
Go to <DoNotTranslate>**one.newrelic.com > [Data management hub](https://one.newrelic.com/data-management-hub) > Data retention > Select logging live archives > Edit data retention**</DoNotTranslate>: Select your archive duration to store your logs for as long as you need.
</figcaption>
</Step>


<Step>
### Create a partition using live archives [#partitions]

Expand All @@ -96,19 +75,20 @@ A partition is a way to group logs based on specific criteria. Creating partitio
2. To the left of the logs query bar, click <DoNotTranslate>**Data partitions**</DoNotTranslate>, then click <DoNotTranslate>**Create data partition**</DoNotTranslate>.
3. Define a Partition name as an alphanumeric string that begins with Log_.
4. Select the field <DoNotTranslate>**Use live archives on this partition**</DoNotTranslate>.
5. The <DoNotTranslate>**Total effective retention**</DoNotTranslate> days will be updated to reflect the regular retention _and_ the [live archives retention](/docs/logs/get-started/live-archives-billing/#what-counts).

<img
title= "create a live archives partition"
alt= "A screenshot showing a user how to create a partition using live archives"
src={logsCreateaPartitionUsingLiveArchives}
/>
<img
title= "create a live archives partition"
alt= "A screenshot showing a user how to create a partition using live archives"
src={logsCreateaPartitionUsingLiveArchives}
/>

<figcaption>
Go to <DoNotTranslate>**[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Logs > Data partitions**</DoNotTranslate> to create a partition using live archives.
</figcaption>
<figcaption>
Go to <DoNotTranslate>**[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Logs > Data partitions**</DoNotTranslate> to create a partition using live archives.
</figcaption>

5. The <DoNotTranslate>**Total effective retention**</DoNotTranslate> days will be updated to reflect the regular retention _and_ the [live archives retention](/docs/logs/get-started/live-archives-billing/#what-counts).

You can also edit an existing partition clicking the <Icon name="fe-more-horizontal"/> icon on the <DoNotTranslate>**Data partitions**</DoNotTranslate> page.
You can also edit an existing partition clicking the <Icon name="fe-more-horizontal"/> icon on the <DoNotTranslate>**Data partitions**</DoNotTranslate> page. If you use an existing partition, the same parameters that you selected for the original partition will apply to your live archive data.

<img
width="60%;"
Expand All @@ -117,12 +97,16 @@ You can also edit an existing partition clicking the <Icon name="fe-more-horizon
src={logsEditDataPartitionUsingLiveArchives}
/>

For more information about how to use partitions, see our doc [Organize data with partitions](/docs/logs/ui-data/data-partitions/).
</Step>

<Step>
### Query live archives with logs syntax [#query]
## Query your logs

There are two ways to query live archives: with Lucene or NRQL. If you want to use logs syntax that you're familiar with outside of New Relic, we suggest querying in the <DoNotTranslate>**Logs**</DoNotTranslate> UI. You'll need the **Live archives query** capability to query live archives.
There are a few ways to query live archives: Lucene, NRQL, or NRQL in the NerdGraph API. For all three methods, you'll need the **Live archives query** capability to query live archives.

### Query live archives with Lucene query syntax [#query]
f you're familiar with Lucene query syntax, you can execute it in our <DoNotTranslate>**Logs**</DoNotTranslate> UI.

1. Go to <DoNotTranslate>**[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Logs**</DoNotTranslate>.
2. Using the search bar, run your desired query. Practicing your query first is best to ensure it works as expected.
Expand All @@ -139,19 +123,16 @@ There are two ways to query live archives: with Lucene or NRQL. If you want to u
Go to <DoNotTranslate>**[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Logs**</DoNotTranslate> to query data stored with live archives.
</figcaption>

</Step>


<Step>
### Query live archives with NRQL [#nrql]

If you want to query using the same language you use for errors inbox or alerts, we suggest querying in NRQL. You'll need the **Live archives query** capability to query live archives.
If you want to query using the same language you use for errors inbox or alerts, we suggest querying in NRQL.

1. Go to <DoNotTranslate>**[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Logs**</DoNotTranslate>.
2. Enter the NRQL query you want to run.
3. Practice the query first to make sure it works as expected.
4. Toggle <DoNotTranslate>**Use live archives**</DoNotTranslate>.
5. A screen will show you that you're about to query live archives. Querying live archives is more expensive, so only select <DoNotTranslate>**Query live archives**</DoNotTranslate> if you're ready.
2. Click **NRQL** in the top right corner.
3. Enter the NRQL query you want to run.
4. Practice the query first to make sure it works as expected.
5. Toggle <DoNotTranslate>**Use live archives**</DoNotTranslate>.
6. A screen will show you that you're about to query live archives. Querying live archives is more expensive, so only select <DoNotTranslate>**Query live archives**</DoNotTranslate> if you're ready.

<img
title= "nrql live archives"
Expand All @@ -162,12 +143,10 @@ If you want to query using the same language you use for errors inbox or alerts,
<figcaption>
Go to <DoNotTranslate>**[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Logs**</DoNotTranslate>, then select <DoNotTranslate>**NRQL**</DoNotTranslate> to query data stored with live archives using the query builder.
</figcaption>
</Step>

<Step>
### Query live archives using Nerdgraph API [#api]
### Query live archives using NerdGraph API [#api]

You can also query your live archives data using the Nerdgraph API. You'll need the **Live archives query** capability to query live archives. You must include the `{eventNamespaces: "Logging:Archive"}` in the NRQL options to specify the live archives `eventNamespace`.
You can also query your live archives data using NRQL in the NerdGraph API. You must include the `{eventNamespaces: "Logging:Archive"}` in the NRQL options to specify the live archives `eventNamespace`.

Here is an example query:

Expand All @@ -193,7 +172,6 @@ query {

Live archives is billed differently than the rest of your logs. You can read more about it in our [Live archives storage: Billing and rules documentation](/docs/logs/get-started/live-archives-billing/). Live archives has three billing pillars.


* <DoNotTranslate>**Data**</DoNotTranslate>: You will be charged for when you originally ingest the regular logs.
* <DoNotTranslate>**Storage**</DoNotTranslate>: Live archives charges a fee for storing your data with live archives.
* <DoNotTranslate>**Compute**</DoNotTranslate>: Queries of live archives data will be billed in Compute Capacity Units. See our [Usage plan documentation](https://docs.newrelic.com/docs/licenses/license-information/usage-plans/new-relic-usage-plan/#list-price) for more information about Compute Capacity Units.
Expand All @@ -220,12 +198,12 @@ Once you query live archives, you can explore your compute consumption in the co
Let's say you need to check if a user logged into a machine from their computer to fulfill a ticket on September 1st. You can use live archives to help confirm that this took place by completing the following steps:

1. Check the time frame that the user logged into their machine.
2. Test your query in <DoNotTranslate>**Logs**</DoNotTranslate> to ensure it's working as expected.
2. Test your query in <DoNotTranslate>**Logs**</DoNotTranslate> to ensure it's working as expected. This is a dry run and won't return any data, but you're just validating the query. If it works, you can then run the more expensive query on your archived logs.

Example query:
```
"logtype":"linux_audit" "type":"USER_LOGIN" "hostname":"apache_svr01"
```
Example query:
```
"logtype":"linux_audit" "type":"USER_LOGIN" "hostname":"apache_svr01"
```
3. Change the time selector to search for the dates the user logged into their machine.
4. Select <DoNotTranslate>**Query live archives**</DoNotTranslate>.

Expand Down

0 comments on commit 52897c7

Please sign in to comment.