diff --git a/manage-data/data-store/manage-data-from-the-command-line.md b/manage-data/data-store/manage-data-from-the-command-line.md index 60ec06d3a..330206077 100644 --- a/manage-data/data-store/manage-data-from-the-command-line.md +++ b/manage-data/data-store/manage-data-from-the-command-line.md @@ -6,9 +6,133 @@ mapped_urls: # Manage data from the command line -% What needs to be done: Lift-and-shift +Learn how to index, update, retrieve, search, and delete documents in an {{es}} cluster from the command line. -% Use migrated content from existing pages that map to this page: +::::{tip} +If you are looking for a user interface for {{es}} and your data, head on over to [Kibana](/get-started/the-stack.md)! Not only are there amazing visualization and index management tools, Kibana includes realistic sample data sets to play with so that you can get to know what you *could* do with your data. +:::: + +## Before you begin [before-you-begin] + +On the **Overview** page for your new cluster in the Cloud UI, copy the {{es}} endpoint URL under **Endpoints**. + +These examples use the `elastic` user. If you didn’t copy down the password for the `elastic` user, you can [reset the password](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). + +To use these examples, you also need to have the [curl](http://curl.haxx.se/) command installed. + + +## Indexing [indexing] + +To index a document into {{es}}, `POST` your document: + +```bash +curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc -XPOST -H 'Content-Type: application/json' -d '{ + "title": "One", "tags": ["ruby"] +}' +``` + +To show that the operation worked, {{es}} returns a JSON response that looks like `{"_index":"my_index","_type":"_doc","_id":"0KNPhW4BnhCSymaq_3SI","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":0,"_primary_term":1}`. + +In this example, the index `my_index` is created dynamically when the first document is inserted into it. All documents in {{es}} have a `type` and an `id`, which is echoed as `"_type":"_doc"` and `_id":"0KNPhW4BnhCSymaq_3SI` in the JSON response. If no ID is specified during indexing, a random `id` is generated. + + +### Bulk indexing [bulk-indexing] + +To achieve the best possible performance, use the bulk API. + +To index some additional documents with the bulk API: + +```bash +curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/_bulk -XPOST -H 'Content-Type: application/json' -d ' +{"index": {}} +{"title": "Two", "tags": ["ruby", "python"] } +{"index": {}} +{"title": "Three", "tags": ["java"] } +{"index": {}} +{"title": "Four", "tags": ["ruby", "php"] } +' +``` + +Elasticsearch returns a JSON response similar to this one: + +```json +{"took":694,"errors":false,"items":[{"index":{"_index":"my_index","_type":"_doc","_id":"0aNqhW4BnhCSymaqFHQn","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}},{"index":{"_index":"my_index","_type":"_doc","_id":"0qNqhW4BnhCSymaqFHQn","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1,"status":201}},{"index":{"_index":"my_index","_type":"_doc","_id":"06NqhW4BnhCSymaqFHQn","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":2,"_primary_term":1,"status":201}}]} +``` + + +## Updating [updating] + +To update an existing document in {{es}}, `POST` the updated document to `http://ELASTICSEARCH_URL/my_index/_doc/ID`, where the ID is the `_id` of the document. + +For example, to update the last document indexed from the previous example with `"_id":"06NqhW4BnhCSymaqFHQn"`: + +```bash +curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/06NqhW4BnhCSymaqFHQn -XPOST -H 'Content-Type: application/json' -d '{ + "title": "Four updated", "tags": ["ruby", "php", "python"] +}' +``` + +The JSON response shows that the version counter for the document got incremented to `_version":2` to reflect the update. + + +## Retrieving documents [retrieving-documents] + +To take a look at a specific document you indexed, here the last document we updated with the ID `0KNPhW4BnhCSymaq_3SI`: + +```bash +curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/06NqhW4BnhCSymaqFHQn +``` + +This request didn’t include `GET`, as the method is implied if you don’t specify anything else. If the document you are looking for exists, {{es}} returns `found":true` along with the document as part of the JSON response. Otherwise, the JSON response contains `"found":false`. + + +## Searching [searching] + +You issue search requests for documents with one of these {{es}} endpoints: + +```bash +https://ELASTICSEARCH_URL/_search +https://ELASTICSEARCH_URL/INDEX_NAME/_search +``` + +Either a `GET` or a `POST` request with some URI search parameters works, or omit the method to default to `GET` request: + +```bash +curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/_search?q=title:T* +``` + +For an explanation of the allowed parameters, check [URI Search](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search). + +To make {{es}} return a more human readable JSON response, add `?pretty=true` to the request: + +```bash +curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/_search?pretty=true -H 'Content-Type: application/json' -d '{ + "query": { + "query_string": {"query": "*"} + } +}' +``` + +For performance reasons, `?pretty=true` is not recommended in production. You can verify the performance difference yourself by checking the `took` field in the JSON response which tells you how long Elasticsearch took to evaluate the search in milliseconds. When we tested these examples ourselves, the difference was `"took" : 4` against `"took" : 18`, a substantial difference. + +For a full explanation of how the request body is structured, check [Elasticsearch Request Body documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html). You can also execute multiple queries in one request with the [Multi Search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-msearch). + + +## Deleting [deleting] + +You delete documents from {{es}} by sending `DELETE` requests. + +To delete a single document by ID from an earlier example: + +```bash +curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/06NqhW4BnhCSymaqFHQn -XDELETE +``` + +To delete a whole index, here `my_index`: + +```bash +curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index -XDELETE +``` + +The JSON response returns `{"acknowledged":true}` to indicate that the index deletion was a success. -% - [ ] ./raw-migrated-files/cloud/cloud/ec-working-with-elasticsearch.md -% - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-working-with-elasticsearch.md \ No newline at end of file diff --git a/manage-data/ingest/transform-enrich/data-enrichment.md b/manage-data/ingest/transform-enrich/data-enrichment.md index 369e68e30..4d6e2d4df 100644 --- a/manage-data/ingest/transform-enrich/data-enrichment.md +++ b/manage-data/ingest/transform-enrich/data-enrichment.md @@ -6,15 +6,85 @@ mapped_urls: # Data enrichment -% What needs to be done: Lift-and-shift +You can use the [enrich processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/enrich-processor.md) to add data from your existing indices to incoming documents during ingest. -% Use migrated content from existing pages that map to this page: +For example, you can use the enrich processor to: -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/ingest-enriching-data.md -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-mgmt.md +* Identify web services or vendors based on known IP addresses +* Add product information to retail orders based on product IDs +* Supplement contact information based on an email address +* Add postal codes based on user coordinates -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): + +## How the enrich processor works [how-enrich-works] + +Most processors are self-contained and only change *existing* data in incoming documents. + +:::{image} ../../../images/elasticsearch-reference-ingest-process.svg +:alt: ingest process +::: + +The enrich processor adds *new* data to incoming documents and requires a few special components: + +:::{image} ../../../images/elasticsearch-reference-enrich-process.svg +:alt: enrich process +::: + +$$$enrich-policy$$$ + +enrich policy +: A set of configuration options used to add the right enrich data to the right incoming documents. + +An enrich policy contains: + +* A list of one or more *source indices* which store enrich data as documents +* The *policy type* which determines how the processor matches the enrich data to incoming documents +* A *match field* from the source indices used to match incoming documents +* *Enrich fields* containing enrich data from the source indices you want to add to incoming documents + +Before it can be used with an enrich processor, an enrich policy must be [executed](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-execute-policy). When executed, an enrich policy uses enrich data from the policy’s source indices to create a streamlined system index called the *enrich index*. The processor uses this index to match and enrich incoming documents. + + +$$$source-index$$$ + +source index +: An index which stores enrich data you’d like to add to incoming documents. You can create and manage these indices just like a regular {{es}} index. You can use multiple source indices in an enrich policy. You also can use the same source index in multiple enrich policies. $$$enrich-index$$$ -$$$enrich-policy$$$ \ No newline at end of file +enrich index +: A special system index tied to a specific enrich policy. + +Directly matching incoming documents to documents in source indices could be slow and resource intensive. To speed things up, the enrich processor uses an enrich index. + +Enrich indices contain enrich data from source indices but have a few special properties to help streamline them: + +* They are system indices, meaning they’re managed internally by {{es}} and only intended for use with enrich processors and the {{esql}} `ENRICH` command. +* They always begin with `.enrich-*`. +* They are read-only, meaning you can’t directly change them. +* They are [force merged](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge) for fast retrieval. + +## Manage enrich policies [manage-enrich-policies] + +Use the **Enrich Policies** view to add data from your existing indices to incoming documents during ingest. An enrich policy contains: + +* The policy type that determines how the policy matches the enrich data to incoming documents +* The source indices that store enrich data as documents +* The fields from the source indices used to match incoming documents +* The enrich fields containing enrich data from the source indices that you want to add to incoming documents +* An optional [query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-match-all-query.md). + +:::{image} ../../../images/elasticsearch-reference-management-enrich-policies.png +:alt: Enrich policies +:class: screenshot +::: + +When creating an enrich policy, the UI walks you through the configuration setup and selecting the fields. Before you can use the policy with an enrich processor or {{esql}} query, you must execute the policy. + +When executed, an enrich policy uses enrich data from the policy’s source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents. + +Check out these examples: + +* [Example: Enrich your data based on geolocation](/manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md) +* [Example: Enrich your data based on exact values](/manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md) +* [Example: Enrich your data by matching a value to a range](/manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md) diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-working-with-elasticsearch.md b/raw-migrated-files/cloud/cloud-enterprise/ece-working-with-elasticsearch.md deleted file mode 100644 index 00b5dcb79..000000000 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-working-with-elasticsearch.md +++ /dev/null @@ -1,134 +0,0 @@ -# Manage data from the command line [ece-working-with-elasticsearch] - -Learn how to index, update, retrieve, search, and delete documents in an {{es}} cluster from the command line. - -::::{tip} -If you are looking for a user interface for {{es}} and your data, head on over to [Kibana](../../../deploy-manage/deploy/cloud-enterprise/create-deployment.md)! Not only are there amazing visualization and index management tools, Kibana includes realistic sample data sets to play with so that you can get to know what you *could* do with your data. -:::: - - - -## Before you begin [ece_before_you_begin_12] - -On the **Overview** page for your new cluster in the Cloud UI, copy the {{es}} endpoint URL under **Endpoints**. - -These examples use the `elastic` user. If you didn’t copy down the password for the `elastic` user, you can [reset the password](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). - -To use these examples, you also need to have the [curl](http://curl.haxx.se/) command installed. - - -## Indexing [ece_indexing] - -To index a document into {{es}}, `POST` your document: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc -XPOST -H 'Content-Type: application/json' -d '{ - "title": "One", "tags": ["ruby"] -}' -``` - -To show that the operation worked, {{es}} returns a JSON response that looks like `{"_index":"my_index","_type":"_doc","_id":"0KNPhW4BnhCSymaq_3SI","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":0,"_primary_term":1}`. - -In this example, the index `my_index` is created dynamically when the first document is inserted into it. All documents in {{es}} have a `type` and an `id`, which is echoed as `"_type":"_doc"` and `_id":"0KNPhW4BnhCSymaq_3SI` in the JSON response. If no ID is specified during indexing, a random `id` is generated. - - -### Bulk indexing [ece_bulk_indexing] - -To achieve the best possible performance, use the bulk API. - -To index some additional documents with the bulk API: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/_bulk -XPOST -H 'Content-Type: application/json' -d ' -{"index": {}} -{"title": "Two", "tags": ["ruby", "python"] } -{"index": {}} -{"title": "Three", "tags": ["java"] } -{"index": {}} -{"title": "Four", "tags": ["ruby", "php"] } -' -``` - -Elasticsearch returns a JSON response similar to this one: - -```json -{"took":694,"errors":false,"items":[{"index":{"_index":"my_index","_type":"_doc","_id":"0aNqhW4BnhCSymaqFHQn","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}},{"index":{"_index":"my_index","_type":"_doc","_id":"0qNqhW4BnhCSymaqFHQn","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1,"status":201}},{"index":{"_index":"my_index","_type":"_doc","_id":"06NqhW4BnhCSymaqFHQn","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":2,"_primary_term":1,"status":201}}]} -``` - - -## Updating [ece_updating] - -To update an existing document in {{es}}, `POST` the updated document to `http://ELASTICSEARCH_URL/my_index/_doc/ID`, where the ID is the `_id` of the document. - -For example, to update the last document indexed from the previous example with `"_id":"06NqhW4BnhCSymaqFHQn"`: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/06NqhW4BnhCSymaqFHQn -XPOST -H 'Content-Type: application/json' -d '{ - "title": "Four updated", "tags": ["ruby", "php", "python"] -}' -``` - -The JSON response shows that the version counter for the document got incremented to `_version":2` to reflect the update. - - -## Retrieving documents [ece_retrieving_documents] - -To take a look at a specific document you indexed, here the last document we updated with the ID `0KNPhW4BnhCSymaq_3SI`: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/06NqhW4BnhCSymaqFHQn -``` - -This request didn’t include `GET`, as the method is implied if you don’t specify anything else. If the document you are looking for exists, {{es}} returns `found":true` along with the document as part of the JSON response. Otherwise, the JSON response contains `"found":false`. - - -## Searching [ece_searching] - -You issue search requests for documents with one of these {{es}} endpoints: - -```bash -https://ELASTICSEARCH_URL/_search -https://ELASTICSEARCH_URL/INDEX_NAME/_search -``` - -Either a `GET` or a `POST` request with some URI search parameters works, or omit the method to default to `GET` request: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/_search?q=title:T* -``` - -For an explanation of the allowed parameters, check [URI Search](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search). - -To make {{es}} return a more human readable JSON response, add `?pretty=true` to the request: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/_search?pretty=true -H 'Content-Type: application/json' -d '{ - "query": { - "query_string": {"query": "*"} - } -}' -``` - -For performance reasons, `?pretty=true` is not recommended in production. You can verify the performance difference yourself by checking the `took` field in the JSON response which tells you how long Elasticsearch took to evaluate the search in milliseconds. When we tested these examples ourselves, the difference was `"took" : 4` against `"took" : 18`, a substantial difference. - -For a full explanation of how the request body is structured, check [Elasticsearch Request Body documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html). You can also execute multiple queries in one request with the [Multi Search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-msearch). - - -## Deleting [ece_deleting] - -You delete documents from {{es}} by sending `DELETE` requests. - -To delete a single document by ID from an earlier example: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/06NqhW4BnhCSymaqFHQn -XDELETE -``` - -To delete a whole index, here `my_index`: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index -XDELETE -``` - -The JSON response returns `{"acknowledged":true}` to indicate that the index deletion was a success. - diff --git a/raw-migrated-files/cloud/cloud/ec-working-with-elasticsearch.md b/raw-migrated-files/cloud/cloud/ec-working-with-elasticsearch.md deleted file mode 100644 index 6859e0b99..000000000 --- a/raw-migrated-files/cloud/cloud/ec-working-with-elasticsearch.md +++ /dev/null @@ -1,134 +0,0 @@ -# Manage data from the command line [ec-working-with-elasticsearch] - -Learn how to index, update, retrieve, search, and delete documents in an {{es}} cluster from the command line. - -::::{tip} -If you are looking for a user interface for {{es}} and your data, head on over to [Kibana](../../../deploy-manage/deploy/elastic-cloud/access-kibana.md)! Not only are there amazing visualization and index management tools, Kibana includes realistic sample data sets to play with so that you can get to know what you *could* do with your data. -:::: - - - -## Before you begin [ec_before_you_begin_4] - -On the **Overview** page for your new cluster in the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body), copy the {{es}} endpoint URL under **Endpoints**. - -These examples use the `elastic` user. If you didn’t copy down the password for the `elastic` user, you can [reset the password](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). - -To use these examples, you also need to have the [curl](http://curl.haxx.se/) command installed. - - -## Indexing [ec_indexing] - -To index a document into {{es}}, `POST` your document: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc -XPOST -H 'Content-Type: application/json' -d '{ - "title": "One", "tags": ["ruby"] -}' -``` - -To show that the operation worked, {{es}} returns a JSON response that looks like `{"_index":"my_index","_type":"_doc","_id":"0KNPhW4BnhCSymaq_3SI","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":0,"_primary_term":1}`. - -In this example, the index `my_index` is created dynamically when the first document is inserted into it. All documents in {{es}} have a `type` and an `id`, which is echoed as `"_type":"_doc"` and `_id":"0KNPhW4BnhCSymaq_3SI` in the JSON response. If no ID is specified during indexing, a random `id` is generated. - - -### Bulk indexing [ec_bulk_indexing] - -To achieve the best possible performance, use the bulk API. - -To index some additional documents with the bulk API: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/_bulk -XPOST -H 'Content-Type: application/json' -d ' -{"index": {}} -{"title": "Two", "tags": ["ruby", "python"] } -{"index": {}} -{"title": "Three", "tags": ["java"] } -{"index": {}} -{"title": "Four", "tags": ["ruby", "php"] } -' -``` - -Elasticsearch returns a JSON response similar to this one: - -```json -{"took":694,"errors":false,"items":[{"index":{"_index":"my_index","_type":"_doc","_id":"0aNqhW4BnhCSymaqFHQn","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}},{"index":{"_index":"my_index","_type":"_doc","_id":"0qNqhW4BnhCSymaqFHQn","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1,"status":201}},{"index":{"_index":"my_index","_type":"_doc","_id":"06NqhW4BnhCSymaqFHQn","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":2,"_primary_term":1,"status":201}}]} -``` - - -## Updating [ec_updating] - -To update an existing document in {{es}}, `POST` the updated document to `http://ELASTICSEARCH_URL/my_index/_doc/ID`, where the ID is the `_id` of the document. - -For example, to update the last document indexed from the previous example with `"_id":"06NqhW4BnhCSymaqFHQn"`: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/06NqhW4BnhCSymaqFHQn -XPOST -H 'Content-Type: application/json' -d '{ - "title": "Four updated", "tags": ["ruby", "php", "python"] -}' -``` - -The JSON response shows that the version counter for the document got incremented to `_version":2` to reflect the update. - - -## Retrieving documents [ec_retrieving_documents] - -To take a look at a specific document you indexed, here the last document we updated with the ID `0KNPhW4BnhCSymaq_3SI`: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/06NqhW4BnhCSymaqFHQn -``` - -This request didn’t include `GET`, as the method is implied if you don’t specify anything else. If the document you are looking for exists, {{es}} returns `found":true` along with the document as part of the JSON response. Otherwise, the JSON response contains `"found":false`. - - -## Searching [ec_searching] - -You issue search requests for documents with one of these {{es}} endpoints: - -```bash -https://ELASTICSEARCH_URL/_search -https://ELASTICSEARCH_URL/INDEX_NAME/_search -``` - -Either a `GET` or a `POST` request with some URI search parameters works, or omit the method to default to `GET` request: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/_search?q=title:T* -``` - -For an explanation of the allowed parameters, check [URI Search](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search). - -To make {{es}} return a more human readable JSON response, add `?pretty=true` to the request: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/_search?pretty=true -H 'Content-Type: application/json' -d '{ - "query": { - "query_string": {"query": "*"} - } -}' -``` - -For performance reasons, `?pretty=true` is not recommended in production. You can verify the performance difference yourself by checking the `took` field in the JSON response which tells you how long Elasticsearch took to evaluate the search in milliseconds. When we tested these examples ourselves, the difference was `"took" : 4` against `"took" : 18`, a substantial difference. - -For a full explanation of how the request body is structured, check [Elasticsearch Request Body documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html). You can also execute multiple queries in one request with the [Multi Search API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-msearch). - - -## Deleting [ec_deleting] - -You delete documents from {{es}} by sending `DELETE` requests. - -To delete a single document by ID from an earlier example: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index/_doc/06NqhW4BnhCSymaqFHQn -XDELETE -``` - -To delete a whole index, here `my_index`: - -```bash -curl -u USER:PASSWORD https://ELASTICSEARCH_URL/my_index -XDELETE -``` - -The JSON response returns `{"acknowledged":true}` to indicate that the index deletion was a success. - diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/index-mgmt.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/index-mgmt.md deleted file mode 100644 index c25049bf9..000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/index-mgmt.md +++ /dev/null @@ -1,203 +0,0 @@ -# Index management in {{kib}} [index-mgmt] - -{{kib}}'s **Index Management** features are an easy, convenient way to manage your cluster’s indices, [data streams](../../../manage-data/data-store/data-streams.md), [index templates](../../../manage-data/data-store/templates.md), and [enrich policies](../../../manage-data/ingest/transform-enrich/data-enrichment.md). Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible. - -To use these features, go to **Stack Management** > **Index Management**. - - -## Required permissions [index-mgm-req-permissions] - -If you use {{es}} {{security-features}}, the following [security privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) are required: - -* The `monitor` cluster privilege to access {{kib}}'s **Index Management** features. -* The `view_index_metadata` and `manage` index privileges to view a data stream or index’s data. -* The `manage_index_templates` cluster privilege to manage index templates. - -To add these privileges, go to **Stack Management > Security > Roles** or use the [Create or update roles API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role). - - -## Manage indices [view-edit-indices] - -Investigate your indices and perform operations from the **Indices** view. - -:::{image} ../../../images/elasticsearch-reference-management_index_labels.png -:alt: Index Management UI -:class: screenshot -::: - -* To show details and perform operations such as close, forcemerge, and flush, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices). -* To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow), a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps), or [frozen](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze). -* To drill down into the index [mappings](../../../manage-data/data-store/mapping.md), [settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/index-settings/index.md#index-modules-settings), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index. - - :::{image} ../../../images/elasticsearch-reference-management_index_details.png - :alt: Index Management UI - :class: screenshot - ::: - - - -## Manage data streams [manage-data-streams] - -Investigate your data streams and address lifecycle management needs in the **Data Streams** view. - -The value in the **Indices** column indicates the number of backing indices. Click this number to drill down into details. - -A value in the data retention column indicates that the data stream is managed by a [data stream lifecycle policy](../../../manage-data/lifecycle/data-stream.md). This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by Elasticsearch at a later time. - -:::{image} ../../../images/elasticsearch-reference-management-data-stream-fields.png -:alt: Data stream details -:class: screenshot -::: - -* To view more information about a data stream, such as its generation or its current index lifecycle policy, click the stream’s name. From this view, you can navigate to **Discover** to further explore data within the data stream. -* [preview]To edit the data retention value, open the **Manage** menu, and then click **Edit data retention**. This action is only available if your data stream is not managed by an ILM policy. - - -## Manage index templates [manage-index-templates] - -Create, edit, clone, and delete your index templates in the **Index Templates** view. Changes made to an index template do not affect existing indices. - -:::{image} ../../../images/elasticsearch-reference-management-index-templates.png -:alt: Index templates -:class: screenshot -::: - - -### Try it: Create an index template [_try_it_create_an_index_template] - -In this tutorial, you’ll create an index template and use it to configure two new indices. - -**Step 1. Add a name and index pattern** - -1. In the **Index Templates** view, open the **Create template** wizard. - - :::{image} ../../../images/elasticsearch-reference-management_index_create_wizard.png - :alt: Create wizard - :class: screenshot - ::: - -2. In the **Name** field, enter `my-index-template`. -3. Set **Index pattern** to `my-index-*` so the template matches any index with that index pattern. -4. Leave **Data Stream**, **Priority**, **Version**, and **_meta field** blank or as-is. - -**Step 2. Add settings, mappings, and aliases** - -1. Add [component templates](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) to your index template. - - Component templates are pre-configured sets of mappings, index settings, and aliases you can reuse across multiple index templates. Badges indicate whether a component template contains mappings (**M**), index settings (**S**), aliases (**A**), or a combination of the three. - - Component templates are optional. For this tutorial, do not add any component templates. - - :::{image} ../../../images/elasticsearch-reference-management_index_component_template.png - :alt: Component templates page - :class: screenshot - ::: - -2. Define index settings. These are optional. For this tutorial, leave this section blank. -3. Define a mapping that contains an [object](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/object.md) field named `geo` with a child [`geo_point`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/mapping-reference/geo-point.md) field named `coordinates`: - - :::{image} ../../../images/elasticsearch-reference-management-index-templates-mappings.png - :alt: Mapped fields page - :class: screenshot - ::: - - Alternatively, you can click the **Load JSON** link and define the mapping as JSON: - - ```js - { - "properties": { - "geo": { - "properties": { - "coordinates": { - "type": "geo_point" - } - } - } - } - } - ``` - - You can create additional mapping configurations in the **Dynamic templates** and **Advanced options** tabs. For this tutorial, do not create any additional mappings. - -4. Define an alias named `my-index`: - - ```js - { - "my-index": {} - } - ``` - -5. On the review page, check the summary. If everything looks right, click **Create template**. - -**Step 3. Create new indices** - -You’re now ready to create new indices using your index template. - -1. Index the following documents to create two indices: `my-index-000001` and `my-index-000002`. - - ```console - POST /my-index-000001/_doc - { - "@timestamp": "2019-05-18T15:57:27.541Z", - "ip": "225.44.217.191", - "extension": "jpg", - "response": "200", - "geo": { - "coordinates": { - "lat": 38.53146222, - "lon": -121.7864906 - } - }, - "url": "https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/charles-fullerton.jpg" - } - - POST /my-index-000002/_doc - { - "@timestamp": "2019-05-20T03:44:20.844Z", - "ip": "198.247.165.49", - "extension": "php", - "response": "200", - "geo": { - "coordinates": { - "lat": 37.13189556, - "lon": -76.4929875 - } - }, - "memory": 241720, - "url": "https://theacademyofperformingartsandscience.org/people/type:astronauts/name:laurel-b-clark/profile" - } - ``` - -2. Use the [get index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get) to view the configurations for the new indices. The indices were configured using the index template you created earlier. - - ```console - GET /my-index-000001,my-index-000002 - ``` - - - -## Manage enrich policies [manage-enrich-policies] - -Use the **Enrich Policies** view to add data from your existing indices to incoming documents during ingest. An enrich policy contains: - -* The policy type that determines how the policy matches the enrich data to incoming documents -* The source indices that store enrich data as documents -* The fields from the source indices used to match incoming documents -* The enrich fields containing enrich data from the source indices that you want to add to incoming documents -* An optional [query](asciidocalypse://docs/elasticsearch/docs/reference/query-languages/query-dsl-match-all-query.md). - -:::{image} ../../../images/elasticsearch-reference-management-enrich-policies.png -:alt: Enrich policies -:class: screenshot -::: - -When creating an enrich policy, the UI walks you through the configuration setup and selecting the fields. Before you can use the policy with an enrich processor or {{esql}} query, you must execute the policy. - -When executed, an enrich policy uses enrich data from the policy’s source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents. - -Check out these examples: - -* [Example: Enrich your data based on geolocation](../../../manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md) -* [Example: Enrich your data based on exact values](../../../manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md) -* [Example: Enrich your data by matching a value to a range](../../../manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md) - diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/ingest-enriching-data.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/ingest-enriching-data.md deleted file mode 100644 index d1d5d8155..000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/ingest-enriching-data.md +++ /dev/null @@ -1,65 +0,0 @@ -# Enrich your data [ingest-enriching-data] - -You can use the [enrich processor](asciidocalypse://docs/elasticsearch/docs/reference/ingestion-tools/enrich-processor/enrich-processor.md) to add data from your existing indices to incoming documents during ingest. - -For example, you can use the enrich processor to: - -* Identify web services or vendors based on known IP addresses -* Add product information to retail orders based on product IDs -* Supplement contact information based on an email address -* Add postal codes based on user coordinates - - -## How the enrich processor works [how-enrich-works] - -Most processors are self-contained and only change *existing* data in incoming documents. - -:::{image} ../../../images/elasticsearch-reference-ingest-process.svg -:alt: ingest process -::: - -The enrich processor adds *new* data to incoming documents and requires a few special components: - -:::{image} ../../../images/elasticsearch-reference-enrich-process.svg -:alt: enrich process -::: - -$$$enrich-policy$$$ - -enrich policy -: A set of configuration options used to add the right enrich data to the right incoming documents. - -An enrich policy contains: - -* A list of one or more *source indices* which store enrich data as documents -* The *policy type* which determines how the processor matches the enrich data to incoming documents -* A *match field* from the source indices used to match incoming documents -* *Enrich fields* containing enrich data from the source indices you want to add to incoming documents - -Before it can be used with an enrich processor, an enrich policy must be [executed](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-enrich-execute-policy). When executed, an enrich policy uses enrich data from the policy’s source indices to create a streamlined system index called the *enrich index*. The processor uses this index to match and enrich incoming documents. - - -$$$source-index$$$ - -source index -: An index which stores enrich data you’d like to add to incoming documents. You can create and manage these indices just like a regular {{es}} index. You can use multiple source indices in an enrich policy. You also can use the same source index in multiple enrich policies. - -$$$enrich-index$$$ - -enrich index -: A special system index tied to a specific enrich policy. - -Directly matching incoming documents to documents in source indices could be slow and resource intensive. To speed things up, the enrich processor uses an enrich index. - -Enrich indices contain enrich data from source indices but have a few special properties to help streamline them: - -* They are system indices, meaning they’re managed internally by {{es}} and only intended for use with enrich processors and the {{esql}} `ENRICH` command. -* They always begin with `.enrich-*`. -* They are read-only, meaning you can’t directly change them. -* They are [force merged](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge) for fast retrieval. - - - - - - diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index 4eb07d427..9492be74a 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -78,7 +78,6 @@ toc: - file: cloud/cloud-enterprise/ece-troubleshooting.md - file: cloud/cloud-enterprise/ece-upgrade-deployment.md - file: cloud/cloud-enterprise/ece-upgrade.md - - file: cloud/cloud-enterprise/ece-working-with-elasticsearch.md - file: cloud/cloud-enterprise/editing-user-settings.md - file: cloud/cloud-enterprise/Elastic-Cloud-Enterprise-overview.md - file: cloud/cloud-heroku/index.md @@ -188,7 +187,6 @@ toc: - file: cloud/cloud/ec-traffic-filtering-vnet.md - file: cloud/cloud/ec-traffic-filtering-vpc.md - file: cloud/cloud/ec-upgrade-deployment.md - - file: cloud/cloud/ec-working-with-elasticsearch.md - file: docs-content/serverless/index.md children: - file: docs-content/serverless/_cloud_native_vulnerability_management_dashboard.md @@ -377,10 +375,8 @@ toc: - file: elasticsearch/elasticsearch-reference/file-realm.md - file: elasticsearch/elasticsearch-reference/fips-140-compliance.md - file: elasticsearch/elasticsearch-reference/how-monitoring-works.md - - file: elasticsearch/elasticsearch-reference/index-mgmt.md - file: elasticsearch/elasticsearch-reference/index-modules-allocation.md - file: elasticsearch/elasticsearch-reference/index-modules-mapper.md - - file: elasticsearch/elasticsearch-reference/ingest-enriching-data.md - file: elasticsearch/elasticsearch-reference/install-elasticsearch.md - file: elasticsearch/elasticsearch-reference/ip-filtering.md - file: elasticsearch/elasticsearch-reference/jwt-auth-realm.md