From b90f4a4a447324b4018212f67d229401115a165a Mon Sep 17 00:00:00 2001 From: wajihaparvez Date: Tue, 11 Feb 2025 16:06:50 -0500 Subject: [PATCH 1/5] [Docs] Refine Mapping --- manage-data/data-store/mapping.md | 75 ++++++++++++++++++- .../data-store/mapping/dynamic-mapping.md | 4 +- 2 files changed, 74 insertions(+), 5 deletions(-) diff --git a/manage-data/data-store/mapping.md b/manage-data/data-store/mapping.md index 3e649f92c..fad517e87 100644 --- a/manage-data/data-store/mapping.md +++ b/manage-data/data-store/mapping.md @@ -14,8 +14,8 @@ mapped_urls: % Use migrated content from existing pages that map to this page: -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/mapping.md -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-modules-mapper.md +% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/mapping.md +% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-modules-mapper.md % Notes: redirect only % Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): @@ -28,4 +28,73 @@ $$$mapping-manage-update$$$ $$$mapping-dynamic$$$ -$$$mapping-explicit$$$ \ No newline at end of file +$$$mapping-explicit$$$ + +Mapping is the process of defining how a document and the fields it contains are stored and indexed. + +Each document is a collection of fields, which each have their own [data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). When mapping your data, you create a mapping definition, which contains a list of fields that are pertinent to the document. A mapping definition also includes [metadata fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-fields.html), like the `_source` field, which customize how a document’s associated metadata is handled. + +Depending on where you are in your data journey, use **dynamic mapping** and **explicit mapping** to define your data. For example, you can explicitly map fields where you don’t want to use the defaults, or to gain greater control over which fields are created. Then you can allow {{es}} to dynamically map other fields. + +::::{note} +Before 7.0.0, the mapping definition included a type name. {{es}} 7.0.0 and later no longer accept a default mapping. [Removal of mapping types](../../../manage-data/data-store/mapping/removal-of-mapping-types.md) provides more information. +:::: + +## Dynamic mapping [mapping-dynamic] + +When you use [dynamic mapping](../../../manage-data/data-store/mapping/dynamic-mapping.md), {{es}} automatically attempts to detect the data type of fields in your documents. This allows you to get started quickly by just adding data to an index. If you index additional documents with new fields, {{es}} will add these fields automatically. You can add fields to the top-level mapping, and to inner [`object`](https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html) and [`nested`](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html) fields. + +Use [dynamic templates](../../../manage-data/data-store/mapping/dynamic-templates.md) to define custom mappings that are applied to dynamically added fields based on the matching condition. + +## Explicit mapping [mapping-explicit] + +Use [explicit mapping](../../../manage-data/data-store/mapping/explicit-mapping.md) to define exactly how data types are mapped to fields, customized to your specific use case. + +Defining your own mappings enables you to: + +* Define which string fields should be treated as full-text fields. +* Define which fields contain numbers, dates, or geolocations. +* Use data types that cannot be automatically detected (such as `geo_point` and `geo_shape`.) +* Choose date value [formats](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html), including custom date formats. +* Create custom rules to control the mapping for [dynamically added fields](../../../manage-data/data-store/mapping/dynamic-mapping.md). +* Optimize fields for partial matching. +* Perform language-specific text analysis. + +::::{tip} +It’s often useful to index the same field in different ways for different purposes. For example, you might want to index a string field as both a text field for full-text search and as a keyword field for sorting or aggregating your data. Or, you might choose to use more than one language analyzer to process the contents of a string field that contains user input. + +:::: + +## Runtime fields [runtime-fields] + +Use [runtime fields](../../../manage-data/data-store/mapping/runtime-fields.md) to make schema changes without reindexing. You can use runtime fields in conjunction with indexed fields to balance resource usage and performance. Your index will be smaller, but with slower search performance. + +::::{admonition} Experiment with mapping options +[Define runtime fields in a search request](../../../manage-data/data-store/mapping/define-runtime-fields-in-search-request.md) to experiment with different mapping options, and also fix mistakes in your index mapping values by overriding values in the mapping during the search request. +:::: + +## Manage and update mappings [mapping-manage-update] + +Explicit mappings should be defined at index creation for fields you know in advance. You can still add new fields to mappings at any time, as your data evolves. + +Use the [Update mapping API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html) to update an existing mapping. + +In most cases, you can’t change mappings for fields that are already mapped. These changes require [reindexing](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html). + +However, you can update mappings under certain conditions: + +* You can add new fields to an existing mapping at any time, dynamically or explicitly. +* You can add new [multi-fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html) for existing fields. + + * Documents indexed before the mapping update will not have values for the new multi-fields until they are updated or reindexed. Documents indexed after the mapping change will automatically have values for the new multi-fields. + +* Some [mapping parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-params.html) can be updated for existing fields of certain [data types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). + + +## Prevent mapping explosions [mapping-limit-settings] + +Defining too many fields in an index can lead to a mapping explosion, which can cause out of memory errors and difficult situations from which to recover. + +Consider a situation where every new document inserted introduces new fields, such as with [dynamic mapping](../../../manage-data/data-store/mapping/dynamic-mapping.md). Each new field is added to the index mapping, which can become a problem as the mapping grows. + +Use the [mapping limit settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html) to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion. \ No newline at end of file diff --git a/manage-data/data-store/mapping/dynamic-mapping.md b/manage-data/data-store/mapping/dynamic-mapping.md index 6324ae850..af3c803bf 100644 --- a/manage-data/data-store/mapping/dynamic-mapping.md +++ b/manage-data/data-store/mapping/dynamic-mapping.md @@ -8,14 +8,14 @@ mapped_pages: One of the most important features of {{es}} is that it tries to get out of your way and let you start exploring your data as quickly as possible. To index a document, you don’t have to first create an index, define a mapping type, and define your fields — you can just index a document and the index, type, and fields will display automatically: ```console -PUT data/_doc/1 <1> +PUT data/_doc/1 <1> { "count": 5 } ``` 1. Creates the `data` index, the `_doc` mapping type, and a field called `count` with data type `long`. -The automatic detection and addition of new fields is called *dynamic mapping*. The dynamic mapping rules can be customized to suit your purposes with: +The automatic detection and addition of new fields is called **dynamic mapping**. The dynamic mapping rules can be customized to suit your purposes with: [Dynamic field mappings](dynamic-field-mapping.md) : The rules governing dynamic field detection. From e9d7ce2326ecfcfb51f1d221e2f007b1cb0b871e Mon Sep 17 00:00:00 2001 From: wajihaparvez Date: Tue, 11 Feb 2025 16:11:52 -0500 Subject: [PATCH 2/5] [Docs] Deleting raw migrated file --- .../data-store/mapping/dynamic-templates.md | 2 +- .../elasticsearch-reference/mapping.md | 76 ------------------- raw-migrated-files/toc.yml | 1 - 3 files changed, 1 insertion(+), 78 deletions(-) delete mode 100644 raw-migrated-files/elasticsearch/elasticsearch-reference/mapping.md diff --git a/manage-data/data-store/mapping/dynamic-templates.md b/manage-data/data-store/mapping/dynamic-templates.md index a76e4490a..b2cc660be 100644 --- a/manage-data/data-store/mapping/dynamic-templates.md +++ b/manage-data/data-store/mapping/dynamic-templates.md @@ -5,7 +5,7 @@ mapped_pages: # Dynamic templates [dynamic-templates] -Dynamic templates allow you greater control of how {{es}} maps your data beyond the default [dynamic field mapping rules](dynamic-field-mapping.md). You enable dynamic mapping by setting the dynamic parameter to `true` or `runtime`. You can then use dynamic templates to define custom mappings that can be applied to dynamically added fields based on the matching condition: +Dynamic templates allow you greater control over how {{es}} maps your data beyond the default [dynamic field mapping rules](dynamic-field-mapping.md). You enable dynamic mapping by setting the dynamic parameter to `true` or `runtime`. You can then use dynamic templates to define custom mappings that can be applied to dynamically added fields based on the matching condition: * [`match_mapping_type` and `unmatch_mapping_type`](#match-mapping-type) operate on the data type that {{es}} detects * [`match` and `unmatch`](#match-unmatch) use a pattern to match on the field name diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/mapping.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/mapping.md deleted file mode 100644 index 9841fbe1c..000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/mapping.md +++ /dev/null @@ -1,76 +0,0 @@ -# Mapping [mapping] - -Mapping is the process of defining how a document, and the fields it contains, are stored and indexed. - -Each document is a collection of fields, which each have their own [data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). When mapping your data, you create a mapping definition, which contains a list of fields that are pertinent to the document. A mapping definition also includes [metadata fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-fields.html), like the `_source` field, which customize how a document’s associated metadata is handled. - -Use *dynamic mapping* and *explicit mapping* to define your data. Each method provides different benefits based on where you are in your data journey. For example, explicitly map fields where you don’t want to use the defaults, or to gain greater control over which fields are created. You can then allow {{es}} to add other fields dynamically. - -::::{note} -Before 7.0.0, the mapping definition included a type name. {{es}} 7.0.0 and later no longer accept a *default* mapping. See [*Removal of mapping types*](../../../manage-data/data-store/mapping/removal-of-mapping-types.md). -:::: - - -::::{admonition} Experiment with mapping options -[Define runtime fields in a search request](../../../manage-data/data-store/mapping/define-runtime-fields-in-search-request.md) to experiment with different mapping options, and also fix mistakes in your index mapping values by overriding values in the mapping during the search request. - -:::: - - - -## Dynamic mapping [mapping-dynamic] - -When you use [dynamic mapping](../../../manage-data/data-store/mapping/dynamic-field-mapping.md), {{es}} automatically attempts to detect the data type of fields in your documents. This allows you to get started quickly by just adding data to an index. If you index additional documents with new fields, {{es}} will add these fields automatically. You can add fields to the top-level mapping, and to inner [`object`](https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html) and [`nested`](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html) fields. - -Use [dynamic templates](../../../manage-data/data-store/mapping/dynamic-templates.md) to define custom mappings that are applied to dynamically added fields based on the matching condition. - - -## Explicit mapping [mapping-explicit] - -Use [explicit mapping](../../../manage-data/data-store/mapping/explicit-mapping.md) to define exactly how data types are mapped to fields, customized to your specific use case. - -Defining your own mappings enables you to: - -* Define which string fields should be treated as full-text fields. -* Define which fields contain numbers, dates, or geolocations. -* Use data types that cannot be automatically detected (such as `geo_point` and `geo_shape`.) -* Choose date value [formats](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html), including custom date formats. -* Create custom rules to control the mapping for [dynamically added fields](../../../manage-data/data-store/mapping/dynamic-mapping.md). -* Optimize fields for partial matching. -* Perform language-specific text analysis. - -::::{tip} -It’s often useful to index the same field in different ways for different purposes. For example, you might want to index a string field as both a text field for full-text search and as a keyword field for sorting or aggregating your data. Or, you might choose to use more than one language analyzer to process the contents of a string field that contains user input. - -:::: - - -Use [runtime fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime-mapping-fields.html) to make schema changes without reindexing. You can use runtime fields in conjunction with indexed fields to balance resource usage and performance. Your index will be smaller, but with slower search performance. - - -## Managing and updating mappings [mapping-manage-update] - -Explicit mappings should be defined at index creation for fields you know in advance. You can still add *new fields* to mappings at any time, as your data evolves. - -Use the [Update mapping API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html) to update an existing mapping. - -In most cases, you can’t change mappings for fields that are already mapped. These changes require [reindexing](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html). - -However, you can *update* mappings under certain conditions: - -* You can add new fields to an existing mapping at any time, explicitly or dynamically. -* You can add new [multi-fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html) for existing fields. - - * Documents indexed before the mapping update will not have values for the new multi-fields until they are updated or reindexed. Documents indexed after the mapping change will automatically have values for the new multi-fields. - -* Some [mapping parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-params.html) can be updated for existing fields of certain [data types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). - - -## Prevent mapping explosions [mapping-limit-settings] - -Defining too many fields in an index can lead to a mapping explosion, which can cause out of memory errors and difficult situations to recover from. - -Consider a situation where every new document inserted introduces new fields, such as with [dynamic mapping](../../../manage-data/data-store/mapping/dynamic-mapping.md). Each new field is added to the index mapping, which can become a problem as the mapping grows. - -Use the [mapping limit settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html) to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion. - diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index 43797a87e..67d1a82da 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -591,7 +591,6 @@ toc: - file: elasticsearch/elasticsearch-reference/kerberos-realm.md - file: elasticsearch/elasticsearch-reference/ldap-realm.md - file: elasticsearch/elasticsearch-reference/mapping-roles.md - - file: elasticsearch/elasticsearch-reference/mapping.md - file: elasticsearch/elasticsearch-reference/monitor-elasticsearch-cluster.md - file: elasticsearch/elasticsearch-reference/monitoring-overview.md - file: elasticsearch/elasticsearch-reference/monitoring-production.md From c6ef231fed9787920616a2547f5e263d54f4fa22 Mon Sep 17 00:00:00 2001 From: wajihaparvez Date: Tue, 11 Feb 2025 16:13:26 -0500 Subject: [PATCH 3/5] [Docs] Formatting --- manage-data/data-store/mapping/dynamic-mapping.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manage-data/data-store/mapping/dynamic-mapping.md b/manage-data/data-store/mapping/dynamic-mapping.md index af3c803bf..143fa4c91 100644 --- a/manage-data/data-store/mapping/dynamic-mapping.md +++ b/manage-data/data-store/mapping/dynamic-mapping.md @@ -15,7 +15,7 @@ PUT data/_doc/1 <1> 1. Creates the `data` index, the `_doc` mapping type, and a field called `count` with data type `long`. -The automatic detection and addition of new fields is called **dynamic mapping**. The dynamic mapping rules can be customized to suit your purposes with: +The automatic detection and addition of new fields is called *dynamic mapping*. The dynamic mapping rules can be customized to suit your purposes with: [Dynamic field mappings](dynamic-field-mapping.md) : The rules governing dynamic field detection. From 91aa8fc30bcb151f4a3c0af95ae7b983c62b222e Mon Sep 17 00:00:00 2001 From: wajihaparvez Date: Tue, 11 Feb 2025 17:01:13 -0500 Subject: [PATCH 4/5] [Docs] Fix links --- manage-data/data-store/mapping.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/manage-data/data-store/mapping.md b/manage-data/data-store/mapping.md index fad517e87..6da8a80c0 100644 --- a/manage-data/data-store/mapping.md +++ b/manage-data/data-store/mapping.md @@ -37,18 +37,18 @@ Each document is a collection of fields, which each have their own [data type](h Depending on where you are in your data journey, use **dynamic mapping** and **explicit mapping** to define your data. For example, you can explicitly map fields where you don’t want to use the defaults, or to gain greater control over which fields are created. Then you can allow {{es}} to dynamically map other fields. ::::{note} -Before 7.0.0, the mapping definition included a type name. {{es}} 7.0.0 and later no longer accept a default mapping. [Removal of mapping types](../../../manage-data/data-store/mapping/removal-of-mapping-types.md) provides more information. +Before 7.0.0, the mapping definition included a type name. {{es}} 7.0.0 and later no longer accept a default mapping. [Removal of mapping types](/manage-data/data-store/mapping/removal-of-mapping-types.md) provides more information. :::: ## Dynamic mapping [mapping-dynamic] -When you use [dynamic mapping](../../../manage-data/data-store/mapping/dynamic-mapping.md), {{es}} automatically attempts to detect the data type of fields in your documents. This allows you to get started quickly by just adding data to an index. If you index additional documents with new fields, {{es}} will add these fields automatically. You can add fields to the top-level mapping, and to inner [`object`](https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html) and [`nested`](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html) fields. +When you use [dynamic mapping](/manage-data/data-store/mapping/dynamic-mapping.md), {{es}} automatically attempts to detect the data type of fields in your documents. This allows you to get started quickly by just adding data to an index. If you index additional documents with new fields, {{es}} will add these fields automatically. You can add fields to the top-level mapping, and to inner [`object`](https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html) and [`nested`](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html) fields. -Use [dynamic templates](../../../manage-data/data-store/mapping/dynamic-templates.md) to define custom mappings that are applied to dynamically added fields based on the matching condition. +Use [dynamic templates](/manage-data/data-store/mapping/dynamic-templates.md) to define custom mappings that are applied to dynamically added fields based on the matching condition. ## Explicit mapping [mapping-explicit] -Use [explicit mapping](../../../manage-data/data-store/mapping/explicit-mapping.md) to define exactly how data types are mapped to fields, customized to your specific use case. +Use [explicit mapping](/manage-data/data-store/mapping/explicit-mapping.md) to define exactly how data types are mapped to fields, customized to your specific use case. Defining your own mappings enables you to: @@ -56,7 +56,7 @@ Defining your own mappings enables you to: * Define which fields contain numbers, dates, or geolocations. * Use data types that cannot be automatically detected (such as `geo_point` and `geo_shape`.) * Choose date value [formats](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html), including custom date formats. -* Create custom rules to control the mapping for [dynamically added fields](../../../manage-data/data-store/mapping/dynamic-mapping.md). +* Create custom rules to control the mapping for [dynamically added fields](/manage-data/data-store/mapping/dynamic-mapping.md). * Optimize fields for partial matching. * Perform language-specific text analysis. @@ -67,10 +67,10 @@ It’s often useful to index the same field in different ways for different purp ## Runtime fields [runtime-fields] -Use [runtime fields](../../../manage-data/data-store/mapping/runtime-fields.md) to make schema changes without reindexing. You can use runtime fields in conjunction with indexed fields to balance resource usage and performance. Your index will be smaller, but with slower search performance. +Use [runtime fields](/manage-data/data-store/mapping/runtime-fields.md) to make schema changes without reindexing. You can use runtime fields in conjunction with indexed fields to balance resource usage and performance. Your index will be smaller, but with slower search performance. ::::{admonition} Experiment with mapping options -[Define runtime fields in a search request](../../../manage-data/data-store/mapping/define-runtime-fields-in-search-request.md) to experiment with different mapping options, and also fix mistakes in your index mapping values by overriding values in the mapping during the search request. +[Define runtime fields in a search request](/manage-data/data-store/mapping/define-runtime-fields-in-search-request.md) to experiment with different mapping options, and also fix mistakes in your index mapping values by overriding values in the mapping during the search request. :::: ## Manage and update mappings [mapping-manage-update] @@ -95,6 +95,6 @@ However, you can update mappings under certain conditions: Defining too many fields in an index can lead to a mapping explosion, which can cause out of memory errors and difficult situations from which to recover. -Consider a situation where every new document inserted introduces new fields, such as with [dynamic mapping](../../../manage-data/data-store/mapping/dynamic-mapping.md). Each new field is added to the index mapping, which can become a problem as the mapping grows. +Consider a situation where every new document inserted introduces new fields, such as with [dynamic mapping](/manage-data/data-store/mapping/dynamic-mapping.md). Each new field is added to the index mapping, which can become a problem as the mapping grows. Use the [mapping limit settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html) to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion. \ No newline at end of file From a874cd45d2c6ffffcc34f555163d9307827c5091 Mon Sep 17 00:00:00 2001 From: wajihaparvez Date: Wed, 12 Feb 2025 15:21:03 -0500 Subject: [PATCH 5/5] [Docs] Fix code block --- manage-data/data-store/mapping/dynamic-mapping.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/manage-data/data-store/mapping/dynamic-mapping.md b/manage-data/data-store/mapping/dynamic-mapping.md index 143fa4c91..e257e328e 100644 --- a/manage-data/data-store/mapping/dynamic-mapping.md +++ b/manage-data/data-store/mapping/dynamic-mapping.md @@ -7,14 +7,12 @@ mapped_pages: One of the most important features of {{es}} is that it tries to get out of your way and let you start exploring your data as quickly as possible. To index a document, you don’t have to first create an index, define a mapping type, and define your fields — you can just index a document and the index, type, and fields will display automatically: -```console -PUT data/_doc/1 <1> +```json +PUT data/_doc/1 <1> { "count": 5 } ``` - 1. Creates the `data` index, the `_doc` mapping type, and a field called `count` with data type `long`. - The automatic detection and addition of new fields is called *dynamic mapping*. The dynamic mapping rules can be customized to suit your purposes with: [Dynamic field mappings](dynamic-field-mapping.md)