Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Manage data] Refine Mapping section #408

Merged
merged 7 commits into from
Feb 12, 2025
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 72 additions & 3 deletions manage-data/data-store/mapping.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ mapped_urls:

% Use migrated content from existing pages that map to this page:

% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/mapping.md
% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-modules-mapper.md
% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/mapping.md
% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/index-modules-mapper.md
% Notes: redirect only

% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc):
Expand All @@ -28,4 +28,73 @@ $$$mapping-manage-update$$$

$$$mapping-dynamic$$$

$$$mapping-explicit$$$
$$$mapping-explicit$$$

Mapping is the process of defining how a document and the fields it contains are stored and indexed.

Each document is a collection of fields, which each have their own [data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). When mapping your data, you create a mapping definition, which contains a list of fields that are pertinent to the document. A mapping definition also includes [metadata fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-fields.html), like the `_source` field, which customize how a document’s associated metadata is handled.

Depending on where you are in your data journey, use **dynamic mapping** and **explicit mapping** to define your data. For example, you can explicitly map fields where you don’t want to use the defaults, or to gain greater control over which fields are created. Then you can allow {{es}} to dynamically map other fields.

::::{note}
Before 7.0.0, the mapping definition included a type name. {{es}} 7.0.0 and later no longer accept a default mapping. [Removal of mapping types](/manage-data/data-store/mapping/removal-of-mapping-types.md) provides more information.
::::

## Dynamic mapping [mapping-dynamic]

When you use [dynamic mapping](/manage-data/data-store/mapping/dynamic-mapping.md), {{es}} automatically attempts to detect the data type of fields in your documents. This allows you to get started quickly by just adding data to an index. If you index additional documents with new fields, {{es}} will add these fields automatically. You can add fields to the top-level mapping, and to inner [`object`](https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html) and [`nested`](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html) fields.

Use [dynamic templates](/manage-data/data-store/mapping/dynamic-templates.md) to define custom mappings that are applied to dynamically added fields based on the matching condition.

## Explicit mapping [mapping-explicit]

Use [explicit mapping](/manage-data/data-store/mapping/explicit-mapping.md) to define exactly how data types are mapped to fields, customized to your specific use case.

Defining your own mappings enables you to:

* Define which string fields should be treated as full-text fields.
* Define which fields contain numbers, dates, or geolocations.
* Use data types that cannot be automatically detected (such as `geo_point` and `geo_shape`.)
* Choose date value [formats](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html), including custom date formats.
* Create custom rules to control the mapping for [dynamically added fields](/manage-data/data-store/mapping/dynamic-mapping.md).
* Optimize fields for partial matching.
* Perform language-specific text analysis.

::::{tip}
It’s often useful to index the same field in different ways for different purposes. For example, you might want to index a string field as both a text field for full-text search and as a keyword field for sorting or aggregating your data. Or, you might choose to use more than one language analyzer to process the contents of a string field that contains user input.

::::

## Runtime fields [runtime-fields]

Use [runtime fields](/manage-data/data-store/mapping/runtime-fields.md) to make schema changes without reindexing. You can use runtime fields in conjunction with indexed fields to balance resource usage and performance. Your index will be smaller, but with slower search performance.

::::{admonition} Experiment with mapping options
[Define runtime fields in a search request](/manage-data/data-store/mapping/define-runtime-fields-in-search-request.md) to experiment with different mapping options, and also fix mistakes in your index mapping values by overriding values in the mapping during the search request.
::::

## Manage and update mappings [mapping-manage-update]

Explicit mappings should be defined at index creation for fields you know in advance. You can still add new fields to mappings at any time, as your data evolves.

Use the [Update mapping API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html) to update an existing mapping.

In most cases, you can’t change mappings for fields that are already mapped. These changes require [reindexing](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html).

However, you can update mappings under certain conditions:

* You can add new fields to an existing mapping at any time, dynamically or explicitly.
* You can add new [multi-fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html) for existing fields.

* Documents indexed before the mapping update will not have values for the new multi-fields until they are updated or reindexed. Documents indexed after the mapping change will automatically have values for the new multi-fields.

* Some [mapping parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-params.html) can be updated for existing fields of certain [data types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html).


## Prevent mapping explosions [mapping-limit-settings]

Defining too many fields in an index can lead to a mapping explosion, which can cause out of memory errors and difficult situations from which to recover.

Consider a situation where every new document inserted introduces new fields, such as with [dynamic mapping](/manage-data/data-store/mapping/dynamic-mapping.md). Each new field is added to the index mapping, which can become a problem as the mapping grows.

Use the [mapping limit settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html) to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion.
2 changes: 1 addition & 1 deletion manage-data/data-store/mapping/dynamic-mapping.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ mapped_pages:
One of the most important features of {{es}} is that it tries to get out of your way and let you start exploring your data as quickly as possible. To index a document, you don’t have to first create an index, define a mapping type, and define your fields — you can just index a document and the index, type, and fields will display automatically:

```console
PUT data/_doc/1 <1>
PUT data/_doc/1 <1>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that the "1" doesn't render properly in this code block and in the bullet after. It seems to be formatted the same as other pages where it's rendering like it's supposed to. I tried to troubleshoot this but I'm at a loss. Would appreciate any ideas

Copy link
Contributor

@kilfoyle kilfoyle Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wajihaparvez Good question! If you change "console" to "json" I think it'll work (I tried it locally and that did the trick). That is, it seems to prevent the Markdown build from interpreting the <1> as part of the code example.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @kilfoyle, that worked!

{ "count": 5 }
```

Expand Down
2 changes: 1 addition & 1 deletion manage-data/data-store/mapping/dynamic-templates.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ mapped_pages:

# Dynamic templates [dynamic-templates]

Dynamic templates allow you greater control of how {{es}} maps your data beyond the default [dynamic field mapping rules](dynamic-field-mapping.md). You enable dynamic mapping by setting the dynamic parameter to `true` or `runtime`. You can then use dynamic templates to define custom mappings that can be applied to dynamically added fields based on the matching condition:
Dynamic templates allow you greater control over how {{es}} maps your data beyond the default [dynamic field mapping rules](dynamic-field-mapping.md). You enable dynamic mapping by setting the dynamic parameter to `true` or `runtime`. You can then use dynamic templates to define custom mappings that can be applied to dynamically added fields based on the matching condition:

* [`match_mapping_type` and `unmatch_mapping_type`](#match-mapping-type) operate on the data type that {{es}} detects
* [`match` and `unmatch`](#match-unmatch) use a pattern to match on the field name
Expand Down

This file was deleted.

1 change: 0 additions & 1 deletion raw-migrated-files/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -586,7 +586,6 @@ toc:
- file: elasticsearch/elasticsearch-reference/kerberos-realm.md
- file: elasticsearch/elasticsearch-reference/ldap-realm.md
- file: elasticsearch/elasticsearch-reference/mapping-roles.md
- file: elasticsearch/elasticsearch-reference/mapping.md
- file: elasticsearch/elasticsearch-reference/monitor-elasticsearch-cluster.md
- file: elasticsearch/elasticsearch-reference/monitoring-overview.md
- file: elasticsearch/elasticsearch-reference/monitoring-production.md
Expand Down
Loading