40_One_language_per_doc.asciidoc

One Language per Document

A single predominant language per document requires a relatively simple setup. Documents from different languages can be stored in separate indices — blogs-en, blogs-fr, and so forth — that use the same fields for each index, just with different analyzers:

PUT /blogs-en
{
  "mappings": {
    "post": {
      "properties": {
        "title": {
          "type": "text", (1)
          "fields": {
            "stemmed": {
              "type": "string",
              "analyzer": "english" (2)
            }
          }
        }
      }
    }
  }
}

PUT /blogs-fr
{
  "mappings": {
    "post": {
      "properties": {
        "title": {
          "type": "text", (1)
          "fields": {
            "stemmed": {
              "type": "text",
              "analyzer": "french" (2)
            }
          }
        }
      }
    }
  }
}

1. Both blogs-en and blogs-fr have a type called post that contains the field title.

2. The title.stemmed subfield uses a language-specific analyzer.

This approach is clean and flexible. New languages are easy to add—​just create a new index—​and because each language is completely separate, we don’t suffer from the term-frequency and stemming problems described in [language-pitfalls].

The documents of a single language can be queried independently, or queries can target multiple languages by querying multiple indices. We can even specify a preference for particular languages with the indices_boost parameter:

PUT /blogs-en/post/1
{ "title": "That feeling of déjà vu" }

PUT /blogs-fr/post/1
{ "title": "Ce sentiment de déjà vu" }

GET /blogs-*/post/_search (1)
{
    "query": {
        "multi_match": {
            "query":   "deja vu",
            "fields":  [ "title", "title.stemmed" ], (2)
            "type":    "most_fields"
        }
    },
    "indices_boost": [ (3)
      { "blogs-en": 3 },
      { "blogs-fr": 2 }
    ]
}

1. This search is performed on any index beginning with blogs-.

2. The title.stemmed fields are queried using the analyzer specified in each index.

3. Perhaps the user’s accept-language headers showed a preference for English, and then French, so we boost results from each index accordingly. Any other languages will have a neutral boost of 1.

Foreign Words

Of course, these documents may contain words or sentences in other languages, and these words are unlikely to be stemmed correctly. With predominant-language documents, this is not usually a major problem. The user will often search for the exact words—​for instance, of a quotation from another language—​rather than for inflections of a word. Recall can be improved by using techniques explained in [token-normalization].

Perhaps some words like place names should be queryable in the predominant language and in the original language, such as Munich and München. These words are effectively synonyms, which we discuss in [synonyms].