Hibernate Search ORM ElasticSearch fails search for offset larger than index.max_result_window

[DaHoC]

Describe the bug

When using quarkus-hibernate-search-orm-elasticsearch (3.17.5) with ElasticSearch 8.17.0 and more than max_result_window (default 10000) indexed entities, the search with large offset > 10000 (searchSession.search(...).fetch(searchOffset, searchLimit);) of the search fails with following exception:

HSEARCH400007: Elasticsearch request failed: HSEARCH400090: Elasticsearch response indicates a failure.
Request: POST /<redacted>-read/_search with parameters {from=10000, size=20, track_total_hits=true}
Response: 400 'Bad Request' from 'http://localhost:9200' with body 
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [10020]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "<redacted>-000001",
        "node": "H5cQN2XWSfyISpvBqrFTHQ",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [10020]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
        }
      }
    ],
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [10020]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.",
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [10020]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
      }
    }
  },
  "status": 400
}

Additionally, I did not manage to employ the workaround of increasing the index.max_result_window setting using a custom setting via quarkus.hibernate-search-orm.elasticsearch.schema-management.settings-file with custom setting file content: { "max_result_window": 100000 }, yielding Invalid value. Expected '100000', actual is 'null'

Expected behavior

Hibernate-search should return indexed entities for larger pages when searching.

Actual behavior

Hibernate-Search exception as stated above when performing paginated search with a large offset

How to Reproduce?

Take the hibernate-search-orm-elasticsearch-quickstart project, add following function to LibraryResource:

    @GET
    @Path("author/searchpaginated")
    @Transactional
    public List<Author> searchAuthorsPaginated(@RestQuery String pattern,
                                               @RestQuery Optional<Integer> offset,
                                               @RestQuery Optional<Integer> size) {
        return searchSession.search(Author.class)
                .where(f -> pattern == null || pattern.trim().isEmpty() ? f.matchAll()
                        : f.simpleQueryString()
                        .fields("firstName", "lastName", "books.title").matching(pattern))
                .sort(f -> f.field("lastName_sort").then().field("firstName_sort"))
                .fetchHits(offset.orElse(0), size.orElse(20));
    }

and following test to LibraryResourceTest:

    @Test
    public void testLargeLibrary() throws Exception {
        // Add 10010 authors
        for (int i = 0; i <= 10_010; i++) {
            // Add an author
            RestAssured.given()
                    .contentType(ContentType.URLENC.withCharset(StandardCharsets.UTF_8))
                    .formParam("firstName", String.format("Chucky %d", i))
                    .formParam("lastName", "Wrong")
                    .put("/library/author/")
                    .then()
                    .statusCode(204);
        }

        // Search, hitting the max_result_window ES limit
        RestAssured.when().get("/library/author/searchpaginated?pattern=*&offset=10000&size=10").then()
                .statusCode(200);
    }

Run the test (it takes some time to perform the insertions via HTTP), and you will encounter the same exception as described:

2025-01-08 13:41:29,519 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /library/author/searchpaginated?pattern=*&offset=10000&size=10 failed, error id: 7875d8da-0fac-4f4d-80a9-321979388ab9-1: org.hibernate.search.util.common.SearchException: HSEARCH400007: Elasticsearch request failed: HSEARCH400090: Elasticsearch response indicates a failure.
Request: POST /author-read/_search with parameters {from=10000, size=10, track_total_hits=false}
Response: 400 'Bad Request' from 'http://localhost:32770' with body 
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [10010]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "author-000001",
        "node": "6frpvx0lRcO01DPUJtb9Lg",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [10010]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
        }
      }
    ],
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [10010]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.",
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [10010]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
      }
    }
  },
  "status": 400
}

	at org.hibernate.search.backend.elasticsearch.work.impl.AbstractNonBulkableWork.handleResult(AbstractNonBulkableWork.java:87)
	at org.hibernate.search.backend.elasticsearch.work.impl.AbstractNonBulkableWork.lambda$execute$3(AbstractNonBulkableWork.java:68)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:690)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2223)
	at org.hibernate.search.backend.elasticsearch.client.impl.ElasticsearchClientImpl$1.onFailure(ElasticsearchClientImpl.java:128)
	at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:688)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:413)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:397)
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:182)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Thread.java:1575)
Caused by: org.hibernate.search.util.common.SearchException: HSEARCH400090: Elasticsearch response indicates a failure.
	at org.hibernate.search.backend.elasticsearch.work.impl.ElasticsearchRequestSuccessAssessor.checkSuccess(ElasticsearchRequestSuccessAssessor.java:125)
	at org.hibernate.search.backend.elasticsearch.work.impl.ElasticsearchRequestSuccessAssessor.checkSuccess(ElasticsearchRequestSuccessAssessor.java:103)
	at org.hibernate.search.backend.elasticsearch.work.impl.AbstractNonBulkableWork.handleResult(AbstractNonBulkableWork.java:82)
	... 23 more

Output of uname -a or ver

Linux Penguin 6.11.0-13-generic #14-Ubuntu SMP PREEMPT_DYNAMIC Sat Nov 30 23:51:51 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

openjdk version "23.0.1" 2024-10-15 OpenJDK Runtime Environment (build 23.0.1+11-Ubuntu-1ubuntu124.10.1) OpenJDK 64-Bit Server VM (build 23.0.1+11-Ubuntu-1ubuntu124.10.1, mixed mode, sharing)

Quarkus version or git rev

3.17.5

Build tool (ie. output of mvnw --version or gradlew --version)

Maven home: /home/dahoc/.m2/wrapper/dists/apache-maven-3.9.9/3477a4f1 Java version: 23.0.1, vendor: Ubuntu, runtime: /usr/lib/jvm/java-23-openjdk-amd64 Default locale: de_DE, platform encoding: UTF-8 OS name: "linux", version: "6.11.0-13-generic", arch: "amd64", family: "unix"

Additional information

This might be related to #45164 in that Hibernate-Search is apparently using a discouraged or outdated API (also negatively impacting memory) instead of the recommended one as stated in the exception.

[quarkus-bot]

/cc @gsmet (elasticsearch,hibernate-search), @loicmathieu (elasticsearch), @marko-bekhta (elasticsearch,hibernate-search), @yrodiere (elasticsearch,hibernate-search)

[yrodiere]

Hello,

Thanks for reporting.

Expected behavior

Hibernate-search should return indexed entities for larger pages when searching.

I'm sorry but that's unrealistic expectations. Hibernate Search cannot do what Elasticsearch cannot. And Elasticsearch cannot handle larger pages when searching -- except with workarounds, see below.

Additionally, I did not manage to employ the workaround of increasing the index.max_result_window setting using a custom setting via quarkus.hibernate-search-orm.elasticsearch.schema-management.settings-file with custom setting file content: { "max_result_window": 100000 }, yielding Invalid value. Expected '100000', actual is 'null'

This on the other hand, is expected to work. I'd like to know more.

1. Did you re-create your indexes after adding the setting?
2. Can you please provide a reproducer?

---

Relatedly, a way to work around this limitation in Elasticsearch is to use Elasticsearch's "search_after" feature, which ignores all documents with a sort key below a given one, excluding them from the result window. It's not exposed yet in Hibernate Search (HSEARCH-2601), but we gladly accept PRs ;) and worst case it can possibly be implemented in your application with custom JSON.

[yrodiere]

By the way, to clarify the limitation: what matters is not the number of indexed entities. You can have millions of them. What matters is the number of hits of your query, and how far down that list of hits you want to go.

A query with 10,000,000 hits is fine, if you only inspect the 10,000 first hits.
A query with 10,001 hits is not fine, if you inspect the 10,001st hit, and didn't change Elasticsearch's max_result_window.

That's where search_after helps: it eliminates hits before the given sort key from the result window, allowing you to go further down the list of hits.

[yrodiere]

Oh and FYI, the best way (for you) to work around this limitation is to... not offer the feature to your users in the first place. Just propagate the limitation.
The reason for the limitation in the first place is that going too far down the result window will incur a significant performance hit -- Elasticsearch/Lucene basically has to go through all 10,000 hits in order to reach the 10,001st, so it gets slower and slower (though there are optimizations in some cases).
So you'll notice web applications where search results provide links to go to the 1st, 2nd, ... 10th, ... 100th page, and... that's it. They'll just mention something along the lines of "add more criteria to your query to access other results". Which seems reasonable in many cases, as I don't expect many people will go through hundreds of pages of results.

That won't work if your use case is automated processing of the whole index (or a large part of it), of course. In that case I'd suggest using scrolling if possible, combined with search_after or some custom additional predicates to "partition" your result set into multiple queries.

[DaHoC]

That's where search_after helps: it eliminates hits before the given sort key from the search window, allowing you to go further down the list of hits.

My hope was that either the scroll API or search_after would be used internally (maybe in the future) when I fetch a small dataset (10 entries) with a large offset (i.e. using .fetchHits(10000, 10)), as I do not know how to use them with quarkus-hibernate-search-orm-elasticsearch.

This on the other hand, is expected to work. I'd like to know more.

1. No, the application fails to start up with the mentioned exception
2. Minimal reproducer works just fine, issue was apparently due to an unclean volume/state of ElasticSearch in my app

Just propagate the limitation.

This is my current solution of choice as well, thank you!

Because this is not a bug but rather a limitation of the used endpoints, I will close this ticket.

[yrodiere]

This on the other hand, is expected to work. I'd like to know more.

1. No, the application fails to start up with the mentioned exception

2. Minimal reproducer works just fine, issue was apparently due to an unclean volume/state of ElasticSearch in my app

Thanks for the update!

My hope was that either the scroll API or search_after would be used internally (maybe in the future) when I fetch a small dataset (10 entries) with a large offset (i.e. using .fetchHits(10000, 10)), as I do not know how to use them with quarkus-hibernate-search-orm-elasticsearch.

I think the scroll API has the same limitation (result window limited to 10,000 hits) so I do not think that would help.

The search_after feature is a bit awkward to apply implicitly because it requires the query for the 10,001 element to take as an input the "sort key" for the 10,000th element -- and also, it only works in specific scenarios (sort by value, and sort keys must be unique).

Anyway, in either case, an implicit workaround for this limitation an Elasticserach would incur a significant performance hit. I would personally not implement that, or only as an opt-in feature -- if only to give you an excuse to propagate the limitation :)

Because this is not a bug but rather a limitation of the used endpoints, I will close this ticket.

Works for me, thanks.

Anyone reading this, you probably want https://hibernate.atlassian.net/browse/HSEARCH-2601 to be implemented -- and you can send a pull request!