Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] SQL/PPL tests intermittently fail with an Internal Server Error (500) #468

Closed
dblock opened this issue Aug 5, 2024 · 1 comment · Fixed by #470 or #486
Closed

[BUG] SQL/PPL tests intermittently fail with an Internal Server Error (500) #468

dblock opened this issue Aug 5, 2024 · 1 comment · Fixed by #470 or #486
Assignees
Labels
bug Something isn't working

Comments

@dblock
Copy link
Member

dblock commented Aug 5, 2024

What is the bug?

ERROR   ppl/query.yaml (/home/runner/work/opensearch-api-specification/opensearch-api-specification/tests/ppl/query.yaml)
    ERROR   CHAPTERS
        ERROR   Get PPL query
            PASSED  REQUEST BODY
            ERROR   RESPONSE STATUS (Expected status 200, but received 500: text/plain. Internal Server Error)
            SKIPPED RESPONSE PAYLOAD BODY
            SKIPPED RESPONSE PAYLOAD SCHEMA
            SKIPPED RESPONSE OUTPUT VALUES
2024-08-01T14:02:08.9247018Z [2024-08-01T14:02:05,858][INFO ][o.o.s.s.a.SQLSyntaxParser] [6080215090a3] New Engine Request Query: ( SELECT * FROM table )
2024-08-01T14:02:08.9247664Z [2024-08-01T14:02:05,983][WARN ][o.o.s.o.m.OpenSearchMemoryHealthy] [6080215090a3] Memory usage:1054384512 exceed limit:912680550
2024-08-01T14:02:08.9248242Z [2024-08-01T14:02:05,983][WARN ][o.o.s.o.m.OpenSearchMemoryHealthy] [6080215090a3] Fast failure the current request
2024-08-01T14:02:08.9249007Z [2024-08-01T14:02:05,985][ERROR][o.o.s.l.p.RestSqlAction  ] [6080215090a3] ef30b895-53e0-4444-8034-3480c19b181c Server side error during query execution
2024-08-01T14:02:08.9249390Z java.lang.IllegalStateException: resource is not enough to run the query, quit.
2024-08-01T14:02:08.9250329Z 	at org.opensearch.sql.opensearch.executor.protector.ResourceMonitorPlan.open(ResourceMonitorPlan.java:47) ~[opensearch-2.15.0.0.jar:?]
2024-08-01T14:02:08.9250707Z 	at java.base/java.util.Collections$SingletonList.forEach(Collections.java:5186) ~[?:?]
2024-08-01T14:02:08.9251234Z 	at org.opensearch.sql.planner.physical.PhysicalPlan.open(PhysicalPlan.java:29) ~[core-2.15.0.0.jar:?]
2024-08-01T14:02:08.9252023Z 	at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$0(OpenSearchExecutionEngine.java:49) [opensearch-2.15.0.0.jar:?]
2024-08-01T14:02:08.9252708Z 	at org.opensearch.sql.opensearch.client.OpenSearchNodeClient.schedule(OpenSearchNodeClient.java:177) [opensearch-2.15.0.0.jar:?]
2024-08-01T14:02:08.9253640Z 	at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.execute(OpenSearchExecutionEngine.java:43) [opensearch-2.15.0.0.jar:?]
2024-08-01T14:02:08.9254412Z 	at org.opensearch.sql.executor.QueryService.lambda$executePlan$1(QueryService.java:66) [core-2.15.0.0.jar:?]
2024-08-01T14:02:08.9254706Z 	at java.base/java.util.Optional.ifPresentOrElse(Optional.java:198) [?:?]
2024-08-01T14:02:08.9255209Z 	at org.opensearch.sql.executor.QueryService.executePlan(QueryService.java:63) [core-2.15.0.0.jar:?]
2024-08-01T14:02:08.9255688Z 	at org.opensearch.sql.executor.QueryService.execute(QueryService.java:41) [core-2.15.0.0.jar:?]
2024-08-01T14:02:08.9256179Z 	at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:63) [core-2.15.0.0.jar:?]
2024-08-01T14:02:08.9256928Z 	at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$submit$0(OpenSearchQueryManager.java:31) [opensearch-2.15.0.0.jar:?]
2024-08-01T14:02:08.9257734Z 	at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$1(OpenSearchQueryManager.java:45) [opensearch-2.15.0.0.jar:?]
2024-08-01T14:02:08.9258452Z 	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:882) [opensearch-2.15.0.jar:2.15.0]
2024-08-01T14:02:08.9258901Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
2024-08-01T14:02:08.9259319Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
2024-08-01T14:02:08.9259532Z 	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

How can one reproduce the bug?

What is the expected behavior?

Always succeed.

Do you have any additional context?

I attempted to fix this in #459 (comment) by increasing the value of the breaker.

@dblock
Copy link
Member Author

dblock commented Aug 8, 2024

This wasn't fixed by #470 :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
1 participant