When server is being asked lots of queries under heavy load a single search on top returns zero results

[smokingwheels]

start up settings.
JAVA_ARGS="-XX:+UseG1GC -XX:MaxGCPauseMillis=400 -XX:G1HeapRegionSize=64m -XX:MetaspaceSize=1024m -XX:MaxMetaspaceSize=2048m -Xss2048k $JAVA_ARGS";
I have 300 GB assigned.
My Yacy server is under a search load of 15 Hz.

`I 2024/08/30 19:56:09 NETWORK * SEARCH failed (solr), localpeer (Error executing query) java.io.IOException: Error executing query at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getResponseByParams(EmbeddedSolrConnector.java:376) at net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getResponseByParams(MirrorSolrConnector.java:272) at net.yacy.peers.Protocol.solrQuery(Protocol.java:1225) at net.yacy.peers.RemoteSearch$2.run(RemoteSearch.java:411) Caused by: org.apache.solr.common.SolrException: Error while processing facet fields: InterruptedException at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:936) at org.apache.solr.handler.component.FacetComponent.getFacetCounts(FacetComponent.java:337) at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:278) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:420) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:207) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2866) at net.yacy.cora.federate.solr.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:231) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:217) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:927) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:940) at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getResponseByParams(EmbeddedSolrConnector.java:369) ... 3 more Caused by: java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1100) at java.base/java.util.concurrent.Semaphore.acquire(Semaphore.java:318) at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:925) ... 13 more

E 2024/08/30 19:56:09 org.apache.solr.handler.RequestHandlerBase org.apache.solr.common.SolrException: Error while processing facet fields: InterruptedException org.apache.solr.common.SolrException: Error while processing facet fields: InterruptedException at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:936) at org.apache.solr.handler.component.FacetComponent.getFacetCounts(FacetComponent.java:337) at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:278) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:420) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:207) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2866) at net.yacy.cora.federate.solr.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:231) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:217) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:927) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:940) at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getResponseByParams(EmbeddedSolrConnector.java:369) at net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getResponseByParams(MirrorSolrConnector.java:272) at net.yacy.peers.Protocol.solrQuery(Protocol.java:1225) at net.yacy.peers.RemoteSearch$2.run(RemoteSearch.java:411) Caused by: java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1100) at java.base/java.util.concurrent.Semaphore.acquire(Semaphore.java:318) at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:925) ... 13 more

I 2024/08/30 19:56:09 NETWORK * SEARCH failed (solr), localpeer (Error executing query) java.io.IOException: Error executing query at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getResponseByParams(EmbeddedSolrConnector.java:376) at net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getResponseByParams(MirrorSolrConnector.java:272) at net.yacy.peers.Protocol.solrQuery(Protocol.java:1225) at net.yacy.peers.RemoteSearch$2.run(RemoteSearch.java:411) Caused by: org.apache.solr.common.SolrException: Error while processing facet fields: InterruptedException at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:936) at org.apache.solr.handler.component.FacetComponent.getFacetCounts(FacetComponent.java:337) at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:278) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:420) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:207) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2866) at net.yacy.cora.federate.solr.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:231) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:217) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:927) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:940) at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getResponseByParams(EmbeddedSolrConnector.java:369) ... 3 more Caused by: java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1100) at java.base/java.util.concurrent.Semaphore.acquire(Semaphore.java:318) at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:925) ... 13 more

I 2024/08/30 19:55:30 LOCAL_SEARCH * EXIT WORD SEARCH: olqw - local_rwi_available(0), local_rwi_stored(0), remote_rwi_available(0), remote_rwi_stored(0), remote_rwi_peerCount(0), local_solr_evicted(0), local_solr_stored(0), remote_solr_available(0), remote_solr_stored(0), remote_solr_peerCount(0), 1344 ms

I 2024/08/30 19:56:09 NETWORK * SEARCH failed (solr), localpeer (Error executing query) java.io.IOException: Error executing query at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getResponseByParams(EmbeddedSolrConnector.java:376) at net.yacy.cora.federate.solr.connector.MirrorSolrConnector.getResponseByParams(MirrorSolrConnector.java:272) at net.yacy.peers.Protocol.solrQuery(Protocol.java:1225) at net.yacy.peers.RemoteSearch$2.run(RemoteSearch.java:411) Caused by: org.apache.solr.common.SolrException: Error while processing facet fields: InterruptedException at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:936) at org.apache.solr.handler.component.FacetComponent.getFacetCounts(FacetComponent.java:337) at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:278) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:420) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:207) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2866) at net.yacy.cora.federate.solr.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:231) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:217) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:927) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:940) at net.yacy.cora.federate.solr.connector.EmbeddedSolrConnector.getResponseByParams(EmbeddedSolrConnector.java:369) ... 3 more Caused by: java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1100) at java.base/java.util.concurrent.Semaphore.acquire(Semaphore.java:318) at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:925) ... 13 more

I 2024/08/30 19:56:09 NETWORK * SEARCH failed (solr), localpeer (Error executing query) java.io.IOException: Error`

I stopped asking and yacy started responding to queries.

`I 2024/08/30 20:07:45 NETWORK * local search (solr): localpeer sent 10/11 references

I 2024/08/30 20:07:45 NETWORK * SEARCH (solr), returned 10 out of 11 documents and 5 facets [url_file_ext_s, author_sxt, url_protocol_s, host_s, language_s] from peer pX4MdxchuNOm:agent-ramdisk2

I 2024/08/30 20:07:45 NETWORK * local search (solr): localpeer sent 10/20 references

I 2024/08/30 20:07:45 NETWORK * SEARCH (solr), returned 10 out of 20 documents and 5 facets [url_file_ext_s, author_sxt, url_protocol_s, host_s, language_s] from peer pX4MdxchuNOm:agent-ramdisk2`

[okybaca]

mentioned also in #499 and #406

[smokingwheels]

I have had this with fresh peer no loading yacy is asking passive senior peers to provide remote results.
I have had as high as 17 remote peers not providing any results when it fault condition.
The fault condition lasted about 2 hours after resetting the index twice.

Peer agent-gakinad-w-27 is one of my test peers not online for very long.

No search results on forum

I 2025/01/18 05:20:18 NETWORK * REMOTE SEARCH - no answer from remote peer mGK-Up06fq_S:agent-gakinad-w-27 I 2025/01/18 05:20:18 NETWORK * REMOTE SEARCH - no answer from remote peer OBK4YWZveKmc:MDS-YACY I 2025/01/18 05:20:23 LOCAL_SEARCH * ACCESS CONTROL: WHITELISTED CLIENT FROM 192.168.1.20 gets no search restrictions I 2025/01/18 05:20:23 LOCAL_SEARCH * INIT WORD SEARCH: vx commodore:dn_HPZLSZBPXpRccRev5MQ6B - 10 links to be computed, 10 lines to be displayed I 2025/01/18 05:20:23 LOCAL_SEARCH * EXIT WORD SEARCH: vx commodore - local_rwi_available(0), local_rwi_stored(0), remote_rwi_available(0), remote_rwi_stored(4629), remote_rwi_peerCount(12), local_solr_evicted(0), local_solr_stored(0), remote_solr_available(0), remote_solr_stored(0), remote_solr_peerCount(0), 0 ms I 2025/01/18 05:20:23 EmbeddedSolrConnector.getResponseByParams * QUERY: defType=edismax&q=text_t:(vx commodore)~&fq=text_t:"vx"&start=0&rows=30&hl=true&hl.simple.pre=<b>&hl.simple.post=</b>&hl.snippets=5&hl.fl=text_t&timeAllowed=64 I 2025/01/18 05:20:23 DidYouMean * found 0 unsorted terms, returned 0 sorted suggestions; execution time: 18ms; timeout : 100ms. I 2025/01/18 05:20:28 LOCAL_SEARCH * ACCESS CONTROL: WHITELISTED CLIENT FROM 192.168.1.20 gets no search restrictions I 2025/01/18 05:20:28 LOCAL_SEARCH * INIT WORD SEARCH: vx commodore:dn_HPZLSZBPXpRccRev5MQ6B - 10 links to be computed, 10 lines to be displayed I 2025/01/18 05:20:28 LOCAL_SEARCH * EXIT WORD SEARCH: vx commodore - local_rwi_available(0), local_rwi_stored(0), remote_rwi_available(0), remote_rwi_stored(4629), remote_rwi_peerCount(12), local_solr_evicted(0), local_solr_stored(0), remote_solr_available(0), remote_solr_stored(0), remote_solr_peerCount(0), 0 ms I 2025/01/18 05:20:28 EmbeddedSolrConnector.getResponseByParams * QUERY: defType=edismax&q=text_t:(vx commodore)~&fq=text_t:"vx"&start=0&rows=30&hl=true&hl.simple.pre=<b>&hl.simple.post=</b>&hl.snippets=5&hl.fl=text_t&timeAllowed=64 I 2025/01/18 05:20:28 DidYouMean * found 0 unsorted terms, returned 0 sorted suggestions; execution time: 6ms; timeout : 100ms. I 2025/01/18 05:20:31 LOCAL_SEARCH * ACCESS CONTROL: WHITELISTED CLIENT FROM 192.168.1.20 gets no search restrictions I 2025/01/18 05:20:31 LOCAL_SEARCH * INIT WORD SEARCH: vx commodore:dn_HPZLSZBPXpRccRev5MQ6B - 10 links to be computed, 10 lines to be displayed I 2025/01/18 05:20:31 LOCAL_SEARCH * EXIT WORD SEARCH: vx commodore - local_rwi_available(0), local_rwi_stored(0), remote_rwi_available(0), remote_rwi_stored(4629), remote_rwi_peerCount(12), local_solr_evicted(0), local_solr_stored(0), remote_solr_available(0), remote_solr_stored(0), remote_solr_peerCount(0), 0 ms I 2025/01/18 05:20:31 EmbeddedSolrConnector.getResponseByParams * QUERY: defType=edismax&q=text_t:(vx commodore)~&fq=text_t:"vx"&start=0&rows=30&hl=true&hl.simple.pre=<b>&hl.simple.post=</b>&hl.snippets=5&hl.fl=text_t&timeAllowed=64 I 2025/01/18 05:20:31 DidYouMean * found 0 unsorted terms, returned 0 sorted suggestions; execution time: 5ms; timeout : 100ms. I 2025/01/18 05:20:40 LOCAL_SEARCH * ACCESS CONTROL: WHITELISTED CLIENT FROM 192.168.1.20 gets no search restrictions I 2025/01/18 05:20:40 LOCAL_SEARCH * INIT WORD SEARCH: vx commodore:dn_HPZLSZBPXpRccRev5MQ6B - 10 links to be computed, 10 lines to be displayed I 2025/01/18 05:20:40 LOCAL_SEARCH * EXIT WORD SEARCH: vx commodore - local_rwi_available(0), local_rwi_stored(0), remote_rwi_available(0), remote_rwi_stored(4629), remote_rwi_peerCount(12), local_solr_evicted(0), local_solr_stored(0), remote_solr_available(0), remote_solr_stored(0), remote_solr_peerCount(0), 0 ms