Clean install with default values gives "search_phase_execution_exception: all shards failed" error

[jlphillipsphd]

I am trying to get magda running on a k3s cluster (I also have tried similarly under k3d), but I get "Internal Server Error" when accessing the Datasets and Organizations page...

Looking at the logs it seems to be related to ElasticSeach somehow:

kubectl -n magda logs deployment/search-api 

[ERROR] [01/10/2022 23:30:08.189] [search-api-akka.actor.default-dispatcher-29] [akka.actor.ActorSystemImpl(search-api)] Exception when searching                        
au.csiro.data61.magda.search.elasticsearch.Exceptions.ESException: search_phase_execution_exception: all shards failed                                                  
        at au.csiro.data61.magda.search.elasticsearch.Exceptions.ESGenericException$.unapply(Exceptions.scala:109)                                                      
        at au.csiro.data61.magda.search.elasticsearch.ElasticSearchQueryer.$anonfun$searchOrganisations$3(ElasticSearchQueryer.scala:963)                               
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)                                                                         
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)                                                                      
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)                                                                                 
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Looking specifically at the es-data-0 (I get the same error under production ES settings as well when looking at the logs for the es-client), I see:

kubectl -n magda logs statefulset/es-data

[2022-01-10T23:54:56,413][WARN ][r.suppressed             ] [r1lL6eJ] path: /datasets46/_search, params: {index=datasets46}                                     [21/1871]
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed                                                                                         
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) [elasticsearch-6.5.4.jar:6.5.4]                  
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) [elasticsearch-6.5.4.jar:6.5.4]                
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) [elasticsearch-6.5.4.jar:6.5.4]                     
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) [elasticsearch-6.5.4.jar:6.5.4]                                
        at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) [elasticsearch-6.5.4.jar:6.5.4]                                     
        at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:464) [elasticsearch-6.5.4.jar:6.$
.4]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1130) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1247) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1221) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:347) [elasticsearch-6.5.4.jar:6.5.4]                                            [1/1871]
        at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:341) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.search.SearchContextException: no mapping found for `publisher.aggKeywords.keyword` in order to collapse on
        at org.elasticsearch.search.collapse.CollapseBuilder.build(CollapseBuilder.java:234) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:916) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:616) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:592) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:367) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.search.SearchService.access$100(SearchService.java:121) ~[elasticsearch-6.5.4.jar:6.5.4]
        at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339) ~[elasticsearch-6.5.4.jar:6.5.4]
        ... 9 more

I've also tested on 0.0.56 and 1.1.0 and it happens on both versions... so I am not sure what's going on and would like to be able to use Magda instead of CKAN for our project data. Any pointers would be greatly appreciated!

Update:
I just got to know microk8s and deployed v1.1.0 on that as well now, only because I don't know what else to tweak...
I am getting the same errors on microk8s as well, so it's not really specific to k3s/k3d.
This is really just a plain install, from scratch: shouldn't this work?

[t83714]

Hi @jlphillipsphd,
I think your installation might need some initial datasets.
It might help to check out our tutorial repo:
https://github.com/magda-io/magda-brown-bag
Task 2 will install a connector to pull some datasets for your installation.