Deploy/Kubernetes/Search

[dikshauat]

Hi
@Shounak
@rajeevsathish
@keshavprasad,

Please help me with the following error . We are getting it for Deploy/Kubernetes/Search.

TASK [sunbird-deploy : helm upgrade] *******************************************
fatal: [localhost]: FAILED! => {
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m search /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Search/kubernetes/helm_charts/core/search -n dev",
"delta": "0:10:03.736707",
"end": "2022-09-15 11:56:58.891638",
"rc": 1,
"start": "2022-09-15 11:46:55.154931"
}

STDOUT:

Release "search" does not exist. Installing it now.

STDERR:

Error: release search failed, and has been uninstalled due to atomic being set: timed out waiting for the condition

MSG:

non-zero return code

PLAY RECAP *********************************************************************
localhost : ok=5 changed=5 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0

[santhosh-tg]

Hi @dikshauat while deploying the service, can you please check the pod logs to get more information on why its failing ? the timeout is set to 10 minutes so the deployment will be rolled back after that.

[dikshauat]

Hi santosh-tg,

Thanks for responding. It is this we are getting in pods' logs

Initialized application actors for search-service
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/home/sunbird/search-service-1.0-SNAPSHOT/lib/com.google.inject.guice-4.2.2.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-09-20 11:02:26,570 Slf4jLogger started
Initializing SignalHandler...
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
2022-09-20 11:02:27,800 Application started (Prod) (no global state)
2022-09-20 11:02:28,122 Listening for HTTP on /0.0.0.0:9000

[beepdot]

Please deploy the job, go back to Kubernetes and login to pod and please share the output of
curl localhost:9000/health
You will have to do this before helm rolls back the deployment.

[dikshauat]

Hi @keshavprasadms , thanks for your reply. Tried that but I am not able to get into any of the pods. Please check the logs below and kindly help us with the issue

router-7cb45b877d-4smfd 1/1 Running 0 5d22h
search-678df64f5c-d69d5 0/1 Running 0 67s
statsd-exporter-84577db468-nj687 1/1 Running 0 6d22h
taxonomy-5564584bb6-q6vk4 1/1 Running 163 5d22h
telemetry-6d85bf4858-848z9 1/1 Running 162 5d22h
search-678df64f5c-d69d5 0/1 Running 1 76s
^Cdikshauat@cloudshell:~ (diksha-poc)kubectl exec -it search-678df64f5c-d69d5 -- /bin/shsh
Error from server (NotFound): pods "search-678df64f5c-d69d5" not found
dikshauat@cloudshell:~ (diksha-poc)$ kubectl exec -it search-678df64f5c-d69d5 -- /bin/sh
Error from server (NotFound): pods "search-678df64f5c-d69d5" not found
dikshauat@cloudshell:~ (diksha-poc)$ kubectl exec -it "search-678df64f5c-d69d5" -- /bin/sh
Error from server (NotFound): pods "search-678df64f5c-d69d5" not found
dikshauat@cloudshell:~ (diksha-poc)$ kubectl exec -it search-678df64f5c-d69d5 -- /bash
Error from server (NotFound): pods "search-678df64f5c-d69d5" not found

[beepdot]

Not found is a pod not found error. Probably you are giving wrong pod id. Please recheck. You should do this within 10 minutes of deployment as rollback will delete the pods. Please refer to sample command as below

kubectl exec -it -n dev search-b7589d5f9-vmh48 -- sh
/home/sunbird $ 

[dikshauat]

Thank you @keshavprasadms . here is the the inside of pod. please check and help us with the solution

search-678df64f5c-mbmlk 0/1 Running 1 76s
^Cdikshauat@cloudshell:~ (diksha-poc)$
kubectl exec -it -n dev search-678df64f5c-mbmlk -- sh
/home/sunbird $ curl localhost:9000/health
/home/sunbird $ command terminated with exit code 137
dikshauat@cloudshell:~ (diksha-poc)$ ^C
dikshauat@cloudshell:~ (diksha-poc)$ kubectl exec -it -n dev search-678df64f5c-mbmlk -- sh
/home/sunbird $ curl localhost:9000/health
{"id":"api.search-service.health","ver":"3.0","ts":"2022-09-21T06:52:40ZZ","params":{"resmsgid":"0d19371b-b548-461b-95d5-686ee66cc2aa","msgid":null,"err":null,"status":"failed","errmsg":"Ela/home/sunbird $ curl localhost:9000

<title>Not Found</title> <style type="text/css"> html, body, pre { margin: 0; padding: 0; font-family: Monaco, 'Lucida Console', monospace; background: #ECECEC; } h1 { margin: 0; background: #AD632A; padding: 20px 45px; color: #fff; text-shadow: 1px 1px 1px rgba(0,0,0,.3); border-bottom: 1px solid #9F5805; font-size: 28px; } p#detail { margin: 0; padding: 15px 45px; background: #F6A960; border-top: 4px solid #D29052; color: #733512; text-shadow: 1px 1px 1px rgba(255,255,255,.3); font-size: 14px; border-bottom: 1px solid #BA7F5B; } </style>

</head>
<body>
    <h1>Not Found</h1>

    <p id="detail">
        For request 'GET /'
    </p>

</body>

/home/sunbird $ command terminated with exit code 137

[dikshauat]

yes @keshavprasadms , I am able to see all the messages. I went through each one of them and found that I did exactly the same what you suggested

[beepdot]

Can you please check why the job is skipping all the important tasks? It shows skipping for each task.

TASK [es-mapping : Copy indices file] ******************************************
skipping: [10.128.0.23] => (item=all)

TASK [es-mapping : Copy mapping file] ******************************************
skipping: [10.128.0.23] => (item=all)

TASK [es-mapping : Copy ingest-pipeline file] **********************************
skipping: [10.128.0.23] => (item=all)

TASK [es-mapping : Copy indices file] ******************************************
skipping: [10.128.0.23]

[dikshauat]

I really don't know @keshavprasadms . Can you please tell me why is it happening?

[beepdot]

Please share me a screenshot of the ESMapping parameters page.
Go to ESMapping job -> Click on the latest build number -> Click Parameters

[dikshauat]

here is the required snapshot

[dikshauat]

@keshavprasadms ,

Thank you for your response, here we are sharing the screenshot as you mentioned .

[dikshauat]

@keshavprasadms , that person has used 4.3.0 initially and then you asked him to switch to 4.6.0 . But we have never used an incorrect version for ESmapping. Despite of using the correct tag i.e 4.6.0_RC4 , we are not able to bring the search pod up. I even matched the things we have in our config.xml file on jenkins VM with the content of the file you have referred and found no difference. Please suggest

[beepdot]

This is how the job should look. You can delete and recreate the job using the config.xml from github. In your screenshot, I cannot see this.

[dikshauat]

@keshavprasadms , instead of deleting the job, I added a choice parameter with all the three values i.e. indices, mappings and pipelines and ran the job. ESMapping was successful. I then ran Search and it also became successful. Search pod is now up on the cluster but I am not able to search anything on the portal. Trying to figure out the issue.

Thank you so very much for all your help.

[beepdot]

Thats great to hear. You will be able to search only after you create some contents and data. By default there will be no data that you can search.

[dikshauat]

@keshavprasadms ,

please help me with this error

[dikshauat]

[beepdot]

We don't want to discuss too many issues in one thread. Please open new threads for new issues as it will help the community search for similar issues easily.

[luckynvdu]

@dikshauat can you make this thread answered if your particular query is resolved

[jaingaurav126]

Thanks for resolving the issue. Marking it as answered.