Deployment to Google Kubernetes fails

[MSSP-BLKing]

Overview of the issue

During kubectl-apply.sh
Received error
error: error validating "console/jhipster-elasticsearch.yml": error validating data:
ValidationError(StatefulSet.spec): missing required field "selector" in
io.k8s.api.apps.v1.StatefulSetSpec; if you choose to ignore these errors, turn validation off with --
validate=false

Logstash Deployment Fails “Pods are pending”

jhipster-import-dashboard Job fails on “Pods are pending”

JHipster Console starts, but when you access it you will see that the elasticsearch plugin failed. No pages will load, only the status page is displayed with the error “ Unable to connect to Elasticsearch at http://jhipster-elasticsearch:9200/.”

Motivation for or Use Case

Without the JHipster Console, logstash, and elasticsearch JHipster is not at it's full capability.

Reproduce the error

Steps are really long, I wrote a confluence article and made it public here https://blkingllc.atlassian.net/wiki/spaces/JHIP/pages/723582987/Google+Kubernetes

Related issues

Not one that's open.

Suggest a Fix

Days worth of troubleshooting and I haven't found the solution :(

JHipster Version(s)

6.6.0

JHipster configuration

Entity configuration(s) entityName.json files generated in the .jhipster directory

No Entities are configured.

Browsers and Operating System

Chrome Latest and Ubuntu; but using GKE Kubernetes

• Checking this box is mandatory (this is just to show you read everything)

[SudharakaP]

@BLKingLLC : Can you please provide the .yo-rc.json file that you used to deploy the project that you saw this error? 😄

[MSSP-BLKing]

{
"generator-jhipster": {
"promptValues": {
"packageName": "net.blking"
},
"jhipsterVersion": "6.4.1",
"applicationType": "monolith",
"baseName": "burgershop",
"packageName": "net.blking",
"packageFolder": "net/blking",
"serverPort": "8080",
"authenticationType": "jwt",
"cacheProvider": "ehcache",
"enableHibernateCache": true,
"websocket": false,
"databaseType": "sql",
"devDatabaseType": "h2Disk",
"prodDatabaseType": "mysql",
"searchEngine": "elasticsearch",
"messageBroker": false,
"serviceDiscoveryType": false,
"buildTool": "maven",
"enableSwaggerCodegen": false,
"jwtSecretKey": "MmJmOTMzM2VkZWQ1MTY2MzZlZDEwMWE3ZWVmMjMxNmNiMDg5MjdmMDFkYzI3OTRkODk0MThkMTFiODE5N2E4OWM2ZTQwNjI5ZTNiNjQ5YTk3YjE0MDAxYTQzNjQzMjg1NWQ5ZWY5NDM2MTVhZjg0NDU1NTE3NmY3ZGU2MzI2MjM=",
"embeddableLaunchScript": false,
"useSass": true,
"clientPackageManager": "npm",
"clientFramework": "angularX",
"clientTheme": "none",
"clientThemeVariant": "",
"testFrameworks": [],
"jhiPrefix": "jhi",
"entitySuffix": "",
"dtoSuffix": "DTO",
"otherModules": [],
"enableTranslation": false,
"blueprints": [],
"appsFolders": [
"burgershop"
],
"directoryPath": "../",
"dockerRepositoryName": "gcr.io/ody-offer-letter",
"dockerPushCommand": "gcloud docker push",
"kubernetesNamespace": "burgershop",
"kubernetesServiceType": "LoadBalancer",
"ingressDomain": "",
"monitoring": "elk",
"istio": false
}
}

[SudharakaP]

@BLKingLLC : I tried this out using the same .yo-rc.json file you provided and I couldn't recreate this exact same error. However I did see some error which might also point to the problem you are facing.

I had to increase the memory for Elasticsearch, since otherwise Elasticsearch fails to start and thus the application fails on startup as well. After increasing the memory allocation everything seems to run fine. 😄

Please try increasing the memory to Elasticsearch. Chane lines 41-47 in burgershop-elasticsearch.yml as below and redeploy.

resources:
    requests:
        memory: '1Gi'
        cpu: '2'
    limits:
        memory: '2Gi'
        cpu: '2'

I can look into increasing the memory requirement for Elasticsearch if this works for you; let us know. 🤔

[MSSP-BLKing]

@SudharakaP I tried your modification and it didn't help. What did help, however was changing my google cluster virtual CPUs; so I think what needs to happen is just a documentation update to state what the minimum virtual CPUs and Memory are for a full deployment.

I do, however, still get a failure on loading the burgershop app itself. It crashes with this error Could not connect to http://burgershop-elasticsearch.burgershop.svc.cluster.local:9200

[SudharakaP]

@BLKingLLC : Can you elaborate more on what change you did for your virtual CPUs; what memory did you use? I can do some more testing on this to find out some reasonable defaults. 😄

It seems that your app cannot connect to the elasticsearch cluster. In my case I had the same error but when the memory was increased it went away. Could you share the elasticsearch cluster logs? I think you can access them through the GCP Stackdriver logging. 😄

[MSSP-BLKing]

@SudharakaP I was originally creating the cluster with this command gcloud container clusters create burgershop This didn't provide enough CPUs so I created a cluster through the console and picked a High CPU cluster. This solved the not enough cpu error.

Even with the cpu/memory change that you suggested I get this error Cannot schedule pods: persistentvolumeclaim "burgershop-mariadb-pvc" not found. Then when I change the burgershop-elasticsearch.yml to point to burgershop-mysql-pvc it never initializes. Even with the memory and cpu changes.

[PierreBesson]

Indeed you need quite a bit of compute resources, to run the JHipster Console in Kubernetes. However it's hard to say what the absolute minimum would be. @BLKingLLC could you share with us the specs of the cluster which was "too small" and the one which was "big enough".

[MSSP-BLKing]

The small/default cluster had 3 vCPUs and 11.25GB of memory whereas the high-cpu cluster had 8 vCPUs and 30.0GB of memory.

[saturnism]

@mraible and i ran into the same issue recently.

the PVC claim is fixed in 6.7.1 iirc by @xetys

there were additional issues #11302 #11301 to be fixed in #11303

[pascalgrimaud]

if I understand well, it should be fixed by #11303
Feel free to comment if it's not the case so I can reopen