Build for Deploy/Kuberenetes/APIManager failing

[vayu068]

Hi,
Deploy/Kuberenetes/APIManager failing. It seems Kong API 0.14.1 doesn't support Postgress authentication encrypted using SCRAM SHA .
Please let me know Postgress & Kong image version to be used.

Using release tag: release-4.6.0_RC4

Please find full Jenkins logs:

Started by user Pawan Verma
Checking out git https://github.com/project-sunbird/sunbird-devops.git into /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager@script/2b17fab10110704d031125b5ff6396246bdbc852b6799e400e2777264b895f3b to read kubernetes/pipelines/deploy_core/Jenkinsfile
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Cloning repository https://github.com/project-sunbird/sunbird-devops.git

git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager@script/2b17fab10110704d031125b5ff6396246bdbc852b6799e400e2777264b895f3b # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.34.1'
git fetch --tags --force --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
First time build. Skipping changelog.
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.34.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Cloning repository https://github.com/project-sunbird/sunbird-devops.git
git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager@libs/deploy-conf # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.34.1'
git fetch --no-tags --force --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
First time build. Skipping changelog.
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Cloning repository https://github.com/project-sunbird/sunbird-devops.git
git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.34.1'
git fetch --tags --force --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy)
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/dev/Kubernetes/APIManager
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/APIManager
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/APIManager
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] echo
  �[1m�[32mFound environment variable named hub_org with value as: vayu068�[0m
  [Pipeline] copyArtifacts
  Copied 1 artifact from "ArtifactUpload » dev » Core » APIManager" build number 3
  [Pipeline] sh
• jq -r .image_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  �[1m�[33mimage_tag not specified, using the image_tag specified in metadata.json. �[0m
  [Pipeline] sh
• jq -r .image_tag metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] sh

• echo Deploy/dev/Kubernetes/APIManager
  [Pipeline] sh
• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=Kubernetes, jobName=APIManager, absolute_job_path=ArtifactUpload/dev/Core/APIManager, agent=master, image_name=kong, image_tag=0.14.1, currentWs=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/kubernetes/ansible/deploy_core_service.yml, ansibleExtraArgs=--extra-vars "chart_path=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/kubernetes/helm_charts/core/apimanager release_name=apimanager image_tag=0.14.1 image_name=kong role_name=sunbird-deploy" --vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/private
[Pipeline] {
[Pipeline] echo
�[1m�[33mInfo: Branch override is enabled�[0m
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-private-repo
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Cloning repository https://github.com/vayu068/Sunbird_private.git

git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/private # timeout=10
Fetching upstream changes from https://github.com/vayu068/Sunbird_private.git
git --version # timeout=10
git --version # 'git version 2.34.1'
using GIT_ASKPASS to set credentials
git fetch --no-tags --force --progress --depth=1 -- https://github.com/vayu068/Sunbird_private.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/vayu068/Sunbird_private.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6d8113b604861ea485408dc7aeb06e9d063a4119 (origin/master)
git config core.sparsecheckout # timeout=10
git checkout -f 6d8113b604861ea485408dc7aeb06e9d063a4119 # timeout=10
Commit message: "Update secrets.yml"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/private/ansible/inventory/dev/Kubernetes/common.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/private/ansible/inventory/dev/Kubernetes/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/private/ansible/inventory/dev/Kubernetes/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/kubernetes/ansible/deploy_core_service.yml --extra-vars chart_path=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/kubernetes/helm_charts/core/apimanager release_name=apimanager image_tag=0.14.1 image_name=kong role_name=sunbird-deploy --vault-password-file /var/lib/jenkins/secrets/vault-pass
  �[1;35m[WARNING]: Invalid characters were found in group names but not replaced, use�[0m
  �[1;35m-vvvv to see details�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/�[0m
  �[1;35mAPIManager/ansible/inventory/env/hosts with yaml plugin: Syntax Error while�[0m
  �[1;35mloading YAML. did not find expected The error appears to be�[0m
  �[1;35min '/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/ansible/invento�[0m
  �[1;35mry/env/hosts': line 3, column 1, but may be elsewhere in the file depending on�[0m
  �[1;35mthe exact syntax problem. The offending line appears to be: # Replace the�[0m
  �[1;35mvalue deployer to your server login user name, for example ubuntu�[0m
  �[1;35mansible_ssh_user=jenkins ^ here�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/�[0m
  �[1;35mAPIManager/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/worksp�[0m
  �[1;35mace/Deploy/dev/Kubernetes/APIManager/ansible/inventory/env/hosts:144: Section�[0m
  �[1;35m[core:children] includes undefined group: postgresql-slave�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/API�[0m
  �[1;35mManager/ansible/inventory/env/hosts as an inventory source�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: No inventory was parsed, only implicit localhost is available�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: provided hosts list is empty, only localhost is available. Note that�[0m
  �[1;35mthe implicit localhost does not match 'all'�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/�[0m
  �[1;35mdev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-�[0m
  �[1;35mdeploy/defaults/main.yml, line 3, column 1, found a duplicate dict key�[0m
  �[1;35m(sunbird_time_zone). Using last defined value only.�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/�[0m
  �[1;35mdev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-�[0m
  �[1;35mdeploy/defaults/main.yml, line 3, column 1, found a duplicate dict key�[0m
  �[1;35m(nginx_per_ip_connection_limit). Using last defined value only.�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/�[0m
  �[1;35mdev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-�[0m
  �[1;35mdeploy/defaults/main.yml, line 3, column 1, found a duplicate dict key�[0m
  �[1;35m(sunbird_portal_player_cdn_enabled). Using last defined value only.�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/�[0m
  �[1;35mdev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-�[0m
  �[1;35mdeploy/defaults/main.yml, line 3, column 1, found a duplicate dict key�[0m
  �[1;35m(dial_liveness_readiness). Using last defined value only.�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/�[0m
  �[1;35mdev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-�[0m
  �[1;35mdeploy/defaults/main.yml, line 928, column 5, found a duplicate dict key�[0m
  �[1;35m(failureThreshold). Using last defined value only.�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/�[0m
  �[1;35mdev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-�[0m
  �[1;35mdeploy/defaults/main.yml, line 938, column 5, found a duplicate dict key�[0m
  �[1;35m(failureThreshold). Using last defined value only.�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/�[0m
  �[1;35mdev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-�[0m
  �[1;35mdeploy/defaults/main.yml, line 951, column 5, found a duplicate dict key�[0m
  �[1;35m(failureThreshold). Using last defined value only.�[0m
  �[1;35m�[0m
  �[1;35m[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/�[0m
  �[1;35mdev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-�[0m
  �[1;35mdeploy/defaults/main.yml, line 961, column 5, found a duplicate dict key�[0m
  �[1;35m(failureThreshold). Using last defined value only.�[0m
  �[1;35m�[0m

PLAY [local] *******************************************************************
�[1;35m[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/�[0m
�[1;35mdev/Kubernetes/APIManager/ansible/inventory/env/group_vars/all.yml, line 4,�[0m
�[1;35mcolumn 1, found a duplicate dict key (app_es_snapshot_host). Using last defined�[0m
�[1;35mvalue only.�[0m
�[1;35m�[0m
�[1;35m[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/�[0m
�[1;35mdev/Kubernetes/APIManager/ansible/inventory/env/group_vars/all.yml, line 4,�[0m
�[1;35mcolumn 1, found a duplicate dict key (sunbird_es_host). Using last defined�[0m
�[1;35mvalue only.�[0m
�[1;35m�[0m
�[1;35m[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/�[0m
�[1;35mdev/Kubernetes/APIManager/ansible/inventory/env/group_vars/all.yml, line 4,�[0m
�[1;35mcolumn 1, found a duplicate dict key (log_es_snapshot_host). Using last defined�[0m
�[1;35mvalue only.�[0m
�[1;35m�[0m

TASK [check for kong db migration] *********************************************

TASK [kong-migration : check if tables are existing in kong postgres database] ***
�[0;33mchanged: [localhost]�[0m

TASK [kong-migration : run kong migrations for first run if tables are empty] ***
�[0;31mfatal: [localhost]: FAILED! => {�[0m
�[0;31m "changed": false,�[0m
�[0;31m "status": 1�[0m
�[0;31m}�[0m
�[0;31m�[0m
�[0;31mMSG:�[0m
�[0;31m�[0m
�[0;31m2022/08/25 18:02:23 [verbose] Kong: 0.14.1�[0m
�[0;31mError: �[0m
�[0;31m/usr/local/share/lua/5.1/pgmoon/init.lua:211: don't know how to auth: 10�[0m
�[0;31mstack traceback:�[0m
�[0;31m2022/08/25 18:02:23 [verbose] no config file found at /etc/kong/kong.conf�[0m
�[0;31m2022/08/25 18:02:23 [verbose] no config file found at /etc/kong.conf�[0m
�[0;31m2022/08/25 18:02:23 [verbose] no config file, skipping loading�[0m
�[0;31m2022/08/25 18:02:23 [verbose] prefix in use: /usr/local/kong�[0m
�[0;31m2022/08/25 18:02:23 [verbose] running datastore migrations�[0m
�[0;31m [C]: in function 'auth'�[0m
�[0;31m /usr/local/share/lua/5.1/pgmoon/init.lua:211: in function 'connect'�[0m
�[0;31m /usr/local/share/lua/5.1/kong/dao/db/postgres.lua:348: in function 'query'�[0m
�[0;31m /usr/local/share/lua/5.1/kong/dao/db/postgres.lua:579: in function 'current_migrations'�[0m
�[0;31m /usr/local/share/lua/5.1/kong/dao/factory.lua:379: in function 'current_migrations'�[0m
�[0;31m /usr/local/share/lua/5.1/kong/dao/factory.lua:521: in function 'run_migrations'�[0m
�[0;31m /usr/local/share/lua/5.1/kong/cmd/migrations.lua:37: in function 'cmd_exec'�[0m
�[0;31m /usr/local/share/lua/5.1/kong/cmd/init.lua:87: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:87>�[0m
�[0;31m [C]: in function 'xpcall'�[0m
�[0;31m /usr/local/share/lua/5.1/kong/cmd/init.lua:87: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:44>�[0m
�[0;31m /usr/local/bin/kong:7: in function 'file_gen'�[0m
�[0;31m init_worker_by_lua:48: in function <init_worker_by_lua:46>�[0m
�[0;31m [C]: in function 'xpcall'�[0m
�[0;31m init_worker_by_lua:55: in function <init_worker_by_lua:53>�[0m
�[0;31m�[0m
�[0;31m�[0m

PLAY RECAP *********************************************************************
�[0;31mlocalhost�[0m : �[0;32mok=1 �[0m �[0;33mchanged=1 �[0m unreachable=0 �[0;31mfailed=1 �[0m skipped=0 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• git name-rev --name-only HEAD
• rev
• cut -d / -f1
• rev
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/APIManager
  [Pipeline] echo
  �[33m�[1mCould not find env specific Slack channel. Check for global slack channel..�[0m
  [Pipeline] echo
  �[33m�[1mCould not find slack environment variable. Skipping slack notification..�[0m
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/dev/Kubernetes/APIManager
  [Pipeline] echo
  �[33m�[1mCould not find env specific email group. Check for global email group..�[0m
  [Pipeline] echo
  �[33m�[1mCould not find global email group variable. Skipping email notification..�[0m
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[beepdot]

Please check if Kong support such authentication mechanisms? https://docs.konghq.com/gateway-oss/0.14.x/
If it doesn't, there is nothing we can do from Sunbird end. I would advise you to reach out to kong devs on their forums.
Other option would be to use an auth which is supported. You could also explore pgpool and pgbouncer if it fits your usecase as an intermediary between kong and postgres.

[vayu068]

Rebuilt Core/ApiManager and then redeployed. It worked.