Deploy -> KnowledgePlatform ->FlinkJobs

[shounakacharya]

System Configuration Details:

1. KP, DP, DB, Yarn, Jenkins, Other-Services: Ubuntu 18.04 LTS, 4vCPU, 16GB RAM, 250 GB HDD
2. Google Kubernetes engine: 1.21.12-gke.2200 with 4 worker nodes. Note: Currently Sunbird is not compatible with K8s APIs >= 1.22 and we have checked this
3. All systems are running on GCP
4. Github Release Tag: branch_or_tag: release-4.6.0_RC2, job_names_to_deploy: Select All

@gandham-santhosh and @keshavprasadms

Jenkins Logs:

Started by user Admin Deployer
Checking out git https://github.com/project-sunbird/sunbird-learning-platform.git into /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs@script to read kubernetes/pipelines/deploy/Jenkinsfile
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Cloning repository https://github.com/project-sunbird/sunbird-learning-platform.git

git init /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs@script # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-learning-platform.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-learning-platform.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-learning-platform.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/release-4.6.0_RC2^{commit} # timeout=10
git rev-parse release-4.6.0_RC2^{commit} # timeout=10
Checking out Revision d708cf824572355205448d7695e9885b033c769a (release-4.6.0_RC2)
git config core.sparsecheckout # timeout=10
git checkout -f d708cf824572355205448d7695e9885b033c769a # timeout=10
Commit message: "Issue #SB-28039 merge: Merge pull request #1657 from krgauraw/release-4.6.0"
First time build. Skipping changelog.
Running in Durability level: MAX_SURVIVABILITY
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Cloning repository https://github.com/project-sunbird/sunbird-devops.git
git init /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs@libs/deploy-conf # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
First time build. Skipping changelog.
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Cloning repository https://github.com/project-sunbird/sunbird-learning-platform.git
git init /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-learning-platform.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-learning-platform.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-learning-platform.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/release-4.6.0_RC2^{commit} # timeout=10
git rev-parse release-4.6.0_RC2^{commit} # timeout=10
Checking out Revision d708cf824572355205448d7695e9885b033c769a (release-4.6.0_RC2)
git config core.sparsecheckout # timeout=10
git checkout -f d708cf824572355205448d7695e9885b033c769a # timeout=10
Commit message: "Issue #SB-28039 merge: Merge pull request #1657 from krgauraw/release-4.6.0"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy)
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/dev/KnowledgePlatform/FlinkJobs
  [Pipeline] sh
• echo Deploy/dev/KnowledgePlatform/FlinkJobs
  [Pipeline] sh
• echo Deploy/dev/KnowledgePlatform/FlinkJobs
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] echo
  Found environment variable named hub_org with value as: shounakacharya
  [Pipeline] copyArtifacts
  Copied 1 artifact from "ArtifactUpload » dev » KnowledgePlatform » FlinkJobs" build number 4
  [Pipeline] sh
• jq -r .image_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  image_tag not specified, using the image_tag specified in metadata.json.
  [Pipeline] sh
• jq -r .image_tag metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] sh

• echo Deploy/dev/KnowledgePlatform/FlinkJobs
  [Pipeline] sh
• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=KnowledgePlatform, jobName=FlinkJobs, absolute_job_path=ArtifactUpload/dev/KnowledgePlatform/FlinkJobs, agent=master, image_name=knowledge-platform-jobs, image_tag=release-4.6.0_RC5_3bc6f034_2, currentWs=/var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/ansible/deploy_jobs.yml, ansibleExtraArgs=--extra-vars "chart_path=/var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs job_names_to_deploy=video-stream-generator,questionset-publish,search-indexer,asset-enrichment,audit-event-generator,audit-history-indexer,collection-cert-pre-processor,collection-certificate-generator,auto-creator-v2,metrics-data-transformer,content-publish,content-auto-creator,qrcode-image-generator image_tag=release-4.6.0_RC5_3bc6f034_2 image_name=knowledge-platform-jobs" --vault-password-file /var/lib/jenkins/secrets/vault-pass -v}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-cred
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Cloning repository https://github.com/shounakacharya/ansible.git

git init /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/private # timeout=10
Fetching upstream changes from https://github.com/shounakacharya/ansible.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_ASKPASS to set credentials
git fetch --no-tags --progress --depth=1 -- https://github.com/shounakacharya/ansible.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/shounakacharya/ansible.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/main^{commit} # timeout=10
Checking out Revision 9b404cbd9d273d08452839b72b2a56580f5f0b23 (origin/main)
git config core.sparsecheckout # timeout=10
git checkout -f 9b404cbd9d273d08452839b72b2a56580f5f0b23 # timeout=10
Commit message: "update load balancer IP for private k8s ingress"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/private/ansible/inventory/dev/KnowledgePlatform/common.yml /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/private/ansible/inventory/dev/KnowledgePlatform/hosts /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/private/ansible/inventory/dev/KnowledgePlatform/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/ansible/deploy_jobs.yml --extra-vars chart_path=/var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs job_names_to_deploy=video-stream-generator,questionset-publish,search-indexer,asset-enrichment,audit-event-generator,audit-history-indexer,collection-cert-pre-processor,collection-certificate-generator,auto-creator-v2,metrics-data-transformer,content-publish,content-auto-creator,qrcode-image-generator image_tag=release-4.6.0_RC5_3bc6f034_2 image_name=knowledge-platform-jobs --vault-password-file /var/lib/jenkins/secrets/vault-pass -v
  No config file found; using defaults
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

PLAY [local] *******************************************************************

TASK [flink-jobs-deploy : template values.yaml file] ***************************
[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host localhost should use
/usr/bin/python3, but is using /usr/bin/python for backward compatibility with
prior Ansible releases. A future Ansible release will default to using the
discovered platform python for this host. See https://docs.ansible.com/ansible/
2.8/reference_appendices/interpreter_discovery.html for more information. This
feature will be removed in version 2.12. Deprecation warnings can be disabled
by setting deprecation_warnings=False in ansible.cfg.
changed: [localhost] => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": true,
"checksum": "9c81512ab3e3c6dec3ef68ddb80e4730d0000021",
"dest": "/var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs/values.yaml",
"gid": 116,
"group": "jenkins",
"md5sum": "28884af7ca61ca551e50fd933309c1e3",
"mode": "0644",
"owner": "jenkins",
"size": 26038,
"src": "/var/lib/jenkins/.ansible/tmp/ansible-tmp-1660819847.633285-31297-83262200910673/source",
"state": "file",
"uid": 112
}

TASK [flink-jobs-deploy : remove the job.batch] ********************************
failed: [localhost] (item=video-stream-generator) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch video-stream-generator-jobmanager -n flink-kp-dev",
"delta": "0:00:11.449527",
"end": "2022-08-18 16:21:00.201625",
"item": "video-stream-generator",
"rc": 1,
"start": "2022-08-18 16:20:48.752098"
}

STDERR:

Error from server (NotFound): jobs.batch "video-stream-generator-jobmanager" not found

MSG:

non-zero return code

failed: [localhost] (item=questionset-publish) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch questionset-publish-jobmanager -n flink-kp-dev",
"delta": "0:00:00.142422",
"end": "2022-08-18 16:21:00.465572",
"item": "questionset-publish",
"rc": 1,
"start": "2022-08-18 16:21:00.323150"
}

STDERR:

Error from server (NotFound): jobs.batch "questionset-publish-jobmanager" not found

MSG:

non-zero return code

failed: [localhost] (item=search-indexer) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch search-indexer-jobmanager -n flink-kp-dev",
"delta": "0:00:00.157055",
"end": "2022-08-18 16:21:00.722209",
"item": "search-indexer",
"rc": 1,
"start": "2022-08-18 16:21:00.565154"
}

STDERR:

Error from server (NotFound): jobs.batch "search-indexer-jobmanager" not found

MSG:

non-zero return code

failed: [localhost] (item=asset-enrichment) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch asset-enrichment-jobmanager -n flink-kp-dev",
"delta": "0:00:00.152622",
"end": "2022-08-18 16:21:01.005533",
"item": "asset-enrichment",
"rc": 1,
"start": "2022-08-18 16:21:00.852911"
}

STDERR:

Error from server (NotFound): jobs.batch "asset-enrichment-jobmanager" not found

MSG:

non-zero return code

failed: [localhost] (item=audit-event-generator) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch audit-event-generator-jobmanager -n flink-kp-dev",
"delta": "0:00:00.154780",
"end": "2022-08-18 16:21:01.294925",
"item": "audit-event-generator",
"rc": 1,
"start": "2022-08-18 16:21:01.140145"
}

STDERR:

Error from server (NotFound): jobs.batch "audit-event-generator-jobmanager" not found

MSG:

non-zero return code

failed: [localhost] (item=audit-history-indexer) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch audit-history-indexer-jobmanager -n flink-kp-dev",
"delta": "0:00:00.149308",
"end": "2022-08-18 16:21:01.577577",
"item": "audit-history-indexer",
"rc": 1,
"start": "2022-08-18 16:21:01.428269"
}

STDERR:

Error from server (NotFound): jobs.batch "audit-history-indexer-jobmanager" not found

MSG:

non-zero return code

failed: [localhost] (item=collection-cert-pre-processor) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch collection-cert-pre-processor-jobmanager -n flink-kp-dev",
"delta": "0:00:00.155755",
"end": "2022-08-18 16:21:01.864937",
"item": "collection-cert-pre-processor",
"rc": 1,
"start": "2022-08-18 16:21:01.709182"
}

STDERR:

Error from server (NotFound): jobs.batch "collection-cert-pre-processor-jobmanager" not found

MSG:

non-zero return code

failed: [localhost] (item=collection-certificate-generator) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch collection-certificate-generator-jobmanager -n flink-kp-dev",
"delta": "0:00:00.162101",
"end": "2022-08-18 16:21:02.156248",
"item": "collection-certificate-generator",
"rc": 1,
"start": "2022-08-18 16:21:01.994147"
}

STDERR:

Error from server (NotFound): jobs.batch "collection-certificate-generator-jobmanager" not found

MSG:

non-zero return code

failed: [localhost] (item=auto-creator-v2) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch auto-creator-v2-jobmanager -n flink-kp-dev",
"delta": "0:00:00.155205",
"end": "2022-08-18 16:21:02.441727",
"item": "auto-creator-v2",
"rc": 1,
"start": "2022-08-18 16:21:02.286522"
}

STDERR:

Error from server (NotFound): jobs.batch "auto-creator-v2-jobmanager" not found

MSG:

non-zero return code

failed: [localhost] (item=metrics-data-transformer) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch metrics-data-transformer-jobmanager -n flink-kp-dev",
"delta": "0:00:00.154164",
"end": "2022-08-18 16:21:02.730108",
"item": "metrics-data-transformer",
"rc": 1,
"start": "2022-08-18 16:21:02.575944"
}

STDERR:

Error from server (NotFound): jobs.batch "metrics-data-transformer-jobmanager" not found

MSG:

non-zero return code

failed: [localhost] (item=content-publish) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch content-publish-jobmanager -n flink-kp-dev",
"delta": "0:00:00.152328",
"end": "2022-08-18 16:21:03.002867",
"item": "content-publish",
"rc": 1,
"start": "2022-08-18 16:21:02.850539"
}

STDERR:

Error from server (NotFound): jobs.batch "content-publish-jobmanager" not found

MSG:

non-zero return code

failed: [localhost] (item=content-auto-creator) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch content-auto-creator-jobmanager -n flink-kp-dev",
"delta": "0:00:00.154390",
"end": "2022-08-18 16:21:03.297417",
"item": "content-auto-creator",
"rc": 1,
"start": "2022-08-18 16:21:03.143027"
}

STDERR:

Error from server (NotFound): jobs.batch "content-auto-creator-jobmanager" not found

MSG:

non-zero return code

failed: [localhost] (item=qrcode-image-generator) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch qrcode-image-generator-jobmanager -n flink-kp-dev",
"delta": "0:00:00.151935",
"end": "2022-08-18 16:21:03.584761",
"item": "qrcode-image-generator",
"rc": 1,
"start": "2022-08-18 16:21:03.432826"
}

STDERR:

Error from server (NotFound): jobs.batch "qrcode-image-generator-jobmanager" not found

MSG:

non-zero return code

...ignoring

TASK [flink-jobs-deploy : helm upgrade] ****************************************
changed: [localhost] => (item=video-stream-generator) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install video-stream-generator /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs --set job_classname=org.sunbird.job.videostream.task.VideoStreamGeneratorStreamTask --set taskmanager.replicas=1 --set taskmanager.cpu_requests=0.3 -n flink-kp-dev",
"delta": "0:00:01.862156",
"end": "2022-08-18 16:21:05.613820",
"item": "video-stream-generator",
"rc": 0,
"start": "2022-08-18 16:21:03.751664"
}

STDOUT:

Release "video-stream-generator" does not exist. Installing it now.
NAME: video-stream-generator
LAST DEPLOYED: Thu Aug 18 16:21:05 2022
NAMESPACE: flink-kp-dev
STATUS: deployed
REVISION: 1
TEST SUITE: None

changed: [localhost] => (item=questionset-publish) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install questionset-publish /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs --set job_classname=org.sunbird.job.questionset.task.QuestionSetPublishStreamTask --set taskmanager.replicas=1 --set taskmanager.cpu_requests=0.3 -n flink-kp-dev",
"delta": "0:00:00.672908",
"end": "2022-08-18 16:21:06.412323",
"item": "questionset-publish",
"rc": 0,
"start": "2022-08-18 16:21:05.739415"
}

STDOUT:

Release "questionset-publish" does not exist. Installing it now.
NAME: questionset-publish
LAST DEPLOYED: Thu Aug 18 16:21:06 2022
NAMESPACE: flink-kp-dev
STATUS: deployed
REVISION: 1
TEST SUITE: None

changed: [localhost] => (item=search-indexer) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install search-indexer /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs --set job_classname=org.sunbird.job.searchindexer.task.SearchIndexerStreamTask --set taskmanager.replicas=1 --set taskmanager.cpu_requests=0.3 -n flink-kp-dev",
"delta": "0:00:00.754831",
"end": "2022-08-18 16:21:07.285206",
"item": "search-indexer",
"rc": 0,
"start": "2022-08-18 16:21:06.530375"
}

STDOUT:

Release "search-indexer" does not exist. Installing it now.
NAME: search-indexer
LAST DEPLOYED: Thu Aug 18 16:21:06 2022
NAMESPACE: flink-kp-dev
STATUS: deployed
REVISION: 1
TEST SUITE: None

changed: [localhost] => (item=asset-enrichment) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install asset-enrichment /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs --set job_classname=org.sunbird.job.assetenricment.task.AssetEnrichmentStreamTask --set taskmanager.replicas=1 --set taskmanager.cpu_requests=0.3 -n flink-kp-dev",
"delta": "0:00:00.994572",
"end": "2022-08-18 16:21:08.408728",
"item": "asset-enrichment",
"rc": 0,
"start": "2022-08-18 16:21:07.414156"
}

STDOUT:

Release "asset-enrichment" does not exist. Installing it now.
NAME: asset-enrichment
LAST DEPLOYED: Thu Aug 18 16:21:07 2022
NAMESPACE: flink-kp-dev
STATUS: deployed
REVISION: 1
TEST SUITE: None

changed: [localhost] => (item=audit-event-generator) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install audit-event-generator /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs --set job_classname=org.sunbird.job.auditevent.task.AuditEventGeneratorStreamTask --set taskmanager.replicas=1 --set taskmanager.cpu_requests=0.3 -n flink-kp-dev",
"delta": "0:00:00.751019",
"end": "2022-08-18 16:21:09.295648",
"item": "audit-event-generator",
"rc": 0,
"start": "2022-08-18 16:21:08.544629"
}

STDOUT:

Release "audit-event-generator" does not exist. Installing it now.
NAME: audit-event-generator
LAST DEPLOYED: Thu Aug 18 16:21:08 2022
NAMESPACE: flink-kp-dev
STATUS: deployed
REVISION: 1
TEST SUITE: None

changed: [localhost] => (item=audit-history-indexer) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install audit-history-indexer /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs --set job_classname=org.sunbird.job.audithistory.task.AuditHistoryIndexerStreamTask --set taskmanager.replicas=1 --set taskmanager.cpu_requests=0.3 -n flink-kp-dev",
"delta": "0:00:00.935639",
"end": "2022-08-18 16:21:10.359202",
"item": "audit-history-indexer",
"rc": 0,
"start": "2022-08-18 16:21:09.423563"
}

STDOUT:

Release "audit-history-indexer" does not exist. Installing it now.
NAME: audit-history-indexer
LAST DEPLOYED: Thu Aug 18 16:21:09 2022
NAMESPACE: flink-kp-dev
STATUS: deployed
REVISION: 1
TEST SUITE: None

changed: [localhost] => (item=collection-cert-pre-processor) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install collection-cert-pre-processor /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs --set job_classname=org.sunbird.job.collectioncert.task.CollectionCertPreProcessorTask --set taskmanager.replicas=1 --set taskmanager.cpu_requests=0.3 -n flink-kp-dev",
"delta": "0:00:00.832892",
"end": "2022-08-18 16:21:11.321851",
"item": "collection-cert-pre-processor",
"rc": 0,
"start": "2022-08-18 16:21:10.488959"
}

STDOUT:

Release "collection-cert-pre-processor" does not exist. Installing it now.
NAME: collection-cert-pre-processor
LAST DEPLOYED: Thu Aug 18 16:21:10 2022
NAMESPACE: flink-kp-dev
STATUS: deployed
REVISION: 1
TEST SUITE: None

changed: [localhost] => (item=collection-certificate-generator) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install collection-certificate-generator /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs --set job_classname=org.sunbird.job.certgen.task.CertificateGeneratorStreamTask --set taskmanager.replicas=1 --set taskmanager.cpu_requests=0.3 -n flink-kp-dev",
"delta": "0:00:00.800999",
"end": "2022-08-18 16:21:12.264900",
"item": "collection-certificate-generator",
"rc": 0,
"start": "2022-08-18 16:21:11.463901"
}

STDOUT:

Release "collection-certificate-generator" does not exist. Installing it now.
NAME: collection-certificate-generator
LAST DEPLOYED: Thu Aug 18 16:21:11 2022
NAMESPACE: flink-kp-dev
STATUS: deployed
REVISION: 1
TEST SUITE: None

changed: [localhost] => (item=auto-creator-v2) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install auto-creator-v2 /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs --set job_classname=org.sunbird.job.task.AutoCreatorV2StreamTask --set taskmanager.replicas=1 --set taskmanager.cpu_requests=0.3 -n flink-kp-dev",
"delta": "0:00:00.845737",
"end": "2022-08-18 16:21:13.227320",
"item": "auto-creator-v2",
"rc": 0,
"start": "2022-08-18 16:21:12.381583"
}

STDOUT:

Release "auto-creator-v2" does not exist. Installing it now.
NAME: auto-creator-v2
LAST DEPLOYED: Thu Aug 18 16:21:12 2022
NAMESPACE: flink-kp-dev
STATUS: deployed
REVISION: 1
TEST SUITE: None

changed: [localhost] => (item=metrics-data-transformer) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install metrics-data-transformer /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs --set job_classname=org.sunbird.job.metricstransformer.task.MetricsDataTransformerStreamTask --set taskmanager.replicas=1 --set taskmanager.cpu_requests=0.3 -n flink-kp-dev",
"delta": "0:00:00.882051",
"end": "2022-08-18 16:21:14.244060",
"item": "metrics-data-transformer",
"rc": 0,
"start": "2022-08-18 16:21:13.362009"
}

STDOUT:

Release "metrics-data-transformer" does not exist. Installing it now.
NAME: metrics-data-transformer
LAST DEPLOYED: Thu Aug 18 16:21:13 2022
NAMESPACE: flink-kp-dev
STATUS: deployed
REVISION: 1
TEST SUITE: None

changed: [localhost] => (item=content-publish) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install content-publish /var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/helm_charts/datapipeline_jobs --set job_classname=org.sunbird.job.content.task.ContentPublishStreamTask --set taskmanager.replicas=1 --set taskmanager.cpu_requests=0.3 -n flink-kp-dev",
"delta": "0:00:01.032222",
"end": "2022-08-18 16:21:15.411420",
"item": "content-publish",
"rc": 0,
"start": "2022-08-18 16:21:14.379198"
}

STDOUT:

Release "content-publish" does not exist. Installing it now.
NAME: content-publish
LAST DEPLOYED: Thu Aug 18 16:21:15 2022
NAMESPACE: flink-kp-dev
STATUS: deployed
REVISION: 1
TEST SUITE: None

fatal: [localhost]: FAILED! => {}

MSG:

The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'content-auto-creator'

The error appears to be in '/var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/ansible/roles/flink-jobs-deploy/tasks/main.yml': line 12, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

• name: helm upgrade
  ^ here

PLAY RECAP *********************************************************************
localhost : ok=2 changed=2 unreachable=0 failed=1 skipped=0 rescued=0 ignored=1

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• git name-rev --name-only HEAD+
  rev
• cut -d / -f1
• rev
  [Pipeline] sh
• echo Deploy/dev/KnowledgePlatform/FlinkJobs
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/dev/KnowledgePlatform/FlinkJobs
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[beepdot]

Each option of the jenkins deployment job corresponds to one dictionary from this file - https://github.com/project-sunbird/sunbird-learning-platform/blob/release-4.6.0/kubernetes/helm_charts/datapipeline_jobs/values.j2

Example:

The above jenkins job option uses this dictionary - https://github.com/project-sunbird/sunbird-learning-platform/blob/release-4.6.0/kubernetes/helm_charts/datapipeline_jobs/values.j2#L119

It looks like in this release of the code, we dont have any jobs named content-auto-creator. Please deselect this option while deploying the job or remove it from jenkins job configuration.

If you see similar errors as below for other jobs, please follow similar process to remove / deselect them when running the jenkins job

The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'content-auto-creator'

If you had used release-4.6.0 branch while setting up the jenkins server, you should have not even seen this value in your job configuration - https://github.com/project-sunbird/sunbird-devops/blob/release-4.6.0/deploy/jenkins/jobs/Deploy/jobs/dev/jobs/KnowledgePlatform/jobs/FlinkJobs/config.xml#L100-L111

This job was introduced only in release-4.8.0 - https://github.com/project-sunbird/sunbird-devops/blob/release-4.8.0/deploy/jenkins/jobs/Deploy/jobs/dev/jobs/KnowledgePlatform/jobs/FlinkJobs/config.xml#L112

Please use same release branch to setup jenkins server as per the Sunbird-Ed release version.
If Sunbird-Ed version is release-4.6.0, then same branch should be used to setup jenkins server from https://github.com/project-sunbird/sunbird-devops

[shounakacharya]

@keshavprasadms : Thanks. That solved the problem. Marking as answered!

Thanks
Shounak

[monojit18]

@keshavprasadms I am getting the following error. Can you please suggest?

=============================================================================
failed: [localhost] (item=null) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "kubectl delete job.batch null-jobmanager -n flink-kp-dev",
"delta": "0:00:00.287577",
"end": "2022-09-10 09:46:16.207676",
"item": "null",
"rc": 1,
"start": "2022-09-10 09:46:15.920099"
}

STDERR:

Error from server (NotFound): jobs.batch "null-jobmanager" not found

And then,

TASK [flink-jobs-deploy : helm upgrade] ****************************************
fatal: [localhost]: FAILED! => {}

MSG:

The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'null'

The error appears to be in '/var/lib/jenkins/workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/ansible/roles/flink-jobs-deploy/tasks/main.yml': line 12, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

• name: helm upgrade
  ^ here
  ====================================================================================

[monojit18]

@keshavprasadms, @shounakacharya

I have added a string variable in Configure section of the Jenkins job - job_names_to_deploy
with values - video-stream-generator,questionset-publish,search-indexer,asset-enrichment,audit-event-generator,audit-history-indexer,collection-cert-pre-processor,collection-certificate-generator,auto-creator-v2,metrics-data-transformer,content-publish

This is from the link that you had provided earlier - https://github.com/project-sunbird/sunbird-devops/blob/release-4.6.0/deploy/jenkins/jobs/Deploy/jobs/dev/jobs/KnowledgePlatform/jobs/FlinkJobs/config.xml#L100-L111

I saw that is how this file handles the jobs names - workspace/Deploy/dev/KnowledgePlatform/FlinkJobs/kubernetes/ansible/roles/flink-jobs-deploy/tasks/main.yml

This worked for me
Can you please confirm if this approach is correct?

[beepdot]

Yes thats's correct @monojit18