You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi Bhavani,
I have tried to setup the airflow using the script shared by you by building own docker image ( mayurdhande007/my-airflow-example:airflow_v1.1 as well as mayurdhande007/my-airflow-example:airflow_v1.2 ) and ran the deploy.sh as suggested by updating AIRFLOW_IMAGE and AIRFLOW_TAG values in deploy.sh.
After this, it spun up the postgres pod but failing to start the init container.
PFB events for pods:
root@cks-master:~/airflow/airflow-kube-setup/scripts/kube# kubectl describe pod airflow-f499b4866-x748r
Name: airflow-f499b4866-x748r
Namespace: airflow-example
Priority: 0
Node: cks-worker/10.148.0.3
Start Time: Sun, 16 May 2021 05:23:03 +0000
Labels: name=airflow
pod-template-hash=f499b4866
Annotations:
Status: Pending
IP: 10.0.0.17
IPs:
IP: 10.0.0.17
Controlled By: ReplicaSet/airflow-f499b4866
Init Containers:
init:
Container ID: docker://621eacb558016a22f74405759b65831ede42ae9e465a67d233d112d12d8bd4e9
Image: mayurdhande007/my-airflow-example:airflow_v1.1
Image ID: docker-pullable://mayurdhande007/my-airflow-example@sha256:f04102718f01f44c3ac9a38ff7ea9ab8b076ec427002a104b32816f9e95e9783
Port:
Host Port:
Command:
bash
Args:
-cx
./tmp/airflow-test-env-init.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sun, 16 May 2021 05:23:32 +0000
Finished: Sun, 16 May 2021 05:23:34 +0000
Ready: False
Restart Count: 2
Environment:
SQL_ALCHEMY_CONN: <set to the key 'sql_alchemy_conn' in secret 'airflow-secrets'> Optional: false
Mounts:
/root/airflow/airflow.cfg from airflow-configmap (rw,path="airflow.cfg")
/root/airflow/dags from airflow-dags (rw)
/var/run/secrets/kubernetes.io/serviceaccount from airflow-token-bq59v (ro)
Containers:
webserver:
Container ID:
Image: mayurdhande007/my-airflow-example:airflow_v1.1
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
Args:
webserver
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
AIRFLOW_KUBE_NAMESPACE: airflow-example (v1:metadata.namespace)
SQL_ALCHEMY_CONN: <set to the key 'sql_alchemy_conn' in secret 'airflow-secrets'> Optional: false
Mounts:
/root/airflow/airflow.cfg from airflow-configmap (rw,path="airflow.cfg")
/root/airflow/dags from airflow-dags (rw)
/root/airflow/logs from airflow-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from airflow-token-bq59v (ro)
scheduler:
Container ID:
Image: mayurdhande007/my-airflow-example:airflow_v1.1
Image ID:
Port:
Host Port:
Args:
scheduler
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
AIRFLOW_KUBE_NAMESPACE: airflow-example (v1:metadata.namespace)
SQL_ALCHEMY_CONN: <set to the key 'sql_alchemy_conn' in secret 'airflow-secrets'> Optional: false
Mounts:
/root/airflow/airflow.cfg from airflow-configmap (rw,path="airflow.cfg")
/root/airflow/dags from airflow-dags (rw)
/root/airflow/logs from airflow-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from airflow-token-bq59v (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
airflow-dags:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: airflow-dags
ReadOnly: false
airflow-dags-fake:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
airflow-dags-git:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
airflow-logs:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: airflow-logs
ReadOnly: false
airflow-configmap:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-configmap
Optional: false
airflow-token-bq59v:
Type: Secret (a volume populated by a Secret)
SecretName: airflow-token-bq59v
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
Normal Scheduled 52s default-scheduler Successfully assigned airflow-example/airflow-f499b4866-x748r to cks-worker
Normal Pulled 47s kubelet Successfully pulled image "mayurdhande007/my-airflow-example:airflow_v1.1" in 2.856313921s
Normal Pulled 41s kubelet Successfully pulled image "mayurdhande007/my-airflow-example:airflow_v1.1" in 2.743754591s
Normal Pulling 26s (x3 over 49s) kubelet Pulling image "mayurdhande007/my-airflow-example:airflow_v1.1"
Normal Created 23s (x3 over 46s) kubelet Created container init
Normal Started 23s (x3 over 46s) kubelet Started container init
Normal Pulled 23s kubelet Successfully pulled image "mayurdhande007/my-airflow-example:airflow_v1.1" in 2.745180192s
Warning BackOff 6s (x3 over 39s) kubelet Back-off restarting failed container
Hi Bhavani,
I have tried to setup the airflow using the script shared by you by building own docker image ( mayurdhande007/my-airflow-example:airflow_v1.1 as well as mayurdhande007/my-airflow-example:airflow_v1.2 ) and ran the deploy.sh as suggested by updating AIRFLOW_IMAGE and AIRFLOW_TAG values in deploy.sh.
After this, it spun up the postgres pod but failing to start the init container.
PFB events for pods:
root@cks-master:~/airflow/airflow-kube-setup/scripts/kube# kubectl describe pod airflow-f499b4866-x748r
Name: airflow-f499b4866-x748r
Namespace: airflow-example
Priority: 0
Node: cks-worker/10.148.0.3
Start Time: Sun, 16 May 2021 05:23:03 +0000
Labels: name=airflow
pod-template-hash=f499b4866
Annotations:
Status: Pending
IP: 10.0.0.17
IPs:
IP: 10.0.0.17
Controlled By: ReplicaSet/airflow-f499b4866
Init Containers:
init:
Container ID: docker://621eacb558016a22f74405759b65831ede42ae9e465a67d233d112d12d8bd4e9
Image: mayurdhande007/my-airflow-example:airflow_v1.1
Image ID: docker-pullable://mayurdhande007/my-airflow-example@sha256:f04102718f01f44c3ac9a38ff7ea9ab8b076ec427002a104b32816f9e95e9783
Port:
Host Port:
Command:
bash
Args:
-cx
./tmp/airflow-test-env-init.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sun, 16 May 2021 05:23:32 +0000
Finished: Sun, 16 May 2021 05:23:34 +0000
Ready: False
Restart Count: 2
Environment:
SQL_ALCHEMY_CONN: <set to the key 'sql_alchemy_conn' in secret 'airflow-secrets'> Optional: false
Mounts:
/root/airflow/airflow.cfg from airflow-configmap (rw,path="airflow.cfg")
/root/airflow/dags from airflow-dags (rw)
/var/run/secrets/kubernetes.io/serviceaccount from airflow-token-bq59v (ro)
Containers:
webserver:
Container ID:
Image: mayurdhande007/my-airflow-example:airflow_v1.1
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
Args:
webserver
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
AIRFLOW_KUBE_NAMESPACE: airflow-example (v1:metadata.namespace)
SQL_ALCHEMY_CONN: <set to the key 'sql_alchemy_conn' in secret 'airflow-secrets'> Optional: false
Mounts:
/root/airflow/airflow.cfg from airflow-configmap (rw,path="airflow.cfg")
/root/airflow/dags from airflow-dags (rw)
/root/airflow/logs from airflow-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from airflow-token-bq59v (ro)
scheduler:
Container ID:
Image: mayurdhande007/my-airflow-example:airflow_v1.1
Image ID:
Port:
Host Port:
Args:
scheduler
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
AIRFLOW_KUBE_NAMESPACE: airflow-example (v1:metadata.namespace)
SQL_ALCHEMY_CONN: <set to the key 'sql_alchemy_conn' in secret 'airflow-secrets'> Optional: false
Mounts:
/root/airflow/airflow.cfg from airflow-configmap (rw,path="airflow.cfg")
/root/airflow/dags from airflow-dags (rw)
/root/airflow/logs from airflow-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from airflow-token-bq59v (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
airflow-dags:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: airflow-dags
ReadOnly: false
airflow-dags-fake:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
airflow-dags-git:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
airflow-logs:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: airflow-logs
ReadOnly: false
airflow-configmap:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-configmap
Optional: false
airflow-token-bq59v:
Type: Secret (a volume populated by a Secret)
SecretName: airflow-token-bq59v
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
Normal Scheduled 52s default-scheduler Successfully assigned airflow-example/airflow-f499b4866-x748r to cks-worker
Normal Pulled 47s kubelet Successfully pulled image "mayurdhande007/my-airflow-example:airflow_v1.1" in 2.856313921s
Normal Pulled 41s kubelet Successfully pulled image "mayurdhande007/my-airflow-example:airflow_v1.1" in 2.743754591s
Normal Pulling 26s (x3 over 49s) kubelet Pulling image "mayurdhande007/my-airflow-example:airflow_v1.1"
Normal Created 23s (x3 over 46s) kubelet Created container init
Normal Started 23s (x3 over 46s) kubelet Started container init
Normal Pulled 23s kubelet Successfully pulled image "mayurdhande007/my-airflow-example:airflow_v1.1" in 2.745180192s
Warning BackOff 6s (x3 over 39s) kubelet Back-off restarting failed container
I have uploaded the updated templates in github "https://github.com/dhandma/my-airflow-setup-k8s.git" for reference.
Any help would be appreciated.
Many Thanks ,
Dhandma
The text was updated successfully, but these errors were encountered: