Urgent: Mandatory configs required for Flink HA

[rupeshmore85]

Hello,

We are using spotify operator to schedule our flink jobs(includes HA). However in case when the node on which the jobmanager is hosted goes down, we see that the flinkcluster, it's statefulsets and ha configmaps are deleted, causing downtime.

Note: We are running flinkcluster in a Application mode and below are the settings.

  job:
    autoSavepointSeconds: 86400
    restartPolicy: FromSavepointOnFailure
    maxStateAgeToRestoreSeconds: 518400
    takeSavepointOnUpdate: true
    jarFile: <application.jar>
    savepointsDir: s3://<directory>/jobname/savepoints/
    allowNonRestoredState: true
  flinkProperties:
    high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
    kubernetes.cluster-id: flink-cluster-ha
    kubernetes.namespace: flink-cluster-ns
    high-availability.storageDir: s3://<directory>/recovery/

Questions:

1. What are all the mandatory configs required for job, JM or TM, statefulsets so that the flinkcluster doesn't get deleted ?
2. How sends the SIGTERM , k8s or the flink operator?
3. We also tried adding the below cleanupPolicy, however we see that the job no more takes the savepoint once the node roll happens on the node the JM is hosted. The flinkcluster doesn't get deleted due to the cleanupPolicy and when the JM is hosted on another node, the flink job stops taking savepoints. We need savepoints on new app version deployments.

job:
   cleanupPolicy:
      afterJobCancelled: DeleteCluster
      afterJobFails: KeepCluster
      afterJobSucceeds: KeepCluster

Regards,
Rupesh

[saahilkhurana]

We have tried this with both old version of Spotify Flink operator and the v.0.5.1-alpha.2 and we observe the same behavior.

[rupeshmore85]

Hi @regadas @live-wire can you please help with this discussion when you can ? Its related to the same issue #683