Package v1beta1 contains API Schema definitions for the flinkoperator v1beta1 API group
Appears in:
| Field | Description |
|---|---|
name string |
BatchScheduler name. |
queue string |
(Optional) Queue defines the queue in which resources will be allocates; if queue is not specified, resources will be allocated in the schedulers default queue. |
priorityClassName string |
(Optional) If specified, indicates the PodGroup's priority. "system-node-critical" and "system-cluster-critical" are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the priority will be default or zero if there is no default. |
CleanupPolicy defines the action to take after job finishes. Use one of KeepCluster, DeleteCluster, DeleteTaskManager for the below fields.
Appears in:
| Field | Description |
|---|---|
afterJobSucceeds CleanupAction |
Action to take after job succeeds, default: DeleteCluster. |
afterJobFails CleanupAction |
Action to take after job fails, default: KeepCluster. |
afterJobCancelled CleanupAction |
Action to take after job is cancelled, default: DeleteCluster. |
Appears in:
| Field | Description |
|---|---|
name string |
The resource name of the component. |
state ComponentState |
The state of the component. |
FlinkCluster is the Schema for the flinkclusters API
| Field | Description |
|---|---|
apiVersion string |
flinkoperator.k8s.io/v1beta1 |
kind string |
FlinkCluster |
metadata ObjectMeta |
Refer to Kubernetes API documentation for fields of metadata. |
spec FlinkClusterSpec |
FlinkClusterComponentsStatus defines the observed status of the components of a FlinkCluster.
Appears in:
| Field | Description |
|---|---|
configMap ConfigMapStatus |
The state of configMap. |
jobManager JobManagerStatus |
The state of JobManager. |
jobManagerService JobManagerServiceStatus |
The state of JobManager service. |
jobManagerIngress JobManagerIngressStatus |
The state of JobManager ingress. |
taskManager TaskManagerStatus |
The state of TaskManager. |
job JobStatus |
The status of the job, available only when JobSpec is provided. |
Control state
Appears in:
| Field | Description |
|---|---|
name string |
Control name |
details object (keys:string, values:string) |
Control data |
state string |
State |
message string |
Message |
updateTime string |
State update time |
FlinkClusterSpec defines the desired state of FlinkCluster
Appears in:
| Field | Description |
|---|---|
flinkVersion string |
The version of Flink to be managed. This version must match the version in the image. |
image ImageSpec |
Flink image for JobManager, TaskManager and job containers. |
serviceAccountName string |
(Optional) The service account assigned to JobManager, TaskManager and Job submitter Pods. If empty, the default service account in the namespace will be used. |
batchSchedulerName string |
Deprecated: BatchSchedulerName specifies the batch scheduler name for JobManager, TaskManager. If empty, no batch scheduling is enabled. |
batchScheduler BatchSchedulerSpec |
(Optional) BatchScheduler specifies the batch scheduler for JobManager, TaskManager. If empty, no batch scheduling is enabled. |
podDisruptionBudget PodDisruptionBudgetSpec |
(Optional) Defines the PodDisruptionBudget for JobManager and TaskManager. If empty, no PodDisruptionBudget is created. |
jobManager JobManagerSpec |
(Optional) Flink JobManager spec. |
taskManager TaskManagerSpec |
(Optional) Flink TaskManager spec. |
job JobSpec |
(Optional) Job spec. If specified, this cluster is an ephemeral Job Cluster, which will be automatically terminated after the job finishes; otherwise, it is a long-running Session Cluster. |
envVars EnvVar array |
(Optional) Environment variables shared by all JobManager, TaskManager and job containers. More info |
envFrom EnvFromSource array |
(Optional) Environment variables injected from a source, shared by all JobManager, TaskManager and job containers. More info |
flinkProperties object (keys:string, values:string) |
(Optional) Flink properties which are appened to flink-conf.yaml. |
hadoopConfig HadoopConfig |
(Optional) Config for Hadoop. |
gcpConfig GCPConfig |
(Optional) Config for GCP. |
logConfig object (keys:string, values:string) |
(Optional) The logging configuration, which should have keys 'log4j-console.properties' and 'logback-console.xml'. These will end up in the 'flink-config-volume' ConfigMap, which gets mounted at /opt/flink/conf. If not provided, defaults that log to console only will be used. - log4j-console.properties: The contents of the log4j properties file to use. If not provided, a default that logs only to stdout will be provided. - logback-console.xml: The contents of the logback XML file to use. If not provided, a default that logs only to stdout will be provided. - Other arbitrary keys are also allowed, and will become part of the ConfigMap. |
revisionHistoryLimit integer |
The maximum number of revision history to keep, default: 10. |
recreateOnUpdate boolean |
Recreate components when updating flinkcluster, default: true. |
GCPConfig defines configs for GCP.
Appears in:
| Field | Description |
|---|---|
serviceAccount GCPServiceAccount |
GCP service account. |
GCPServiceAccount defines the config about GCP service account.
Appears in:
| Field | Description |
|---|---|
secretName string |
The name of the Secret holding the GCP service account key file. The Secret must be in the same namespace as the FlinkCluster. |
keyFile string |
The name of the service account key file. |
mountPath string |
The path where to mount the Volume of the Secret. |
HadoopConfig defines configs for Hadoop.
Appears in:
| Field | Description |
|---|---|
configMapName string |
The name of the ConfigMap which contains the Hadoop config files. The ConfigMap must be in the same namespace as the FlinkCluster. |
mountPath string |
The path where to mount the Volume of the ConfigMap. default: /etc/hadoop/conf. |
Appears in:
| Field | Description |
|---|---|
minReplicas integer |
minReplicas is the lower limit for the number of replicas to which the autoscaler can scale down. It defaults to 1 pod. minReplicas is allowed to be 0 if the alpha feature gate HPAScaleToZero is enabled and at least one Object or External metric is configured. Scaling is active as long as at least one metric value is available. |
maxReplicas integer |
maxReplicas is the upper limit for the number of replicas to which the autoscaler can scale up. It cannot be less that minReplicas. |
metrics MetricSpec array |
metrics contains the specifications for which to use to calculate the desired replica count (the maximum replica count across all metrics will be used). The desired replica count is calculated multiplying the ratio between the target value and the current value by the current number of pods. Ergo, metrics used must decrease as the pod count is increased, and vice-versa. See the individual metric source types for more information about how each type of metric must respond. If not set, the default metric will be set to 80% average CPU utilization. |
behavior HorizontalPodAutoscalerBehavior |
behavior configures the scaling behavior of the target in both Up and Down directions (scaleUp and scaleDown fields respectively). If not set, the default HPAScalingRules for scale up and scale down are used. |
ImageSpec defines Flink image of JobManager and TaskManager containers.
Appears in:
| Field | Description |
|---|---|
name string |
Flink image name. |
pullPolicy PullPolicy |
Image pull policy. One of Always, Never, IfNotPresent, default: Always. if :latest tag is specified, or IfNotPresent otherwise. More info |
pullSecrets LocalObjectReference array |
(Optional) Secrets for image pull. More info |
JobManagerIngressSpec defines ingress of JobManager
Appears in:
| Field | Description |
|---|---|
hostFormat string |
(Optional) Ingress host format. ex) {{$clusterName}}.example.com |
annotations object (keys:string, values:string) |
_(Optional)_Annotations for ingress configuration. More info |
useTls boolean |
TLS use, default: false. |
tlsSecretName string |
_(Optional)_TLS secret name. |
JobManagerIngressStatus defines the status of a JobManager ingress.
Appears in:
| Field | Description |
|---|---|
name string |
The name of the Kubernetes ingress resource. |
state ComponentState |
The state of the component. |
urls string array |
The URLs of ingress. |
JobManagerPorts defines ports of JobManager.
Appears in:
| Field | Description |
|---|---|
rpc integer |
RPC port, default: 6123. |
blob integer |
Blob port, default: 6124. |
query integer |
Query port, default: 6125. |
ui integer |
UI port, default: 8081. |
JobManagerServiceStatus defines the observed state of FlinkCluster
Appears in:
| Field | Description |
|---|---|
name string |
The name of the Kubernetes jobManager service. |
state ComponentState |
The state of the component. |
nodePort integer |
(Optional) The node port, present when accessScope is NodePort. |
loadBalancerIngress LoadBalancerIngress array |
(Optional) The load balancer ingress, present when accessScope is VPC or External |
JobManagerSpec defines properties of JobManager.
Appears in:
| Field | Description |
|---|---|
replicas integer |
The number of JobManager replicas, default: 1 |
accessScope string |
Access scope, default: Cluster. Cluster: accessible from within the same cluster. VPC: accessible from within the same VPC. External: accessible from the internet. NodePort: accessible through node port. Headless: pod IPs assumed to be routable and advertised directly with clusterIP: None``. Currently VPC, External` are only available for GKE. |
ServiceAnnotations object (keys:string, values:string) |
(Optional) Define JobManager Service annotations for configuration. |
ServiceLabels object (keys:string, values:string) |
(Optional) Define JobManager Service labels for configuration. |
ingress JobManagerIngressSpec |
(Optional) Provide external access to JobManager UI/API. |
ports JobManagerPorts |
Ports that JobManager listening on. |
extraPorts NamedPort array |
(Optional) Extra ports to be exposed. For example, Flink metrics reporter ports: Prometheus, JMX and so on. Each port number and name must be unique among ports and extraPorts. |
resources ResourceRequirements |
Compute resources required by each JobManager container. default: 2 CPUs with 2Gi Memory. It Cannot be updated. More info |
memoryOffHeapRatio integer |
Percentage of off-heap memory in containers, as a safety margin to avoid OOM kill, default: 25 |
memoryOffHeapMin Quantity |
Minimum amount of off-heap memory in containers, as a safety margin to avoid OOM kill, default: 600M You can express this value like 600M, 572Mi and 600e6 More info |
memoryProcessRatio integer |
For Flink 1.10+. Percentage of memory process, as a safety margin to avoid OOM kill, default: 80 |
volumes Volume array |
(Optional) Volumes in the JobManager pod. More info |
volumeMounts VolumeMount array |
(Optional) Volume mounts in the JobManager container. More info |
volumeClaimTemplates PersistentVolumeClaim array |
(Optional) A template for persistent volume claim each requested and mounted to JobManager pod, This can be used to mount an external volume with a specific storageClass or larger captivity (for larger/faster state backend). More info |
initContainers Container array |
(Optional) Init containers of the Job Manager pod. More info |
affinity Affinity |
(Optional) Defines the affinity of the JobManager pod More info |
nodeSelector object (keys:string, values:string) |
(Optional) Selector which must match a node's labels for the JobManager pod to be scheduled on that node. More info |
tolerations Toleration array |
(Optional) Defines the node affinity of the JobManager pod More info |
sidecars Container array |
(Optional) Sidecar containers running alongside with the JobManager container in the pod. More info |
podAnnotations object (keys:string, values:string) |
(Optional) JobManager StatefulSet pod template annotations. More info |
securityContext PodSecurityContext |
(Optional) SecurityContext of the JobManager pod. More info |
podLabels object (keys:string, values:string) |
(Optional) JobManager StatefulSet pod template labels. More info |
livenessProbe Probe |
Container liveness probe If omitted, a default value will be used. More info |
readinessProbe Probe |
Container readiness probe If omitted, a default value will be used. More info |
hostAliases HostAlias array |
(Optional) Adding entries to JobManager pod /etc/hosts with HostAliases More info |
Appears in:
| Field | Description |
|---|---|
name string |
The resource name of the component. |
state ComponentState |
The state of the component. |
replicas integer |
replicas is the number of desired replicas. |
readyReplicas integer |
readyReplicas is the number of created pods with a Ready Condition. |
ready string |
JobSpec defines properties of a Flink job.
Appears in:
| Field | Description |
|---|---|
classPath string array |
(Optional) Adds URLs to each user code classloader on all nodes in the cluster. The paths must specify a protocol (e.g. file://) and be accessible on all nodes (e.g. by means of a NFS share). The protocol must be supported by the {@link java.net.URLClassLoader}. You may add support to more protocol by setting the java.protocol.handler.pkgs java option |
jarFile string |
(Optional) JAR file of the job. It could be a local file or remote URI, depending on which protocols (e.g., https://, gs://) are supported by the Flink image. |
className string |
(Optional) Fully qualified Java class name of the job. |
pyFile string |
(Optional) Python file of the job. It could be a local file or remote URI (e.g.,https://, gs://). |
pyFiles string |
(Optional) Python files of the job. It could be a local file (with .py/.egg/.zip/.whl), directory or remote URI (e.g.,https://, gs://). See the Flink argument --pyFiles for the detail. |
pyModule string |
(Optional) Python module path of the job entry point. Must use with pythonFiles. |
args string array |
(Optional) Command-line args of the job. |
fromSavepoint string |
(Optional) FromSavepoint where to restore the job from Savepoint where to restore the job from (e.g., gs://my-savepoint/1234). If flink job must be restored from the latest available savepoint when Flink job updating, this field must be unspecified. |
allowNonRestoredState boolean |
Allow non-restored state, default: false. |
savepointsDir string |
(Optional) Savepoints dir where to store savepoints of the job. |
takeSavepointOnUpdate boolean |
(Optional) Should take savepoint before updating job, default: true. If this is set as false, maxStateAgeToRestoreSeconds must be provided to limit the savepoint age to restore. |
maxStateAgeToRestoreSeconds integer |
(Optional) Maximum age of the savepoint that allowed to restore state. This is applied to auto restart on failure, update from stopped state and update without taking savepoint. If nil, job can be restarted only when the latest savepoint is the final job state (created by "stop with savepoint") - that is, only when job can be resumed from the suspended state. |
autoSavepointSeconds integer |
(Optional) Automatically take a savepoint to the savepointsDir every n seconds. |
savepointGeneration integer |
(Optional) Update this field to jobStatus.savepointGeneration + 1 for a running job cluster to trigger a new savepoint to savepointsDir on demand. |
parallelism integer |
(Optional) Job parallelism; if not set parallelism will be #replicas * #slots. |
noLoggingToStdout boolean |
No logging output to STDOUT, default: false. |
volumes Volume array |
(Optional) Volumes in the Job pod. More info |
volumeMounts VolumeMount array |
(Optional) Volume mounts in the Job container. More info |
initContainers Container array |
(Optional) Init containers of the Job pod. A typical use case could be using an init container to download a remote job jar to a local path which is referenced by the jarFile property. More info |
affinity Affinity |
(Optional) Defines the affinity of the Job submitter pod More info |
nodeSelector object (keys:string, values:string) |
(Optional) Selector which must match a node's labels for the Job submitter pod to be scheduled on that node. More info |
tolerations Toleration array |
(Optional) Defines the node affinity of the Job submitter pod More info |
restartPolicy JobRestartPolicy |
Restart policy when the job fails, one of Never, FromSavepointOnFailure, default: Never. Never means the operator will never try to restart a failed job, manual cleanup and restart is required. FromSavepointOnFailure means the operator will try to restart the failed job from the savepoint recorded in the job status if available; otherwise, the job will stay in failed state. This option is usually used together with autoSavepointSeconds and savepointsDir. |
cleanupPolicy CleanupPolicy |
The action to take after job finishes. |
cancelRequested boolean |
Deprecated: (Optional) Request the job to be cancelled. Only applies to running jobs. If savePointsDir is provided, a savepoint will be taken before stopping the job. |
podAnnotations object (keys:string, values:string) |
(Optional) Job pod template annotations. More info |
podLabels object (keys:string, values:string) |
(Optional) Job pod template labels. More info |
resources ResourceRequirements |
(Optional) Compute resources required by each Job container. If omitted, a default value will be used. It Cannot be updated. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ |
securityContext PodSecurityContext |
(Optional) SecurityContext of the Job pod. More info |
hostAliases HostAlias array |
(Optional) Adding entries to Job pod /etc/hosts with HostAliases More info |
mode JobMode |
Job running mode, "Blocking", "Detached", default: "Detached" |
JobStatus defines the status of a job.
Appears in:
| Field | Description |
|---|---|
id string |
The ID of the Flink job. |
name string |
The Name of the Flink job. |
submitterName string |
The name of the Kubernetes job resource. |
submitterExitCode integer |
Exit code of the JubSubmitter job resource. |
state JobState |
The state of the Flink job deployment. |
fromSavepoint string |
The actual savepoint from which this job started. In case of restart, it might be different from the savepoint in the job spec. |
savepointGeneration integer |
The generation of the savepoint in savepointsDir taken by the operator. The value starts from 0 when there is no savepoint and increases by 1 for each successful savepoint. |
savepointLocation string |
Savepoint location. |
savepointTime string |
Last successful savepoint completed timestamp. |
finalSavepoint boolean |
The savepoint recorded in savepointLocation is the final state of the job. |
deployTime string |
The timestamp of the Flink job deployment that creating job submitter. |
startTime string |
The Flink job started timestamp. |
restartCount integer |
The number of restarts. |
completionTime Time |
Job completion time. Present when job is terminated regardless of its state. |
failureReasons string array |
Reasons for the job failure. Present if job state is Failure |
NamedPort defines the container port properties.
Appears in:
| Field | Description |
|---|---|
name string |
(Optional) If specified, this must be an IANA_SVC_NAME and unique within the pod. Each named port in a pod must have a unique name. Name for the port that can be referred to by services. |
containerPort integer |
Number of port to expose on the pod's IP address. This must be a valid port number, 0 < x < 65536. |
protocol string |
Protocol for port. One of UDP, TCP, or SCTP, default: TCP. |
Appears in:
| Field | Description |
|---|---|
currentRevision string |
CurrentRevision indicates the version of FlinkCluster. |
nextRevision string |
NextRevision indicates the version of FlinkCluster updating. |
collisionCount integer |
collisionCount is the count of hash collisions for the FlinkCluster. The controller uses this field as a collision avoidance mechanism when it needs to create the name for the newest ControllerRevision. |
SavepointStatus is the status of savepoint progress.
Appears in:
| Field | Description |
|---|---|
jobID string |
The ID of the Flink job. |
triggerID string |
Savepoint trigger ID. |
triggerTime string |
Savepoint triggered time. |
triggerReason SavepointReason |
Savepoint triggered reason. |
requestTime string |
Savepoint status update time. |
state string |
Savepoint state. |
message string |
Savepoint message. |
TaskManagerPorts defines ports of TaskManager.
Appears in:
| Field | Description |
|---|---|
data integer |
Data port, default: 6121. |
rpc integer |
RPC port, default: 6122. |
query integer |
Query port, default: 6125. |
TaskManagerSpec defines properties of TaskManager.
Appears in:
| Field | Description |
|---|---|
deploymentType DeploymentType |
(Optional) Defines the replica workload's type: StatefulSet or Deployment. If not specified, the default value is StatefulSet. |
replicas integer |
The number of replicas. default: 3 |
ports TaskManagerPorts |
Ports that TaskManager listening on. |
extraPorts NamedPort array |
(Optional) Extra ports to be exposed. For example, Flink metrics reporter ports: Prometheus, JMX and so on. |
resources ResourceRequirements |
Compute resources required by each TaskManager container. default: 2 CPUs with 2Gi Memory. It Cannot be updated. More info |
memoryOffHeapRatio integer |
Percentage of off-heap memory in containers, as a safety margin to avoid OOM kill, default: 25 |
memoryOffHeapMin Quantity |
Minimum amount of off-heap memory in containers, as a safety margin to avoid OOM kill, default: 600M You can express this value like 600M, 572Mi and 600e6 More info |
memoryProcessRatio integer |
For Flink 1.10+. Percentage of process memory, as a safety margin to avoid OOM kill, default: 20 |
volumes Volume array |
(Optional) Volumes in the TaskManager pods. More info |
volumeMounts VolumeMount array |
(Optional) Volume mounts in the TaskManager containers. More info |
volumeClaimTemplates PersistentVolumeClaim array |
(Optional) A template for persistent volume claim each requested and mounted to TaskManager pod, This can be used to mount an external volume with a specific storageClass or larger captivity (for larger/faster state backend). More info If deploymentType: StatefulSet is used, these templates will be added to the taskManager statefulset template, hence mounting persistent-pvcs to the indexed statefulset pods. If deploymentType: Deployment is used, these templates are appended to the Ephemeral Volumes in the PodSpec, hence mounting ephemeral-pvcs to the replicaset pods. |
initContainers Container array |
(Optional) Init containers of the Task Manager pod. More info |
affinity Affinity |
(Optional) Defines the affinity of the Task Manager pod More info |
nodeSelector object (keys:string, values:string) |
(Optional) Selector which must match a node's labels for the Task Manager pod to be scheduled on that node. More info |
tolerations Toleration array |
(Optional) Defines the node affinity of the Task Manager pod More info |
sidecars Container array |
(Optional) Sidecar containers running alongside with the TaskManager container in the pod. More info |
podAnnotations object (keys:string, values:string) |
(Optional) TaskManager StatefulSet pod template annotations. More info |
securityContext PodSecurityContext |
(Optional) SecurityContext of the TaskManager pod. More info |
podLabels object (keys:string, values:string) |
(Optional) TaskManager StatefulSet pod template labels. More info |
livenessProbe Probe |
Container liveness probe If omitted, a default value will be used. More info |
readinessProbe Probe |
Container readiness probe If omitted, a default value will be used. More info |
hostAliases HostAlias array |
(Optional) Adding entries to TaskManager pod /etc/hosts with HostAliases More info |
horizontalPodAutoscaler HorizontalPodAutoscalerSpec |
(Optional) HorizontalPodAutoscaler for TaskManager. More info |
Appears in:
| Field | Description |
|---|---|
name string |
The resource name of the component. |
state ComponentState |
The state of the component. |
replicas integer |
replicas is the number of desired Pods. |
readyReplicas integer |
readyReplicas is the number of created pods with a Ready Condition. |
ready string |
|
selector string |