Kubernetes (K8S) is utilized for hosting applications in a containerized and automated manner. The architecture is divided into Master Nodes and Worker Nodes, each with specific responsibilities:
- Master Nodes: Manage, plan, schedule, and monitor the nodes.
- Worker Nodes: Host applications as containers.
Key components within the nodes include:
- ETCD: Stores cluster information.
- Kube-Scheduler: Schedules containers on nodes.
- Kubelet: Listens for instructions from the API server and manages containers on the node.
- Kube-proxy: Enables communication between services within the cluster.
Containerd serves as the container runtime interface, and interaction with this daemon can be done using tools like ctr, nerdctl, or crictl from a Kubernetes perspective.
Kubernetes interacts with Docker through a middleware utility called dockershim. However, Kubernetes now relies on Containerd instead of Docker, with dockershim being used primarily for debugging purposes.
ETCD is a distributed, reliable key-value store. It operates as a service similar to other services in Linux. To set it up, download the binaries, extract them, and run the service. ETCD runs on port 2379.
In Kubernetes, ETCD stores critical information about the cluster, including nodes, pods, configurations,
secrets, accounts, roles, and role bindings. All information retrieved using the kubectl get
command comes from the ETCD server. Any changes made to the cluster, such as adding nodes or deploying pods
and replica sets, are updated in the ETCD server.
If deploying your cluster from scratch, download the ETCD binaries and run the service.
ETCD is deployed in the cluster as a pod in the kube-system namespace.
To list all keys stored in ETCD, use the following command:
kubectl exec etcd-master -n kube-system -- etcdctl get / --prefix --keys-onlyETCDCTL is the CLI tool used to interact with ETCD. It supports two API versions: Version 2 and Version 3. By default, it uses Version 2, and each version has different sets of commands.
etcdctl backupetcdctl cluster-healthetcdctl mketcdctl mkdiretcdctl set
etcdctl snapshot saveetcdctl endpoint healthetcdctl getetcdctl put
To set the API version, use the following environment variable:
export ETCDCTL_API=3When the API version is not set, it defaults to Version 2. Version 3 commands will not work unless the API version is explicitly set to 3, and vice versa.
To authenticate ETCDCTL to the ETCD API Server, specify the path to the certificate files. These files are available in the etcd-master at the following paths:
--cacert /etc/kubernetes/pki/etcd/ca.crt--cert /etc/kubernetes/pki/etcd/server.crt--key /etc/kubernetes/pki/etcd/server.key
Here is an example command that sets the API version and specifies the certificate files:
kubectl exec etcd-controlplane -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl get / --prefix --keys-only --limit=10 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key"The Kube-API-Server is the primary management component in Kubernetes. It serves as the central communication hub for all components within the cluster.
-
When you run
kubectlcommands, thekubectlutility interacts with the Kube-API-Server. - The Kube-API-Server first authenticates and validates the incoming request. It then retrieves the required data from the ETCD cluster and responds with the requested information.
-
You don't necessarily need to use the
kubectlutility; you can directly interact with the API by sending POST requests.
The Kube-API-Server is the only component that communicates directly with the ETCD cluster.
The Kube-Controller Manager is responsible for monitoring and taking necessary actions on containers within the Kubernetes cluster.
- It continuously monitors the status of the system components and takes necessary actions to remediate any issues.
- In Kubernetes terms, a controller is a process that continuously monitors the state of various components within the system and works towards bringing the whole system to the desired functioning state.
-
Node Controller: Responsible for monitoring the status of the nodes and taking necessary actions to keep the application running. The Node Controller checks the status of the nodes every 5 seconds (heartbeat).
- The node monitor period grace is the duration that the Kubernetes Controller Manager waits before marking a node as unhealthy if it hasn't received a status update from the node. By default, this period is set to 40 seconds.
- The Node Controller gives the node 5 minutes to return before the workload on this node is evacuated to another node.
- Replication Controller: Responsible for monitoring the status of replica sets and ensuring that the desired number of pods are available at all times within the set. If a pod dies, it creates another one.
- There are many other controllers like the Deployment Controller, PV-Binder-Controller, Endpoint Controller, and Namespace-Controller.
All controllers are packaged into a single process known as the Kubernetes Controller Manager.
The Kube-Scheduler is responsible for scheduling pods on nodes within the Kubernetes cluster. It is tasked with determining which pod goes on which node, although it does not actually place the pod on the nodes. This task is performed by the Kubelet, which creates the pod on the node.
- Filter Nodes: The scheduler filters out nodes that do not have sufficient CPU and memory resources as requested by the pod.
- Rank Nodes: The scheduler ranks the remaining nodes to identify the best fit for the pod using a priority function. It calculates the amount of resources that would be free on the nodes after placing the pods on them.
The Kubelet is responsible for registering the node with the Kubernetes cluster. It plays a crucial role in managing and maintaining the state of pods and containers on each node.
- When the Kubelet receives instructions to manage a container or a pod on the node, it requests the container runtime engine (such as Docker or containerd) to pull the required image and run an instance.
- It continuously monitors the state of the pod and its containers and reports this status to the Kube-API-Server.
- The Kubelet must be downloaded and installed on each node, even when using the kubeadm tools.
To set up the Kubelet, download the binaries, extract them, and run the Kubelet as a service on each node.
The Kube-Proxy is a process that runs on each node within the Kubernetes cluster. Its primary responsibility is to manage network connectivity and routing for services.
- Kube-Proxy monitors the cluster for new services. Whenever a new service is created, it configures the appropriate rules on each node to ensure traffic is forwarded to the backend pods.
-
One method Kube-Proxy uses to achieve this is by creating
IPTABLESrules.
Containers are encapsulated into a Kubernetes object known as Pods. If you need to scale your application, you would create additional Pods. In a multi-container Pod, the containers are not of the same kind; these are called sidecar containers, responsible for performing supporting tasks for the main application.
Pods by default share the same storage, network namespace, and fate (they are created together and destroyed together).
When you run kubectl run nginx --image nginx, it will pull the nginx image from Docker Hub.
You can configure Kubernetes to pull images from Docker Hub or a private repository within your organization.
kubectl create -f file-name.ymlkubectl get podskubectl describe pod pod-name
The Replica Set is the newer technology that replaces the older Replication-Controller. It manages the replica sets and ensures that the desired number of pods are running within the Kubernetes cluster.
One key difference between the Replication Controller and the Replica Set is the use of the selector. The selector allows managing pods with this selector even if the pods exist outside of the replica set and were created before the replica set.
The Replica Set is a process that monitors the state of pods. It identifies which pods to monitor in the cluster based on the selector.
To scale in or out the number of pods in the replica set, change it in the yaml file and use the following command:
kubectl replace -f file-name.ymlIf you need to edit an existing running object in Kubernetes, use the following command:
kubectl edit object-type object-nameThis approach is useful when you don't have the running object's file and need to update it. After editing, delete the pods, and Kubernetes will recreate them with the updated configuration.
Kubernetes Deployments provide powerful capabilities, such as rolling updates, rollbacks, and upgrades, ensuring your applications run smoothly and stay up-to-date.
Deployments work in conjunction with Replica Sets, forming a layered approach. When you deploy a Deployment, it automatically manages the associated Replica Set. You can verify this by running the kubectl get replicaset command, which will display the Replica Set created by the Deployment.
For the exam, please use the following commands to create templates for pods or deployments:
kubectl run nginx --image=nginx --dry-run=client -o yamlkubectl create deployment --image=nginx nginx --dry-run=client -o yaml
A DaemonSet ensures that a copy of a Pod runs on each node in the Kubernetes cluster. DaemonSets are particularly useful for deploying system-level applications that need to run on all nodes.
- Runs one copy of your Pod on each node in your cluster.
- Automatically adds a replica of the Pod to any new node added to the cluster.
- Ensures that a copy of the Pod is always present on all nodes in the cluster.
Common use cases for DaemonSets include:
- Monitoring solutions
- Log viewers
- Network solutions
DaemonSets use node-affinity rules to schedule Pods on specific nodes.
The kubelet relies on the kube-apiserver for instructions on creating nodes. But what if there is no kube-apiserver? What if there is no master or cluster at all? In such cases, the kubelet can manage its node independently. The kubelet knows how to create pods, even without a kube-apiserver.
You can configure the kubelet to read the pod definition files from a designated directory, such as /etc/kubernetes/manifests. The kubelet periodically checks this directory for files, reads them, creates the pods, and monitors them. If a pod crashes, the kubelet will recreate it. Removing the definition file from the directory will automatically delete the pod.
These pods, created by the kubelet on its own without intervention from the API server, are called static pods. You cannot create deployments or other Kubernetes objects with static pods; only pods are supported.
If there is a Kubernetes cluster with static pods, running kubectl get pods will show a read-only mirror of the pod. You can describe it, but you cannot delete or edit it.
Static pods are used to deploy master components. This is how tools like kubeadm set up clusters. Static pods appear in the cluster with the node name appended.
The kubelet runs on the controller node as a service. You can check for options of the service using ps -aux or by searching for static pod options in the config file.
-
Identify the node where the static pod is created:
kubectl get pods --all-namespaces -o wide | grep static-greenbox -
SSH to the node and identify the path configured for static pods in the kubelet configuration file:
In this example, the staticPodPath is
ssh node01 ps -ef | grep /usr/bin/kubelet grep -i staticPod /var/lib/kubelet/config.yaml/etc/just-to-mess-with-you. -
Navigate to this directory and delete the YAML file:
rm -rf /etc/just-to-mess-with-you/greenbox.yaml -
Exit the node and check if the static-greenbox pod has been deleted:
kubectl get pods --all-namespaces -o wide | grep static-greenbox
When pods are created, they wait in a scheduling queue until they are scheduled. This queue has priorities, which can be set in the pod YAML file. First, you need to create a priority class.
Kubernetes Services provide a virtual IP for stable communication between components in an application. When a pod is destroyed and recreated, its IP address changes. To ensure consistent communication, Kubernetes creates an object called a service.
-
NodePort Service: This service listens to a port on the node and forwards requests to the pods. It provides external access to the application but is not recommended for production environments.
- Port on the pod (target port)
- Port on the service (port) with its own IP address, called the cluster IP
- Port on the node, which should be within the range 30,000 to 32,767
Labels and selectors are used to link services to the targeted pods. Each node in the cluster exposes the NodePort, allowing access to the pods from any node using its IP.
- Cluster IP Service: This service creates a virtual IP inside the cluster, enabling communication between services such as frontend, backend, and database servers.
- LoadBalancer Service: This service provisions a load balancer for applications in cloud providers, distributing load across different web servers. Even if your apps are deployed on just two nodes, they can be accessed using all nodes in the cluster. To achieve a single URL for end users, a cloud-native load balancer is recommended.
The default service created when Kubernetes is launched is:
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 5m17sIngress in Kubernetes helps your users access your application using a single externally accessible URL. It allows you to configure routing to different services within your cluster based on the URL path, and it also supports SSL security.
Think of Ingress as a layer seven load balancer built into the Kubernetes cluster. It provides advanced routing capabilities based on HTTP/HTTPS requests.
An Ingress Controller is like a load balancer that users can access. There are several solutions available, such as NGINX, Istio, HAProxy, and Traefik. These controllers have additional intelligence built into them to monitor the Kubernetes cluster for new definitions or ingress resources.
We deploy the Ingress Controller as a deployment in the Kubernetes cluster. These solutions are not deployed by default in the Kubernetes cluster and must be deployed manually.
Namespaces are an isolation technique used to isolate projects from each other within a Kubernetes cluster.
- Default
- Kube-system
- Kube-public
When a web application needs to access a database in another namespace, use the following format:
Mysql.connect("db-service.dev.svc.cluster.local")
Format: Service-name.Namespace.Service.Domain-name-of-the-cluster
To create a namespace, use the following command:
kubectl create namespace devIf you need to work in a specific namespace permanently, switch to it and run this command:
kubectl config set-context $(kubectl config current-context) --namespace=devTo limit resources in a namespace, create a resource quota using the following configuration:
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-quota
namespace: dev
spec:
hard:
pods: "10"
requests.cpu: "4"
requests.memory: 5Gi
limits.cpu: "10"
limits.memory: 10Gi
To switch the namespace, use the following command:
kubectl config set-context --current --namespace=K21The imperative approach involves issuing commands to achieve a desired state. This approach is akin to giving direct instructions on what to do.
Example: Running commands like kubectl run or kubectl create to create resources.
- Simple and straightforward for quick tasks.
- Allows for fine-grained control and immediate feedback.
- Can become cumbersome and error-prone for complex environments.
- Not easily repeatable or version-controlled.
The declarative approach involves specifying the desired state of the system, and the system is responsible for achieving that state. This approach is akin to declaring what the end result should look like, rather than how to achieve it.
Example: Using YAML files to define Kubernetes resources and applying them with kubectl apply -f .yaml.
- Easier to manage and maintain complex configurations.
- Changes are version-controlled and more predictable.
- Better suited for automated workflows and continuous integration/continuous deployment (CI/CD) pipelines.
- Initial setup and learning curve might be higher.
- Requires tools and infrastructure to manage the configuration files.
The Kubernetes scheduler is responsible for placing Pods onto Nodes. It watches for newly created Pods that have no Node assigned. For each Pod, it finds the best Node to run it on.
Kubernetes comes with a default scheduler called kube-scheduler. It selects an optimal node based on various factors like resource requirements, hardware/software constraints, and affinity/anti-affinity specifications.
The scheduler performs two main steps:
- Filtering: Finds Nodes where it's feasible to schedule the Pod.
- Scoring: Ranks these Nodes to pick the best one.
You can implement your own scheduler if the default one doesn't meet your needs. Multiple schedulers can run simultaneously.
If the Pod status is pending, there might be a problem scheduling the Pod on a specific Node. In such cases, you can manually specify the node using nodeName: name-of-the-node in the spec field.
Labels and selectors are mechanisms used to bind resources together in Kubernetes.
Example: You can get Pods with --selector app=App1.
Taints on Nodes: Taints prevent unwanted Pods from landing on the node unless the Pod has a matching toleration.
Taints and tolerations don't tell the Pod to go to a particular node; they tell the node to only accept Pods with certain tolerations.
If you need to restrict a Pod from landing on a particular node, use the concept of node and pod affinity.
To see the taints, use this command:
kubectl describe node kubemaster | grep TaintTo delete a taint from a node, use this command:
kubectl taint nodes controlplane node-role.kubernetes.io/control-plane:NoSchedule-Pod Definition with Toleration:
apiVersion: v1 kind: Pod metadata: name: myapp-pod spec: containers: - name: nginx-container image: nginx tolerations: - key: "app" operator: "Equal" value: "blue" effect: "NoSchedule"
Node selectors are used to host a Pod on a specific node.
Limitations exist when using node selectors and labels, such as needing to host your Pod on a large or medium node and not a small node. For this, you use node affinity.
Node affinity ensures that Pods are hosted on specific nodes based on criteria.
The Exists operator doesn't need a value.
In fact, you often use a combination of node affinity and taints/tolerations to ensure your node hosts your Pods only.
Resource requests are a way to specify the minimum amount of CPU and memory resources that a container in a Pod should have. Here’s a bit more detail:
Resource requests ensure that a container has the minimum amount of resources it needs to run efficiently. They are used by the Kubernetes scheduler to decide which node to place the Pod on.
The minimum amount of CPU a container requires. It is measured in CPU units. One CPU, in Kubernetes, is equivalent to 1 vCPU/Core for cloud providers, 1 hyperthread on bare-metal Intel processors, or 1 AWS vCPU.
The minimum amount of memory a container needs, measured in bytes (MiB, GiB).
Resource requests are defined in the Pod or container specification using the resources field in a YAML file. Here’s an example:
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
containers:
- name: example-container
image: nginx
resources:
requests:
memory: "64Mi"
cpu: "250m"
In this example:
- The container requests a minimum of 64 MiB of memory.
- The container requests a minimum of 250 milliCPUs (0.25 CPU).
Kubernetes uses resource requests to schedule Pods onto nodes that have enough available resources.
Pods are classified into different Quality of Service (QoS) classes based on their resource requests and limits. This classification affects the scheduling and eviction behavior of the Pods.
By default, in Kubernetes, there are no restrictions on using CPU or memory. When you use requests, you allocate the requests for the Pod, even if the Pod doesn't really need it or consume it.
A LimitRange sets default values for resource requests and limits if they are not specified by the user. It ensures that all pods and containers have defined resource boundaries, promoting better resource utilization and avoiding resource contention.
Applied at the namespace level. Each namespace can have its own LimitRange.
- Minimum and Maximum Requests: Specifies the minimum and maximum CPU and memory that a container can request.
- Default Requests and Limits: Sets default values for CPU and memory requests and limits if they are not specified.
- Resource Limits: Ensures that no container can exceed these limits for CPU and memory.
Here's an example of how to define a LimitRange in a YAML file:
apiVersion: v1
kind: LimitRange
metadata:
name: example-limitrange
namespace: example-namespace
spec:
limits:
- max:
cpu: "2"
memory: "1Gi"
min:
cpu: "200m"
memory: "100Mi"
default:
cpu: "500m"
memory: "200Mi"
defaultRequest:
cpu: "300m"
memory: "100Mi"
type: Container
In this example:
- max: The maximum CPU is set to 2 CPUs and memory to 1 GiB.
- min: The minimum CPU is set to 200 milliCPUs and memory to 100 MiB.
- default: Sets the default CPU request to 500 milliCPUs and memory to 200 MiB.
- defaultRequest: Sets the default CPU request to 300 milliCPUs and memory to 100 MiB.
- Resource Efficiency: Helps in efficient utilization of resources by ensuring that every pod/container has an appropriate amount of resources allocated.
- Stability: Prevents a single pod/container from consuming excessive resources, which can impact the performance of other workloads in the namespace.
- Consistency: Provides a consistent environment by setting default values for resource requests and limits.
To create a LimitRange, use the following command:
kubectl create -f <filename>.yamlTo view LimitRanges in a namespace, use:
kubectl get limitrange -n <namespace>By using LimitRange, you can ensure that your Kubernetes clusters run smoothly, with balanced resource allocation and utilization.
Monitoring involves tracking node-level metrics such as the number of nodes in the cluster, node health, CPU, memory, network, and disk utilization.
One of the most popular solutions for monitoring a Kubernetes cluster is the metrics server. The metrics server is an in-memory solution that does not store metrics on disk. If you need to monitor historical data, you will need an advanced solution.
The Kubelet also contains a subcomponent known as cAdvisor (Container Advisor). cAdvisor is responsible for retrieving performance metrics from pods and exposing them through the Kubelet API to make the metrics available to the metrics server.
Download the metrics server and deploy it as a deployment. Use the following commands to view metrics:
kubectl top nodekubectl top podTo see the logs of a pod, use the following command:
kubectl logs -f pod-nameRollout refers to the process of deploying updates to your applications.
- When you deploy your application, it creates revision 1 of the app.
- When you update the app, it creates revision 2 of the app.
- These revisions help us to roll back to a previous version of deployment if necessary.
To see the rollout status, run the following command:
kubectl rollout status deployment/myapp-deploymentTo see the revision history, run the following command:
kubectl rollout history deployment/myapp-deployment- Recreate Strategy: Destroy all the pods and run the new version of the pods. The application is down during the upgrade.
- Rolling Update: Destroy one pod and create another one (update the pods one by one). This is the default deployment strategy.
To roll back your updated deployment, run the following command:
kubectl rollout undo deployment/myapp-deployment
To force replace a deployment, run the following command:
kubectl replace --force -f ubuntu-sleeper-3.yaml
Environment variables (ENV Vars) in Kubernetes are used to pass configuration data to the containers running in a Pod. They allow you to customize the behavior of applications without changing the code. Here’s a detailed overview:
You can define environment variables in your Pod or container specifications using the env field in a YAML file. Here's an example:
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
containers:
- name: example-container
image: nginx
env:
- name: ENV_VAR_NAME
value: "value"
In this example:
- The
ENV_VAR_NAMEis set to "value" in the container running in theexample-pod.
These are defined with a fixed value directly in the Pod specification, as shown in the example above.
These can reference values from Kubernetes resources like ConfigMaps, Secrets, or downward APIs.
Example with ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: example-configmap
data:
CONFIG_VAR: "config-value"
---
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
containers:
- name: example-container
image: nginx
env:
- name: CONFIG_VAR
valueFrom:
configMapKeyRef:
name: example-configmap
key: CONFIG_VAR
To store sensitive information, you can use Kubernetes Secrets:
apiVersion: v1
kind: Secret
metadata:
name: example-secret
type: Opaque
data:
SECRET_KEY: c2VjcmV0LXZhbHVl # base64 encoded value
---
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
containers:
- name: example-container
image: nginx
env:
- name: SECRET_KEY
valueFrom:
secretKeyRef:
name: example-secret
key: SECRET_KEY
The Downward API allows you to expose Pod and container fields as environment variables:
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
containers:
- name: example-container
image: nginx
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
In this example, the environment variable POD_NAME will hold the name of the Pod.
- Use ConfigMaps and Secrets: For better manageability and security, store non-sensitive and sensitive data in ConfigMaps and Secrets.
- Keep it Simple: Avoid complex logic in environment variables; use them for simple configurations.
- Base64 Encoding: Remember to encode sensitive values in Secrets using base64.
ConfigMaps in Kubernetes are used to store configuration data in key-value pairs. This allows you to decouple configuration artifacts from image content to keep containerized applications portable. Here's a detailed overview:
You can define a ConfigMap using a YAML file. Here’s an example:
apiVersion: v1
kind: ConfigMap
metadata:
name: example-configmap
data:
CONFIG_KEY: "config-value"
ANOTHER_KEY: "another-value"
In this example, example-configmap contains two keys, CONFIG_KEY and ANOTHER_KEY, with their respective values.
To use a ConfigMap in a Pod, you reference the ConfigMap in the Pod’s specification. Here’s an example:
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
containers:
- name: example-container
image: nginx
env:
- name: CONFIG_KEY
valueFrom:
configMapKeyRef:
name: example-configmap
key: CONFIG_KEY
In this example:
- The environment variable
CONFIG_KEYin the container is populated with the value from theexample-configmap.
ConfigMaps can also be mounted as volumes to provide configuration files to containers. Here’s an example:
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
containers:
- name: example-container
image: nginx
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
name: example-configmap
In this example:
- The
example-configmapis mounted at/etc/configinside the container.
- Separate Configuration: Store configuration data separately from application code to make applications portable.
- Version Control: Store ConfigMaps in version control systems (e.g., Git) to keep track of changes.
- Secure Sensitive Data: Use Secrets instead of ConfigMaps for sensitive data.
Secrets in Kubernetes are used to store sensitive information such as passwords, OAuth tokens, and SSH keys. Unlike ConfigMaps, which are intended for non-sensitive data, Secrets ensure that sensitive data is stored securely and is only accessible to authorized Pods.
You can define a Secret using a YAML file. Here’s an example:
apiVersion: v1
kind: Secret
metadata:
name: example-secret
type: Opaque
data:
USERNAME: dXNlcm5hbWU= # base64 encoded value
PASSWORD: cGFzc3dvcmQ= # base64 encoded value
In this example, example-secret contains two keys, USERNAME and PASSWORD, with their respective base64 encoded values.
To use a Secret in a Pod, you reference the Secret in the Pod’s specification. Here’s an example:
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
containers:
- name: example-container
image: nginx
env:
- name: USERNAME
valueFrom:
secretKeyRef:
name: example-secret
key: USERNAME
- name: PASSWORD
valueFrom:
secretKeyRef:
name: example-secret
key: PASSWORD
In this example:
- The environment variables
USERNAMEandPASSWORDin the container are populated with the values from theexample-secret.
Secrets can also be mounted as volumes to provide sensitive files to containers. Here’s an example:
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
containers:
- name: example-container
image: nginx
volumeMounts:
- name: secret-volume
mountPath: /etc/secret
volumes:
- name: secret-volume
secret:
secretName: example-secret
In this example:
- The
example-secretis mounted at/etc/secretinside the container.
- Base64 Encoding: Ensure sensitive values in Secrets are base64 encoded.
- RBAC Policies: Use Kubernetes Role-Based Access Control (RBAC) to control access to Secrets.
- Encryption: Enable encryption at rest for Secrets in the Kubernetes cluster.
- Data Storage: Secrets store data in an encoded format.
To view the details of a secret in YAML format, use the following command:
kubectl get secret app-secret -o yamlNote that Secrets are not encrypted, only encoded. To encrypt the secrets, use an object called EncryptionConfiguration.
Anyone able to create Pods/Deployments in the same namespace can access the secrets.
In Kubernetes, it is possible to have pods that contain multiple containers. These multi-container pods share the same lifecycle, meaning they are created together and destroyed together. They also share the same storage and network namespace.
For example, a multi-container pod may consist of a web container and a log container. The containers within the pod can communicate with each other using localhost and the port number on which the service is running.
If a node goes down in the cluster, the following actions occur:
- If the node comes back online, the pods on it will also come back online.
- If the node does not come back within 5 minutes, the pods are terminated. If the pods are part of a deployment or replicaset, they will be recreated on other nodes.
- When the node comes back online, it will start without any pods scheduled on it.
To evacuate a node from the workload for maintenance purposes, use the following command:
kubectl drain node-1This command gracefully terminates the pods on the node and recreates them on another node. The node is also marked as unschedulable.
When the node comes back online, run the following command to make it schedulable again:
kubectl uncordon node-1To make a node unschedulable, preventing new pods from being scheduled on it, use the following command:
kubectl cordon node-1Kubernetes versions follow the pattern V1.11.3, where:
- Major Version: Significant changes and new features (e.g., V1).
- Minor Version: Introduces new features and functionality, released every few months (e.g., 11).
- Patch Version: Fixes bugs and issues, released more frequently (e.g., 3).
It is not mandatory to have the same version for all Kubernetes components in the cluster. However, no component should have a version higher than the kube-apiserver.
Only the last three versions of Kubernetes are supported.
Upgrading a cluster involves two major steps:
- Update the master nodes.
- Update the worker nodes.
If the master node fails, management functions like delete, modify, and deploy new apps are unavailable. However, the workload remains available to users. If a pod in a deployment goes down, it will not be recreated automatically.
The nodes will not be upgraded until you manually update the kubelet service on each node. Restarting the service will update the versions on the nodes.
Before upgrading a cluster, ensure that the repository and key are set correctly. Refer to the Kubernetes documentation for detailed instructions.
To get a backup for all resources, use the following command:
kubectl get all --all-namespaces -o yaml > all-deploy-services.yamlTo take a snapshot of the ETCD cluster, you can use the etcdctl utility. The API uses https://127.0.0.1:2379 to access the ETCD database.
Before performing certain operations, you may need to stop the API service.
To restore the ETCD cluster:
- Run the
etcdctlcommand as specified in the documentation. - Configure the ETCD service to use the new directory.
- Restart the daemons and services, then start the kube-apiserver service.
Make sure to use the appropriate options when using etcdctl.
To restore ETCD:
- First, stop the kube-apiserver.
- Move the kube-apiserver manifest from the Static Pod directory to another location. (To find the Static Pod directory, use
ps -aux | grep kubeletand check the options.) - Run the
etcdctlcommand and create a directory as specified in the documentation. - Update the
etcd.yamlfile with the new directory.
To check the kubectl configuration file in the node, use:
kubectl config viewTo switch contexts in Kubernetes, use:
kubectl config use-context <name-of-the-context>- Stacked ETCD: ETCD is stacked with Kubernetes components within the cluster.
- External ETCD: Configure the ETCD database on a separate server.
The first line of defense in Kubernetes security is controlling access to the API server. This involves:
- Authentication: Determining who can access the server.
- Authorization (RBAC): Determining what actions they can perform.
By default, all pods can communicate with each other within the cluster. To prevent unauthorized communication, use network policies.
Kubernetes does not manage service accounts directly. Instead, it relies on external sources such as files with user details or certificates.
All user access is managed by the kube-apiserver, whether accessed via the kubectl tool or directly through the API (e.g., curl https://kube-server-ip:6443). The kube-apiserver first authenticates the request and then processes it.
Authentication mechanisms include static password files, certificates, or identity services. For example, static passwords can be set up with the following format: password123,user4,user-id. The file name is then passed as an option to the kube-apiserver using --basic-auth-file=file-name.
Certificates are a key component of Kubernetes security:
- CA (Certificate Authority): Has a public key (built into browsers) and a private key (used to sign server certificates).
- Server: Generates a public key (sent to users to encrypt symmetric keys) and a private key (used to decrypt symmetric keys).
- User: Generates a symmetric key used for ongoing communication.
Public and private keys are related; one can encrypt data and only the corresponding key can decrypt it. Note that if data is encrypted with the private key, anyone with the public key can decrypt it.
- Public Key: .cert, .pem
- Private Key: .key, .key.pem
- Generate the CA key:
openssl genrsa -out ca.key 2048 - Create a certificate signing request (CSR):
openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr - Generate the CA certificate:
openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt
- Generate the admin user key:
openssl genrsa -out admin.key 2048 - Create a CSR for the admin user:
openssl req -new -key admin.key -subj "/CN=kube-admin" -out admin.csr - Sign the admin user certificate with the CA:
openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -out admin.crt - Optionally, create a CSR with an organization:
openssl req -new -key admin.key -subj "/CN=kube-admin/O=system.masters" -out admin.csr
When using the options --key, --cert, and --cacert with etcdctl, they are used to authenticate requests with ETCD.
To show a specific certificate, use the following command:
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -nooutIf there is a problem with the kube-apiserver pod, check the container log and the ETCD container as the API server gets its data from the ETCD database.
Kubernetes has a built-in Certificates API that manages rotating expired certificates and signing new certificates.
Create a CertificateSigningRequest object and import the CSR encoded with base64. You can view the CSR as an object in Kubernetes using:
kubectl get csrApprove the CSR using:
kubectl certificate approve <name-of-the-csr>To get the YAML file of any object when you don't know where it is, use the following command:
kubectl get [object] [name-of-the-object] -o yamlThe KubeConfig file is used to configure access to the Kubernetes API server. It allows you to specify the necessary options for the kubectl utility to interact with the API server without having to provide those options in every command.
When using kubectl to request actions from the Kubernetes API server, you typically need to specify options such as --server, --client-key, --client-certificate, and --certificate-authority. By setting these options in the KubeConfig file, you can use kubectl without specifying them explicitly each time.
By default, Kubernetes looks for the KubeConfig file in $HOME/.kube/config. If you create the file in this location, you do not need to specify the path to the file explicitly in the kubectl command.
To use a specific context as the default, set the current-context in the KubeConfig file:
current-context: dev-user@ggoggoleTo view the KubeConfig file, use the following command:
kubectl config viewIn Kubernetes, Roles define the permissions to perform specific actions on resources, while RoleBindings link users or groups to the roles created.
Roles specify the objects and the actions that can be performed on them. To view roles in your cluster, use the following command:
kubectl get rolesRoleBindings associate users or groups with the defined roles, effectively granting them the specified permissions. To view role bindings in your cluster, use the following command:
kubectl get rolebindings
To get detailed information about a specific role, use the following command:
kubectl describe role developerTo check whether a specific action on an object is permitted, use the following commands:
kubectl auth can-i create deployments # yes
kubectl auth can-i delete nodes # no
To specify access permissions for a specific user, use the appropriate kubectl auth commands to verify their permissions.
To specify access to a specific object name, ensure you define roles and role bindings that grant the required permissions to the user or group.
Cluster Roles and ClusterRoleBindings in Kubernetes function similarly to Roles and RoleBindings but apply to cluster-wide permissions. This means they grant access to resources across the entire cluster, not just within a specific namespace.
Cluster Roles define the permissions to perform specific actions on cluster-scoped resources. These roles can also be created for namespace-scoped resources, granting the user access to all resources in all namespaces.
ClusterRoleBindings associate users or groups with Cluster Roles, effectively granting them the specified cluster-wide permissions.
A Service Account in Kubernetes is an account used by an application to interact with a Kubernetes cluster. It allows the application to authenticate with the API server and access resources within the cluster.
When you create a Service Account using the following command, a token is automatically generated:
kubectl create serviceaccount <name-of-it>In Kubernetes, images are often stored in private repositories to enhance security and control access. Here’s how you can use images from private repositories in your Kubernetes cluster:
Create a Docker registry secret that contains the credentials required to access the private repository. This can be done using the following command:
kubectl create secret docker-registry myregistrykey --docker-server=<DOCKER_REGISTRY_SERVER> --docker-username=<USERNAME> --docker-password=<PASSWORD> --docker-email=<EMAIL>
- <DOCKER_REGISTRY_SERVER> is the URL of your private Docker registry.
- <USERNAME> is your Docker registry username.
- <PASSWORD> is your Docker registry password.
- <EMAIL> is your Docker registry email.
Include the secret in your pod specification under the imagePullSecrets section:
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: mycontainer
image: <DOCKER_REGISTRY_SERVER>/<IMAGE_NAME>:<TAG>
imagePullSecrets:
- name: myregistrykey
<DOCKER_REGISTRY_SERVER>/<IMAGE_NAME>:<TAG> is the image path and tag in your private repository.
This configuration ensures that Kubernetes will use the provided credentials to pull the image from the private repository.
Deploy the pod and verify that it is able to pull the image from the private repository without any issues.
Network Policies in Kubernetes are used to control the communication between pods. By default, all pods can communicate with each other within the same cluster. Network Policies allow you to specify how groups of pods are allowed to communicate with each other and other network endpoints.
A Network Policy is defined using a YAML file. Here’s an example of a simple Network Policy:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: example-network-policy
namespace: default
spec:
podSelector:
matchLabels:
role: db
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
role: frontend
egress:
- to:
- podSelector:
matchLabels:
role: frontend
In this example:
- The Network Policy named
example-network-policyapplies to thedefaultnamespace. - It selects pods with the label
role: db. - It specifies both ingress and egress rules.
- Only pods with the label
role: frontendcan communicate with the selected pods.
To apply a Network Policy, save the YAML file and use the following command:
kubectl apply -f example-network-policy.yaml- Ingress Policy: Controls incoming traffic to the pods.
- Egress Policy: Controls outgoing traffic from the pods.
- Least Privilege Principle: Apply the principle of least privilege by allowing only necessary communication between pods.
- Test Policies: Test network policies in a staging environment before applying them in production.
- Monitor Traffic: Use monitoring tools to ensure that the network policies are working as expected and not causing unintended disruptions.
The API Server can be configured in an active-active status across nodes. A front-line load balancer can be set up to point to the three API servers, distributing the traffic among them.
The Controller Manager and Scheduler are configured in an active-standby status. Through an election process, only one instance is active at a time.
- When reading from the ETCD cluster, you can read from any node in the cluster.
- For writing data, only one node is responsible for write operations. The nodes elect a leader to manage the writes, while the other nodes act as followers.
- The leader ensures that the writes are distributed to other instances in the cluster. A write operation is only considered complete if the leader gets consent from other members in the cluster.
- The nodes use the RAFT protocol for leader election, which employs random timers for initiating requests.
- Quorum is the minimum number of nodes needed in the cluster to function properly or make a successful write. For a cluster of 3 nodes, the quorum is 2, calculated as Q = N/2 + 1.
Storage in Kubernetes is designed to provide persistent and temporary storage solutions for applications running in the cluster. Here’s an overview of the different storage options available in Kubernetes:
Persistent Volumes are storage resources in the cluster that are provisioned by an administrator. They exist independently of pods and provide a way for users to request storage that persists beyond the lifecycle of individual pods.
apiVersion: v1
kind: PersistentVolume
metadata:
name: my-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /mnt/data
Persistent Volume Claims are requests for storage by a user. They specify the desired capacity and access modes. When a PVC is created, Kubernetes finds a suitable PV to bind it to.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
Dynamic provisioning allows Kubernetes to automatically create a PV for a PVC, eliminating the need for an administrator to manually provision storage. This is achieved using StorageClasses.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: my-storage-class
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-dynamic-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: my-storage-class
Volumes are used to store data within a pod. Kubernetes supports several types of volumes, such as emptyDir, hostPath, and more.
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
containers:
- name: my-container
image: nginx
volumeMounts:
- mountPath: /usr/share/nginx/html
name: my-volume
volumes:
- name: my-volume
emptyDir: {}
Access Modes define how the PV can be mounted and accessed by pods. Common access modes include:
- ReadWriteOnce (RWO): The volume can be mounted as read-write by a single node.
- ReadOnlyMany (ROX): The volume can be mounted as read-only by many nodes.
- ReadWriteMany (RWX): The volume can be mounted as read-write by many nodes.
- Use PVCs: Always use Persistent Volume Claims to request storage rather than directly specifying PVs.
- Dynamic Provisioning: Leverage dynamic provisioning to simplify storage management.
- Backup: Regularly back up your persistent data to avoid data loss.
- CRI (Container Runtime Interface): The standard that defines how Kubernetes communicates with container runtime engines like Docker or CRI.
- CNI (Container Network Interface): Extends support for different networking solutions.
- CSI (Container Storage Interface): Enables Kubernetes to work with different storage solutions.
A pod can be mounted to a specific file directory on a node using hostPath. However, it is not recommended in multi-node clusters because if the pod is recreated on another node, the data in the container will not be the same.
The administrator creates a set of persistent volumes, and developers create persistent volume claims to use the storage.
Once you create a PVC, use it in a pod definition file by specifying the PVC claim name under the persistentVolumeClaim section in the volumes section, like this:
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
containers:
- name: my-container
image: nginx
volumeMounts:
- mountPath: /usr/share/nginx/html
name: my-volume
volumes:
- name: my-volume
persistentVolumeClaim:
claimName: my-pvc
Access mode is very important to ensure the PVC matches the PV. When you delete a PVC used by a pod, the pod will be stuck, and after you delete the pod, it will be deleted.
- Connection Refused: Check the user and password of the database and the deployment.
- Connection Refused: Verify the ports of the service and the pods.
- Access Denied: Ensure the database user is correctly configured.
- Error on Gateway: Check the port of the node in the service deployment file.
- Ensure the file exists.
- Check the permissions on the file.
- Verify the mounts in the pod YAML (host paths).
- Use
topanddfto check resource usage. - Check the status of the kubelet.
- Use
journalctl -u kubeletto review logs. - Verify the certificates are not expired and are issued by the correct CA using:
openssl x509 /var/lib/kubelet/worker01.crt -text - Check
/var/lib/kubelet/config.yamlfor configuration issues. - Ensure the port of the control plane is correct in
/etc/kubernetes/kubelet.conf(should be 6443).