diff --git a/ConfigMaps101/what-are-configmaps.md b/ConfigMaps101/what-are-configmaps.md new file mode 100644 index 00000000..0c4dd113 --- /dev/null +++ b/ConfigMaps101/what-are-configmaps.md @@ -0,0 +1,199 @@ +# ConfigMaps + +ConfigMaps are a fundamental part of any Kubernetes infrastructure. You are unlikely to find any major Kubernetes applications, be it a library, support infrastructure, or cloud provider that doesn't rely on ConfigMaps. So what are they? + +[ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/) is simply a key-value store that keeps your data in plain text. This is in contrast to [secrets](../secerts_configmaps101/secrets-configmaps.md) which also do the same thing, but also encrypt your data. As the name implies, you would generally use ConfigMaps to store configurations that would be separate from your actual application. This way, if you wanted to change a configuration (such as an API endpoint your pod was calling), you could do it by simply changing the ConfigMap without having to change the actual application or deployment yaml, thereby removing the need to re-deploy your application once again. + +ConfigMaps needs to have a structure that specifies which type of resource it is. For example, the ConfigMap for a MongoDB object would look like this: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: my-configmap + namespace: my-namespace +data: + # Key-value pairs representing configuration data + DATABASE_URL: "mongodb://localhost:27017/mydatabase" + API_KEY: "your-api-key-here" + LOG_LEVEL: "info" +``` + +As you can see, the `kind: ConfigMap` is specified here and it follows a general Kubernetes resource pattern. However, unlike most other Kubernetes objects, there is a certain amount of flexibility allowed in this resource, where you could define all sorts of things inside the `data` attribute. For example, if you have a MySQL database and you want a SQL command to run when the database starts, you could have that SQL command as a value in the ConfigMap: + +``` +data: + initdb.sql: |- + INSERT INTO users (username, email) VALUES ('john_doe', 'john.doe@example.com'); +``` + +There are numerous examples of different file structures being used within ConfigMaps across Kubelabs. For example, if you look at the [logging section](../Logging101/fluentd.md), you will see that the fluentd configuration file gets defined in a ConfigMaps with its own format. Meanwhile, if you wanted to add a user to an EKS cluster, you would do so by adding the user in a YAML format into the aws_auth ConfigMap. In this manner, just about every type of file can be defined within a ConfigMap. + +So now you have a pretty good idea of what ConfigMaps are and what they are capable of, as well as knowledge of how you define them. So let's move on to how you would use a ConfigMap that has already been defined. As a resource that holds multiple values, there are several ways you could use the resource in a pod. One of the most popular methods is to add the ConfigMap as a volume mount. In the deployment yaml, simply specify the name of the ConfigMap as a volume, and your pod would have access to the ConfigMap the same way it would have access to any other volume. Another method of accessing the ConfigMap is to pass it as a command line argument. This is useful if you have multiple ConfigMaps, and which one you want available to your pod gets decided at runtime. You could also have the ConfigMap placed as an environment variable for a container and get it to read the environment variable and subsequently the value from the ConfigMap. + +Now that you have a pretty good idea of the flexibility ConfigMaps provide, let's take a look at a full example. We will begin by defining a ConfigMap and then proceed to show all the different ways the ConfigMap can be used in a pod. We will use the same MongoDB example from before: + +``` +# configmap.yaml + +apiVersion: v1 +kind: ConfigMap +metadata: + name: example-configmap +data: + # Key-value pairs for configuration data + DATABASE_URL: "mongodb://mongo-server:27017/mydatabase" + API_KEY: "your_api_key_here" + DEBUG_MODE: "true" +``` + +You can see the ConfigMap defined with the database URL, API key, and debug mode turned on. As with all Kubernetes resources, you will have to deploy this file into your cluster: + +``` +kubectl apply -f configmap.yaml +``` + +Finally, let's reference it in a sample deployment: + +``` +# pod.yaml + +apiVersion: v1 +kind: Pod +metadata: + name: example-pod +spec: + containers: + - name: my-container + image: your-image + env: + - name: DATABASE_URL + valueFrom: + configMapKeyRef: + name: example-configmap + key: DATABASE_URL + - name: API_KEY + valueFrom: + configMapKeyRef: + name: example-configmap + key: API_KEY + - name: DEBUG_MODE + valueFrom: + configMapKeyRef: + name: example-configmap + key: DEBUG_MODE +``` + +In this case, `valueFrom` is used to reference the values from the ConfigMap for each environment variable. Therefore, we are referencing the ConfigMap as env variables. Inside the `valueFrom` key, we also provide the `configMapKeyRef`, with the name of the key being passed in so that the value can be retrieved as a parameter. Since here we only have 3 variables, it wasn't really complicated. However, in a case where there were a large number of vars, you can imagine the deployment yaml would end up getting very repetitive and long, which would eventually make it unreadable. So, let's look at how you can mount the ConfigMap as a volume: + +``` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: my-app +spec: + replicas: 1 + selector: + matchLabels: + app: my-app + template: + metadata: + labels: + app: my-app + spec: + containers: + - name: my-container + image: my-nodejs-app:latest + env: + - name: MONGODB_CONFIG_FILE + value: "/etc/mongodb-config/mongodb-configmap.properties" + volumeMounts: + - name: config-volume + mountPath: "/etc/mongodb-config" + volumes: + - name: config-volume + configMap: + name: mongodb-configmap + +``` + +In this example, the `MONGODB_CONFIG_FILE` environment variable is set to the path of the file containing the entire ConfigMap. This file will be mounted into the container. A volume is defined with the name `config-volume` and is associated with the ConfigMap `mongodb-configmap`. This volume is mounted into the container at the path `/etc/mongodb-config`. This way, there will be no repetition and all of the environment variables would still be accessible. + +You could also safely remove the `env` key and still have all the data in the ConfigMap accessible: + +``` +containers: +- name: mongodb + image: mongo:latest + ports: + - containerPort: 27017 + volumeMounts: + - name: mongodb-config-volume + mountPath: /etc/mongod +volumes: +- name: mongodb-config-volume + configMap: + name: mongodb-config +``` + +The above method will make the ConfigMap be considered as a volume instead of having it entered as an environment variable. This means that if you were to update the ConfigMap while the container was running, the container would take the new values immediately. On the other hand, if you had loaded them up as environment variables, you would have to restart the container since the variables are assigned during pod startup. Or you could directly inject the values from the ConfigMap into your container environment using the `envFrom` key: + +``` +containers: +- name: mongodb + image: mongo:latest + ports: + - containerPort: 27017 + envFrom: + - configMapRef: + name: mongodb-config +``` + +In this version, the envFrom field is used to directly reference the ConfigMap. This will inject all the key-value pairs from the ConfigMap as environment variables into the container. + +You can also load the ConfigMap as a command line argument in a deployment yaml. You do this using the `command` key. For example: + +``` +containers: + - name: mongodb + command: ["mongo", "--database-host", "$(DATABASE_URL)"] + image: mongo:latest + env: + - name: DATABASE_URL + valueFrom: + configMapKeyRef: + name: example-configmap + key: DATABASE_URL +``` + +This command runs when the pod starts, meaning that the value will be extracted from the ConfigMap at this point. + +Now that we have looked at various ways we can reference an existing ConfigMap, let's look at all the ways we can create a ConfigMap. The first method is to declare it in a yaml file, which is the example that has been used so far. However, there is also a way to define it in line with a `kubectl create configmap` command: + +``` +kubectl create configmap example-configmap –from-literal=DATABASE_URL="mongodb://mongo-server:27017/mydatabase" –from-literal=API_KEY"your_api_key_here" –from-literal=DEBUG_MODE="true" +``` + +This would create the same ConfigMap as the one we had defined before. As you can see, the longer the ConfigMap, the longer the command will be. And this ConfigMap with 3 values is itself a pretty messy command, which is why defining it as a yaml is always better. However, there are some use cases where the above method would be preferred. For example, if you wanted to auto-generate a bunch of ConfigMaps and create them in the cluster with a script, or you have a dynamic ConfigMap that changes during deployment time. + +You have already seen how you can define a ConfigMap using a yaml file. However, you can also create a ConfigMap from any other file type as well, such as a text file. You do this using a command similar to the above one, and it automatically creates a resource of kind ConfigMap that has the contents of the file you specified as its data: + +``` +kubectl create configmap example-configmap –from-file=key1=mongodb.txt +``` + +List the ConfigMaps: + +``` +kubectl get configmaps +``` + +You can then describe the ConfigMap to see its full resource definition: + +``` +kubectl describe configmap example-configmap +``` + +Next, let's quickly look at immutable ConfigMaps. If you don't want the contents of a ConfigMap to be changed after you deploy it, you can add the key `immutable` at the root level of the ConfigMap yaml, and it will throw a forbidden error if you try to change the existing ConfigMap. If you want to change the resource from this point forward, you will have to delete the old resource and create a new one. + +So, as you can see, there is a lot of flexibility when it comes to what you can define inside a ConfigMap, as well as in the ways you can refer to the ConfigMap. It's a very simple, yet very useful resource. As you progress along Kubernetes, you will constantly come across this particular resource, so make sure you understand it well before moving ahead to more complex resources. \ No newline at end of file diff --git a/DisasterRecovery101/dr-lab.md b/DisasterRecovery101/dr-lab.md new file mode 100644 index 00000000..13cd2678 --- /dev/null +++ b/DisasterRecovery101/dr-lab.md @@ -0,0 +1,124 @@ +## Lab + +For starters, we will be using the files we created in the [Terraform section](../Terraform101/terraform-eks-lab.md) to set up our VPC, subnet, and ultimately, our cluster. So go ahead and take those files and use them to create the cluster. Once the cluster is up, take the files from the [ArgoCD section](../GitOps101/argocd-eks.md), and deploy them. The first time around, you will have to either deploy the files manually, or use CLI commands to add the repos, create the project, add the cluster created by Terraform, and finally set up the application. As you may have noticed, this is rather time-consuming and not ideal in a DR situation. But the good news is that once your ArgoCD application is up, there is no reason to spend any time setting up the application all over again. So let's take a look at the script that will handle this for us. + +The script will be written in Python and take in a cluster endpoint as an argument. Basically, the only difference between the cluster that is running now vs the DR cluster is the cluster endpoint, so changing this value alone should re-deploy the modules into the new cluster. The output of this script is going to have this structure: + +- A command to add the new cluster to ArgoCD (`argocd cluster add dr-cluster`) +- A command to create any namespaces that will be needed for further deployments (kubectl create ns ) +- A command that sets the correct destination namespace (argocd app set --dest-server ) +- A command that force syncs the application so that it will automatically deploy to the new destination (argocd app sync ) + +The first step is to define the arguments: + +```python +parser = argparse.ArgumentParser(description="Destination cluster endpoint") +parser.add_argument( + "--dest-server", + required=True, + help="Server URL for destination cluster", +) +``` + +Now, we need to use the subprocess library to run ArgoCD CLI commands, similar to if we were running them on a local terminal. The first of these commands will be the list command which is used to list all the applications running on the (now-disabled) cluster: + +``` +argocd app list --project -o name +``` + +We also need to capture the output, so we should assign the result to a variable and set the necessary arguments: + +``` +result = subprocess.run( + ["argocd", "app", "list", "--project", "", "-o", "name"], + text=True, + capture_output=True, + check=True, +) +``` + +Now that we have the list of applications, it is time to start a loop that will go through this list and switch all the modules from one cluster to another. We will keep all the commands pointed to one file using the `with` command: + +``` +with open("change-applications.sh", "w") as file: +``` + +To start, we need to add the cluster to argocd: + +``` +file.write("argocd cluster add \n") +``` + +Make sure that you have added your new cluster to your local kubeconfig. Otherwise, the above `cluster add` command will fail. Since we already have the list of applications, start a `for` loop: + +``` +for application_name in application_names: +``` + +Inside the loop, start running the `set` commands so that each application has the set command running in it: + +``` +argocd_command = f'argocd app set {application_name} --dest-namespace --dest-server {args.dest_server}\n' +``` + +Followed by the sync command to force the application to update and switch to the new cluster: + +``` +argocd_sync = f'argocd app sync {application_name}\n' +``` + +Next, we write both those commands to the file we are creating: + +``` +file.write(argocd_command) +file.write(argocd_sync) +``` + +With that, the loop is complete: + +```python +with open("change-applications.sh", "w") as file: + file.write("argocd cluster add \n") + for application_name in application_names: + argocd_command = f'argocd app set {application_name} --dest-namespace --dest-server {args.dest_server}\n' + argocd_sync = f'argocd app sync {application_name}\n' + file.write(argocd_command) + file.write(argocd_sync) +``` + +If you have additional commands you would like to get run before the deployments start happening (such as the creation of a new namespace), it can be done at the start of the `with` command. You can also modify the `argocd` command to have any of the flags and options [used by the set command](https://argo-cd.readthedocs.io/en/stable/user-guide/commands/argocd_app_set/). So if you wanted to create a namespace called "application" and have the deployments be done into that namespace, you would add the line: + +``` +file.write("kubectl create ns application\n") +``` + +Under the `with` command, then modify the `argocd app set` command by adding `--dest-namespace application`. + +Since you used the `with` command to open the file, you don't have to close it, which means this is all that is required. + +## Usage + +Now that we have the entire DR plan in place, let's view what would happen during a DR situation, and what we would need to do to get everything back up and running. To start, we will be getting the cluster up and running with the below commands: + +`terraform init` +`terraform apply` + +This will set up both the cluster and any resources required to start the cluster. Starting the cluster will take a certain amount of time depending on how many nodes you have assigned to your node group. Once it is up, take the cluster endpoint head over to the location of your Python script, and run it: + +``` +python create_cluster.py --dest-server +``` + +This will create a file called "change-applications.sh". Look it over to ensure that everything you need is in there. It should have any custom commands you placed as well as the set command that changes the location of the cluster, along with force sync commands. If everything looks as it should be, go ahead and run the file: + +``` +sh change-applications.sh +``` + +If you have already logged into ArgoCD CLI via the terminal, then the commands should run sequentially and the pods should start scheduling in the new cluster. You can see this happen in real-time with the ArgoCD CLI. + +And that's it! You now have a full DR solution that will allow you to get your systems up and running in about 30 minutes. In a regular full-scale application, there would be things such as databases where you want to get the DB up and running with minimal data loss in the new region. When it comes to these matters, it's either better to hand off the management of these critical servers to a database service provider entirely so that they can handle the database DR (since database-related issues can get critical) or to have your own on-prem servers that you have a good handle on. You can then use these servers to host your databases during a DR situation. + +## Conclusion + +In this section, we reviewed how to do a full DR setup. This included creating a new EKS cluster, and then using ArgoCD to automatically deploy the existing pods to this newly created cluster. We also considered any possible pitfalls and improvements that would be necessary depending on the situation. \ No newline at end of file diff --git a/DisasterRecovery101/what-is-dr.md b/DisasterRecovery101/what-is-dr.md new file mode 100644 index 00000000..0b41f322 --- /dev/null +++ b/DisasterRecovery101/what-is-dr.md @@ -0,0 +1,15 @@ +# Disaster Recovery + +When you create a production-grade application, you assure your clients a certain amount of uptime. However, during a disaster situation, this uptime stops being guaranteed. For example, if you host your applications on the us-east-1 region of AWS, and that region goes down, you need to be able to get your application up and running on a different region, such as us-west-2, so that your customers get as little downtime as possible. In this section, we will explore the different ways we can set up a full disaster recovery solution for a Kubernetes cluster using tools we are already used to such as Terraform and ArgoCD. Since we will be using files from these two projects, please make sure you have completed the [Terraform](../Terraform101/what-is-terraform.md) and [ArgoCD](../GitOps101/argocd-eks.md) sections before starting this lesson. + +## Overview + +We will start by defining an overview of how our DR system will kick into place. There are two major components of our cluster: the cluster itself, and the applications that run on the cluster. We also have to think about the load balancers and how DNS traffic would get routed to the new cluster ingresses instead of the old ones. There are two ways to set up a Kubernetes cluster. The first is to create it manually, which is going to take a lot of time. Unfortunately, in a DR situation where we are trying to set up a new cluster as fast as possible, this isn't ideal. A much better option is to have your entire infrastructure written as code using a tool such as Terraform, and simply run a command that will automatically create all your infrastructure in the DR region for you. We have already covered the topic of how to use Terraform scripts to set up an EKS cluster [on our Terraform section](../Terraform101/terraform-eks-lab.md). This is what we will be using here. + +The second main part is the applications that run on the cluster. For this, we will be using ArgoCD. One thing to note is that ArgoCD isn't exactly a DR tool, but since it allows you to control deployments entirely using CLI commands, so we can have the commands ready to be deployed when a DR situation arises. So once the Terraform cluster is up, we can deploy our Kubernetes resources using ArgoCD. This can be done using either the ArgoCD interface or the CLI. Once that is done, we will be creating a Python script that generates a shell file. This file will have all the commands needed to take all the deployed ArgoCD applications and immediately point them toward a different cluster. + +So essentially, we would have an ArgoCD project with all the applications released and pointed at your regular cluster. Later, during a DR situation, a bunch of `argocd set` commands would run. These commands would have information about the new cluster endpoint that loops across all the applications in the project and change the cluster endpoint. Then the sync command would be run which gets ArgoCD to deploy all the modules to the new cluster. + +Now that we have a complete overview of the full DR process, let's get started with the lab. + +[Next: DR Lab](./dr-lab.md) \ No newline at end of file diff --git a/EKS101/what-is-eks.md b/EKS101/what-is-eks.md index 792745f6..7a8035c7 100644 --- a/EKS101/what-is-eks.md +++ b/EKS101/what-is-eks.md @@ -52,6 +52,16 @@ kubectl get nodes and if you can see the 2 nodes, then you are all set. +## Additional considerations + +Now that you have the entire cluster running on AWS, there are some things you may want to tweak to your liking. Firstly is the security group. While eksctl creates a default security group that has all the permissions needed to run your EKS cluster, it's best if you go back in and take another look at it. Firstly, ensure that your inbound rules do not allow 0.0.0.0, which would allow all external IPs to connect to your EKS ports. Instead, only allow IPs that you want to access your cluster through. You can do this by specifying the proper CIDR ranges and their associated ports. On the other hand, with outbound ports, allowing 0.0.0.0 is fine since this allows your cluster to communicate with any resource from outside your network. + +The next thing you can look at is the nodegroups. Since you specified `t2.micro` in the above command, your nodegroups will be created with that machine type. You can use the AWS console to add nodegroups with specific tolerations so that only certain pods get scheduled on these nodes. You can read more about taints and tolerations in the [Scheduler101 section](../Scheduler101/Nodes_taints_and_tolerations.md). You can also check the Kubernetes version that is used in your cluster from here. If you followed the above tutorial, you will have a cluster with Kubernetes version 1.24. You can update this version from the console. However, note that a lot of things vary from version to version, and you might end up getting something in your existing application broken if you blindly update your Kubernetes version. + +On the topic of updating, you will also notice an AMI version that is mentioned per each node group. Since you created this cluster recently, you will have the latest AMI version. However, AMIs get updated around twice each month, and while there won't be any major issues if you don't keep your AMIs updated, it is good to update as frequently as possible. Unlike updating the Kubernetes version, AMI updates are relatively safe since they only update the OS to have the latest packages specified by the AWS team. The update can be performed either as a rolling update, or a forced update. A rolling update will create a new node with the new AMI version and move all the pods in the old node to the new node before the old pods are drained and the old node is deleted. A forced update will immediately destroy the old node and start up a new node. The advantage of this method is that it is much faster and will always complete successfully, whereas a rolling update will take much longer and may fail to finish the update if any pods fail to drain. + +Another thing to consider is cost tagging. In a large organization, you would have multiple AWS resources that contribute to a large bill that you get at the end of the month. Usually, teams involved in costing would want to know exactly where the costs come from. If you were dealing with a resource such as an EC2 instance, you would not have to look deeply into this as you can just go into the cost explorer, filter by service, and just ask for the cost of the EC2 instances which would give you an exact amount on how much you spend on the resources. However, this becomes much more complicated with the EKS cluster. Not only do you have EC2 instances running in EKS clusters, but you are also paying for the control plane. Additionally, you also pay for EC2 resources such as load balancers and data transfer, along with a host of other things. To fully capture the total cost of your EKS cluster, you must use [cost allocation tags](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html) + ## Cleaning up Now, remember that all of the above things are AWS resources, and as such, you will be charged if you leave them running without deleting them after you are done. So this means you have a bunch of stuff (VPCs, cluster, EC2 instances) that you have to get rid of, which would have been a pain if you had to do it manually. However, since eksctl created all these resources for you, it can also get rid of all these resources for you, in the same manner, using a single command: diff --git a/Keda101/keda-lab.md b/Keda101/keda-lab.md index f5ada632..4ff9dd30 100644 --- a/Keda101/keda-lab.md +++ b/Keda101/keda-lab.md @@ -234,4 +234,4 @@ With the above configuration, a new Keda job will start every time a message is ## Conclusion -This wraps up the lesson on KEDA. What we tried out was a simple demonstration of a MySQL scaler followed by a demonstration of using various authentication methods to connect and consume messages from AWS SQS. This is a good representation of what you can expect from other data sources. If you were considering using this with a different Kubernetes engine running on a different cloud provider, the concept would still work. Make sure you read through the authentication page, which contains different methods of authentication for different cloud providers. If you want to try out other scalers, make sure you check out the [official samples page](https://github.com/kedacore/samples). \ No newline at end of file +This wraps up the lesson on KEDA. What we tried out was a simple demonstration of a MySQL scaler followed by a demonstration of using various authentication methods to connect and consume messages from AWS SQS. This is a good representation of what you can expect from other data sources. If you were considering using this with a different Kubernetes engine running on a different cloud provider, the concept would still work. Make sure you read through the authentication page, which contains different methods of authentication for different cloud providers. If you want to try out other scalers, make sure you check out the [official samples page](https://github.com/kedacore/samples). diff --git a/Kubezoo/kubezoo-lab.md b/Kubezoo/kubezoo-lab.md new file mode 100644 index 00000000..be2142c1 --- /dev/null +++ b/Kubezoo/kubezoo-lab.md @@ -0,0 +1,55 @@ +# Kubezoo Lab + +Now that we have covered what Kubezoo is, let's take a look at how we can set it up in a standard cluster. You could go ahead and use [Minikube](https://minikube.sigs.k8s.io/docs/start/), or you could create a cluster using [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation). You can also use any Kubernetes cluster you have at the ready. Let's start by cloning the [KubeZoo repo](https://github.com/kubewharf/kubezoo.git): + +``` +git clone https://github.com/kubewharf/kubezoo.git +``` + +Now, go to the root of the repo you just cloned, and run the `make` command: + +``` +make local-up +``` + +This will get Kubezoo up and running on port 6443 as long as the port is free. Check to see if the API resources are up and running: + +``` +kubectl api-resources --context zoo +``` + +Now, let's create a sample tenant. For this, we will be using the config/setup/sample_tenant.yamlsample_tenant.yaml provided in the repo. If you take a look at the tenant yaml file, you will notice that this is a custom resource of type "tenant", and contains just a few lines specifying the type of resources this tenant requires. The name of the tenant is "111111". Since this is a regular Kubernetes resource, let's go ahead and deploy this tenant as we would a normal yaml: + +``` +kubectl apply -f config/setup/sample_tenant.yaml --context zoo +``` + +Check that the tenant is has been setup: + +``` +kubectl get tenant 111111 --context zoo +``` + +Since this tenant is basically a "cluster" in itself, it has it's own kubeconfig that gets created for it. You can extract it using: + +``` +kubectl get tenant 111111 --context zoo -o jsonpath='{.metadata.annotations.kubezoo\.io\/tenant\.kubeconfig\.base64}' | base64 --decode > 111111.kubeconfig +``` + +You should now be able to deploy all sorts of resources to the tenant by specifying the kubeconfig. For example, if you were to deploy a file called "application.yaml" into the tenant, you would use: + +``` +kubectl apply -f application.yaml --kubeconfig 111111.kubeconfig +``` + +You can check the pod as the tenant by specifying the kubeconfig as before: + +``` +kubectl get po --kubeconfig 111111.kubeconfig +``` + +The pod would have been created in the namespace that you assigned to the tenant. If you were to have multiple tenants, you would not be able to see the pods of the other tenants as long as you only have the kubeconfig of the tenant that you are dealing with, which allows for better isolation. Using your regular kubeconfig as a cluster admin, if you were to list all pods with `kubectl get po -A`, you would be able to see all the pods of all the tenants separated by namespace. + +# Conclusion + +This brings us to the end of the section on Kubezoo. Hopefully, by now, you understand what a multi-tenant system is, what the benefits of such a system are, and what possible challenges you could face when using a system. You also know what Kubezoo can do to help alleviate these challenges, specifically when you have constraints such as a smaller development team and a large number of small clients. We also covered a lab on setting up Kubezoo in a kind cluster and deploying the items to the Kubezoo tenant, as well as showing how to interact with multiple tenants as a cluster admin. This covers the basics of Kubezoo. If you want to learn more on the topic, the official Kubezoo [GitHub page](https://github.com/kubewharf/kubezoo) is the best place to start. \ No newline at end of file diff --git a/Kubezoo/what-is-kubezoo.md b/Kubezoo/what-is-kubezoo.md new file mode 100644 index 00000000..e9f4193d --- /dev/null +++ b/Kubezoo/what-is-kubezoo.md @@ -0,0 +1,9 @@ +# Kubezoo + +If you have a large number of small clients that all rely on various services you provide, it makes little sense to have a separate Kubernetes cluster for each of them. An individual cluster will incur costs for control planes and each cluster needs to have supporting resources enabled which will result in resource usage. Multi-tenancy is the solution for this problem, where we only have a single cluster which we split into multiple namespaces, which are each assigned to a client or a "tenant". + +However, this solution comes with its own host of problems. The biggest issue is resource consumption. Imagine you have 4 nodes, each with a memory capacity of 16GB. If you have 3 tenants running on 3 namespaces within a cluster, one of those 3 tenants may consume 75% of the memory with their workloads while the other 2 are left with only 25%. Whatever the distribution may be, there will be a difference in the amount of resources used, and if each tenant is paying the same amount, this will lead to a disparity. It is therefore necessary to individually assign resources to each tenant depending on the amount they have requested so that tenants don't bottleneck each other. + +Now take a different situation. Instead of having 3 large clients, you have hundreds of small users. Each user needs to quickly run workloads in their own private "cluster", and it needs to be quick and efficient. This would be a pretty much impossible-to-manage situation without the proper tools. If we are talking about an average-sized team, it becomes infeasible from a manpower perspective to be able to handle these kinds of quick changes. + +This is where Kubezoo comes in. The solution they provide is Kubernetes API as a Service (KAaaS). Kubezoo allows you to easily share your cluster among hundreds of tenants, and allows sharing both the control plane and the data plane. This makes the resource efficiency as high as simply having a namespace for each tenant. However, unlike a namespace isolation method, this also has increased API compatibility as well as resource isolation. So while there are several different multi-tenancy options to choose from, Kubezoo is one of the best when it comes to handling a large number of small tenants. \ No newline at end of file diff --git a/Logging101/elk-on-kubernetes.md b/Logging101/elk-on-kubernetes.md new file mode 100644 index 00000000..a7cfc204 --- /dev/null +++ b/Logging101/elk-on-kubernetes.md @@ -0,0 +1,148 @@ +## Setting up the ELK stack on a Kubernetes instance + +To set up the ELK stack on Kubernetes, we will be relying on Helm charts. Therefore, make sure you have Helm set up on your machine. If you need a quick refresher on Helm, check out the [Helm section](../Helm101/what-is-helm.md). However, a deep understanding of Helm is not required here. The stack we will be installing is FileBeat, Logstash, Elasticasearch, and Kibana. All of these will ultimately run on the same namespace within containers (pods). Since that will produce container logs, we will be using them as an input data source. + +You will also need a Kubernetes cluster ready since that is where everything will be getting deployed to. Once again, I recommend [Minikube](https://minikube.sigs.k8s.io/docs/start/) to set up a cluster on your local machine, but feel free to use any cluster that is available to you. + +### Process Overview + +Here's a big picture of how this is going to work: + +- The container creates logs which get placed in a path +- filebeat reads the logs from the path and sends them to logstash +- logstash transforms and filters the logs before sending them forward to elasticsearch +- Kibana queries elasticsearch and visualizes the log data + +Note that logstash here is optional. You could very well send logs directly from filebeat to elasticsearch. However, in a real-world situation, rarely, you wouldn't want to use GROK patterns to match specific patterns that allow Kibana to show the data in a better-formatted manner, or filter logs based on the type of log, or add index patterns that allow the data to be retrieved much faster. If you have a large number of logs, logstash will also act as a buffer and prevent sending all the logs at once to elasticsearch, which would overload it. Therefore, we will be sending the data to logstash first in this example. + +As you can imagine, all of the above components are quite complex and require multiple resources to be deployed to get up and running. Therefore, we will be using Helm charts for each of the above components that automatically set up all the required resources for us. We could also use the single Kubernetes manifest file that elastic provides us, but the Helm chart allows for better flexibility. + +### Setting up + +To keep everything logically grouped together, we will be installing all the components in a namespace called `kube-logging`. So create it: + +``` +kubectl create ns kube-logging +``` + +The first component we will look into is filebeat. To set up filebeat, go ahead and get the relevant Helm chart from [its artifacthub page](https://artifacthub.io/packages/helm/elastic/filebeat?modal=install). Then use the provided commands to install the Helm chart: + +``` +helm repo add elastic https://helm.elastic.co +``` + +``` +helm install filebeat elastic/filebeat --version 8.5.1 -n kube-logging +``` + +If your kubeconfig file is set properly and it is pointed to your cluster, the command should run with no issues. You can then open up a terminal instance and run: + +``` +kubectl get po -n kube-logging +``` + +This will show you the filebeat pods that have started running in the kube-logging namespace. While the default configuration is meant to work out of the box, it won't fit our specific needs, so let's go ahead and customize it. To get the values.yaml, head back over to the Helm chart page on [Artifact hub](https://artifacthub.io/packages/helm/elastic/filebeat/7.6.1) and click on the "Default Values" option. Download this default values file. + +We will start by modifying this file. The values file provides configuration to support running filebeat as both a DaemonSet as well as a regular Deployment. By default, the DaemonSet configuration will be active, but we don't need that right now, so let's go ahead and disable it. Under the `daemonset` section, change `enabled: true` to `enabled: false`. You can then skip the rest of the DaemonSet section and head over to the `deployment` section. Then set change enabled to true to get filebeat deployed as a deployment. You might notice that the daemonset section and the deployment section are pretty much the same, so you only need to do the configuration for one section for it to work. + +If you have used filebeat before, you would be well aware of the `filebeat.yaml`. This is the file where you specify the filebeat configurations, such as where to get the logs from, and where to send the logs to. In this case, we will be declaring the yaml within the values.yaml. There is a basic filebeat.yml defined within the values file to help you get started. We will be changing everything here. Replace this block with the below code: + +``` +filebeat.yml: | + filebeat.inputs: + - input_type: log + paths: + - /var/log/containers/*.log + document_type: mixlog + output.logstash: + hosts: ["logstash-logstash:5044"] +``` + +The path provided above will get all the logs produced by the container. We will be marking these logs as type "mixlog" and filtering these out based on this type later in the flow. The last piece of configuration is `output.logstash` which tells where the log should be sent to. Here, we specify logstash instead of elasticsearch. Note that since this is loading a regular filebeat container, you are allowed to do additional configuration that you would normally do with a pod. For example, if instead of just getting the container logs of the same pod-like we are doing here, you wanted to get the logs from a different pod, that would be possible. To do so, you would have to copy the logs from your source container to a network volume mount (such as EFS) which would mean that the volume now has your source pods' logs. To mount these volumes, you would use the `extraVolumes` and `extraVolumeMounts` options that you see in the Helm values file. You can then mount this volume into your filebeat pod and change the filebeat.yml to point at the mounted volume: + +``` +paths: + - /different_pod_logs/*.log +``` + +The filebeat configuration is now complete. We will now handle the second part of the log flow: logstash. As with filebeat, we will be using Helm charts to get logstash up on the Kubernetes cluster. We will also be changing the values file in the same way. + +To start, head over to the logstash chart on [Artifact Hub](https://artifacthub.io/packages/helm/elastic/logstash). As with filebeat, download the values file. While in filebeat we had the filebeat.yml to set our configuration, in logstash we have a `logstash.conf`. We will be declaring the logstash conf in the yaml as we did before. Remove any default logstash conf values that may exist, and replace them with: + +``` +logstashPipeline: + logstash.conf: | + input { + beats { + type => mixlog + port => 5044 + } + } + filter { + if [type] == "mixlog" { + grok { + match => { + "message" => "%{TIMESTAMP_ISO8601:timestamp}" + } + } + } + } + output { elasticsearch { hosts => "http://elasticsearch:9200" } } +``` + +In the above config, we declare that we will be getting inputs from filebeat from port 5044 (which is the port filebeat exposes). This port will send logs of type "mixlog" which we declared in the filebeat.yml. Now, logstash will start gathering all logs that filebeat sends from port 5044, which we can either redirect to Elasticsearch or perform some processing on. In this case, we will be using GROK patterns to do the filtering. GROK patterns are very similar to regex patterns and can be used in other log-matching services such as fluentd. In the above config, we declare that for any log to go through, it must have a timestamp. All sorts of filters can applied, and you can find a comprehensive list of all of them [here](https://www.elastic.co/guide/en/logstash/current/filter-plugins.html). There are also other things logstash can do to your logs apart from filtering them. The final thing we do in the config is to redirect the logs to elasticsearch. For this, we specify the host and port of elasticsearch, and that is all it takes. + +Note that all of these steps assume that you are running the entire stack on one single namespace. If, for example, you have logstash in one namespace and elasticsearch in another namespace, this above configuration will not work. This is because we are simply referring to elasticsearch with "http://elasticsearch:9200" without providing any namespaces, which means elasticsearch will automatically assume that it should look within the same namespace. If you need to specify a different namespace, you would have to use the full svc name: elasticsearch.different-namespace.svc.cluster.local:9200. + +The filebeat configuration is now complete, and we can move on to the next phase of the flow: elasticsearch. As with the previous two components, you can set up elasticsearch using the relevant Helm chart. Go to the page in the [Artifact Hub](https://artifacthub.io/packages/helm/elastic/elasticsearch) and get the provided install command: + +``` +helm install my-elasticsearch elastic/elasticsearch --version 8.5.1 +``` + +Unlike with the previous two modules, the elasticsearch chart can be installed as-is and work properly. This is because elasticsearch is being used here as somewhat of a data store. No data is being transformed or filtered, and since the data is already being fed in by filebeat, there is no need to set up custom volume mounts. Installing the chart will create an elasticsearch statefulset that will create two pods. It will also create a service that runs on port 9200. This way, you have a fully functional elasticsearch istance that logstash can reach. + +However, you also have a second option if you require more flexibility over the elasticsearch cluster. Instead of using Helm, go ahead and use [this yaml](https://gist.github.com/harsh4870/ccd6ef71eaac2f09d7e136307e3ecda6). If you are fine with the level of customizability the Helm chart provides, you can skip this part and head straight to the Kibana section. Once you have created a file from it, we can modify the file so that we have all our fine-grained configs. You might notice that this file only contains a StatefulSet. However, for elasticsearch to be exposed to other services such as logstash and Kibana, it needs a service. So create a file called `elasticsearch_svc.yaml` and add this into it: + +``` +kind: Service +apiVersion: v1 +metadata: + name: elasticsearch + namespace: kube-logging + labels: + app: elasticsearch +spec: + selector: + app: elasticsearch + clusterIP: None + ports: + - port: 9200 + name: rest + - port: 9300 + name: inter-node +``` + +The above yaml will allow services to call elasticsearch for port 9200. As you can see, it is fairly straightforward. So let's move on to the StatefulSet. The only config you need to set is the `cluster.name` config which is where you need to set the name of your elasticsearch cluster. Apart from that, everything else is filled in. In cases where a large amount of data is going to be flowing into your cluster, you can increase the `ES_JAVA_OPTS` value to something higher than 512 MB. You also have fine-grained control over things such as the DNS policy and init commands used when starting the elasticsearch cluster. Once you have finished doing all the necessary configurations, you can go ahead and deploy the elasticsearch cluster with the `kubectl apply` command. You are almost done with the ELK stack setup. + +The final part of the stack is Kibana. Now that you have all the necessary pieces in place to capture, filter, and store your logs in elasticsearch, it is time to start making use of this data by visualizing it. As with the other modules, we can use Helm to install Kibana: + +``` +helm install my-kibana elastic/kibana --version 8.5.1 +``` + +You should also take the values yaml and make modifications to it. Specifically, you might notice that the values file has the elasticsearch host defined as "https://elasticsearch-master:9200". Change this to "https://elasticsearch:9200" since that is the name of the service that runs elasticsearch. Apart from that, there are no major modifications that need doing, and simply installing the chart should get Kibana to start working. While installing this may get Kibana up and running, you won't yet be able to access it from your browser. This is due to the Helm chart having a cluster IP setup. You can use port forwarding to forward your localhost port to the Kibana port and access Kibana this way. + +``` +kubectl port-forward 5601:5601 +``` + +This will allow you to access Kibana at http://localhost:5601. You could also edit the svc and set the type to "NodePort" which would allow you to access the port using the IP of the pod. If you are running a managed cluster on something like EKS, you will have to open the port from the security group or firewall to allow external access to Kibana. However, all this is not necessary in a testing setup, so we will be sticking to port forwarding. + +Now that Kibana is accessible, you might notice that the dashboard is empty. This is because Kibana lacks any indexes. Since we are using logstash and the logstash format, we can simply ask Kibana to index any entries that come formatted with "logstash-*". To set this index, head over to the main menu > stack management > index patterns and choose "Create index pattern". Once you type in the proper index, Kibana will search elasticsearch and match the index pattern with the logs. If no results are found, then something has gone wrong with the process. There might be some issue with either filebeat, logstash, or elasticsearch. If everything is working fine, you should see several logs matching the pattern, and you can go ahead and add the index. With the index added, go back to the dashboard and you should see the logs being presented here. Note that since we used a grok pattern, Kibana will try to fit the data into the provided pattern. In this case, we captured the timestamp so you should see a field called timestamp being separated from the rest of the message. Depending on what your log is, you could use grok patterns to capture various parts of the log. For example, if you are logging any Java errors you get from a Java application, you should be able to separately capture the method from which the error originated, the stack trace, and the severity of the error and have them separated by grok patterns. Afterward, when they are displayed by Kibana, you should be able to see the logs being separated by the fields you specified grok patterns with. You can then use this for further filtering from within Kibana. So for example, if your grok pattern matched a field called "severity", you can use "severity: high" in the Kibana search bar to only get the high severity logs. + +That brings us to the end of setting up an end-to-end ELK setup on your Kubernetes cluster. So now, let us discuss some potential problems with the setup about Kubernetes. + +Firstly, Kubernetes has a highly distributed nature, where each pod runs a separate application. This in itself is not a problem. Even if you have hundreds of pods running a hundred different applications, filebeat is more than capable of handling all those open log files that are being constantly written into and passing them on to logstash. Logstash then manages the in-flow of logs to make sure elasticsearch isn't overwhelmed. The problems start appearing if you have a sudden surge in the number of pods. This is not normal when it comes to everyday Kubernetes use cases. However, if you were using autoscaling jobs that massively scaled up and down depending on the workload, this could happen. One good example is with [KEDA](../Keda101/what-is-keda.md). KEDA looks at whatever metric you specify and massively scales jobs (basically pods) up and down to handle the demand. If each of these jobs writes a log to a common data source (such as EFS or other network file system), you could potentially have hundreds of open log files that are concurrently being written into. At this point, a single instance of filebeat may be unable to keep track of all these log files and either end up skipping some logs or stop pushing logs entirely. The solution for this is to either have multiple replicas of filebeat or to launch filebeat as a sidecar container for each pod that comes up. However, this is beyond the scope of this tutorial. The setup you have created now is enough to push logs for any regular applications. + +With that, we are done with the logging section of this tutorial. \ No newline at end of file diff --git a/Logging101/fluentdbit.md b/Logging101/fluentdbit.md index 9a488d1c..760ff688 100644 --- a/Logging101/fluentdbit.md +++ b/Logging101/fluentdbit.md @@ -78,4 +78,8 @@ And finishes with information regarding the volume mounts for the DaemonSet. To sum up, Fluent Bit is basically fluentd, but with a much smaller footprint and file size. It runs in a more lightweight manner and consumes resources which makes it an ideal log processor for systems that have few resources. There are also a few unique features that Fluent Bit has that fluentd doesn't, and vice versa. -If you haven't had enough about logging with Kuberetnes, jump into the [Kafka section](../Strimzi101/kafka.md) of this course, which provides a basic introduction to Kafka and details how to use [Strizmi](https://strimzi.io) to get a Kafka cluster running within your Kubernetes cluster. \ No newline at end of file +If you haven't had enough about logging with Kuberetnes, jump into the [Kafka section](../Strimzi101/kafka.md) of this course, which provides a basic introduction to Kafka and details how to use [Strizmi](https://strimzi.io) to get a Kafka cluster running within your Kubernetes cluster. + +Now, let's take things a step further and look at how we can set up the entire ELK stack on a Kubernetes cluster. + +[Next: ELK on Kubernetes](./elk-on-kubernetes.md) \ No newline at end of file diff --git a/Logging101/what-is-elasticsearch.md b/Logging101/what-is-elasticsearch.md index c29991d1..8ca53592 100644 --- a/Logging101/what-is-elasticsearch.md +++ b/Logging101/what-is-elasticsearch.md @@ -35,7 +35,9 @@ The best part about the ELK stack is that it is built to run continuously. LogSt ## Setting up the Elasticsearch -Now that you know what each letter of the stack stands for, let's go ahead and set it up. Luckily, Elastic has provided us with a [large sample repo](https://github.com/elastic/examples) that we can use to try out the stack with minimal hassle. In particular, we will be using the MonitoringKubernetes sample that covers all three parts of the stack. Note that this sample substitutes Logstash with [Beats](https://www.elastic.co/beats/), which is an alternative provided by Elastic. We could go for another sample such as the [Twitter sample](https://github.com/elastic/examples/tree/master/Common%20Data%20Formats/twitter), however, this requires access to the Twitter API which isn't readily available. However, feel free to try out any sample in the repo. Before we get into the sample, you will need to have a working stack set up. If you don't, then it would be much faster to get started on [Elastic cloud](http://cloud.elastic.co), which has a free tier that would suffice for this sample. +Note that the below lab is for use with Elastic cloud. If you wish to set up the ELK stack entirely on Kubernetes without relying on Elastic cloud, please skip the rest of this tutorial and head over directly to the [ELK on Kubernetes](./elk-on-kubernetes.md) section. + +Now that you know what each letter of the stack stands for, let's go ahead and set it up. Luckily, Elastic has provided us with a [large sample repo](https://github.com/elastic/examples) that we can use to try out the stack with minimal hassle. In particular, we will be using the MonitoringKubernetes sample that covers all three parts of the stack. Note that this sample substitutes Logstash with [Beats](https://www.elastic.co/beats/), which is an alternative provided by Elastic. We could go for another sample such as the [Twitter sample](https://github.com/elastic/examples/tree/master/Common%20Data%20Formats/twitter), however, this requires access to the Twitter API which isn't readily available. However, feel free to try out any sample in the repo. Before we get into the sample, you will need to have a working stack set up. If you don't, then it would be much faster to get started on [Elastic cloud](http://cloud.elastic.co), which has a free tier that would suffice for this sample. As always, you also need an available Kubernetes cluster. Once again, I recommend [Minikube](https://minikube.sigs.k8s.io/docs/start/) to set up a cluster on your local machine, but feel free to use any cluster that is available to you. Once you have a cluster available, use RBAC to create a cluster role that binds to your elastic user: diff --git a/README.md b/README.md index a72e3a3b..a4fef8e5 100644 --- a/README.md +++ b/README.md @@ -118,6 +118,8 @@ A Curated List of Kubernetes Labs and Tutorials - [Rollback updates to application deployment](./Deployment101/README.md#step-4-rollback-updates-to-application-deployment) - [Cleaning Up](./Deployment101/README.md#step-5-cleanup) +## ConfigMaps101 + - [What are ConfigMaps?](./ConfigMaps101/what-are-configmaps.md) ## Scheduler101 @@ -239,6 +241,7 @@ A Curated List of Kubernetes Labs and Tutorials - [Fluentd](./Logging101/fluentd.md) - [Fluentd on Kubernetes](./Logging101/fluentd-kube.md) - [Fluent Bit](./Logging101/fluentdbit.md) + - [ELK on Kubernetes](./Logging101/elk-on-kubernetes.md) ## Helm101 @@ -320,6 +323,14 @@ A Curated List of Kubernetes Labs and Tutorials - [What is Terraform](./Terraform101/what-is-terraform.md) - [Terraform EKS Lab](./Terraform101/terraform-eks-lab.md) +## Disaster Recover +- [What is Disaster Recovery](./DisasterRecovery101/what-is-dr.md) +- [DR Lab](./DisasterRecovery101/dr-lab.md) + +## Kubezoo +- [What is Kubezoo](./Kubezoo/what-is-kubezoo.md) +- [Kubezoo lab](./Kubezoo/kubezoo-lab.md) + ## For Node Developers - [Kubernetes for Node Developers](./nodejs.md)