|
| 1 | +# Anchor and Octopus |
| 2 | + |
| 3 | +Some applications, especially legacy applications or applications which monitor network traffic, expect to be directly connected to the physical network. In this type of situation, users hope using the macvlan network driver to assign a MAC address to each container’s virtual network interface, making it appear to be a physical network interface directly connected to the physical network. |
| 4 | + |
| 5 | +## There comes anchor |
| 6 | + |
| 7 | +Project anchor mainly contains four compenents, They are: |
| 8 | + |
| 9 | +Anchor is an ipam plugin following the [CNI SPEC](https://github.com/containernetworking/cni/blob/master/SPEC.md). |
| 10 | + |
| 11 | +Octopus is a main plugin that extends [macvlan](https://github.com/containernetworking/plugins/blob/master/plugins/main/macvlan/macvlan.go) to support multiple masters on the node. It is useful when there are multiple VLANs in the cluster. |
| 12 | + |
| 13 | +Monkey is a WebUI that displays and operates the data used by anchor ipam. |
| 14 | + |
| 15 | +The backstage hero is the installation script of the anchor, which configures and maintains the network interfaces of the node. |
| 16 | + |
| 17 | +## CNI and Kubernetes |
| 18 | + |
| 19 | +CNI(Container Network Interface), a CNCF project, consists of a specification and libraries for writing plugins to configure network interfaces in Linux containers, along with a number of supported plugins. CNI concerns itself only with network connectivity of containers and removing allocated resources when the container is deleted. Because of this focus, CNI has a wide range of support and the specification is simple to implement. |
| 20 | + |
| 21 | +It is worth mentioning that kubernetes is just one of CNI runtimes, others including mesos, rkt, openshift. |
| 22 | + |
| 23 | +## MacVLAN |
| 24 | + |
| 25 | +MacVLAN is a Linux network driver that exposes underlay or host interfaces directly to VMs or Containers running in the host. |
| 26 | + |
| 27 | +MacVLAN allows a single physical interface to have multiple MAC and ip addresses using MacVLAN sub-interfaces. MacVLAN interface is typically used for virtualization applications and each MacVLAN interface is connected to a Container or VM. Each container or VM can directly get dhcp address as the host would do. This would help customers who want Containers to be part of their traditional network with the IP addressing scheme that they already have. |
| 28 | + |
| 29 | +When using MacVLAN, the containers is **NOT** reachable to the underlying host interfaces as the packages are intentionally filtered by Linux for additional isolation. This does not meet the SPEC of CNI and *service* in k8s cannot work correnctly. To work around it, we create a new MacVLAN interface and steal the IP and network traffic from the host interface by changing the route table on the node. This work is designed to be done by installation script. |
| 30 | + |
| 31 | +## Installation |
| 32 | + |
| 33 | +Please knowing that, most cloud providers(Amazon, Google, Aliyun) don't allow promiscuous mode, you may deploy Anchor on your own premises. |
| 34 | + |
| 35 | +Recently, I have no resources(No time, no machines) to test whether anchor works well with other runtimes except kubernetes. The document below focuses on kubernetes. |
| 36 | + |
| 37 | +**Prepare the cluster** |
| 38 | + |
| 39 | +* Enable promiscuous mode on switch(or virtual switch) |
| 40 | +* Create a new kubernetes cluster without any CNI plugin |
| 41 | +* Reserve several IPs available for applications |
| 42 | + |
| 43 | +**Install Anchor** |
| 44 | + |
| 45 | +```shell |
| 46 | +curl -O https://raw.githubusercontent.com/hainesc/anchor/master/deployment/anchor.yaml |
| 47 | +``` |
| 48 | + |
| 49 | +Edit the anchor.yaml use your favorite editor, *L* means *Line* below. |
| 50 | + |
| 51 | +* Remove L200 and lines below if the k8s cluster has not enabled RBAC. |
| 52 | +* L8, input the etcd endpoints used as the store for anchor, example at the end of the line. |
| 53 | +* L10 - L12, input the access token of the etcd, do nothing if SSL not enabled. |
| 54 | +* L18, input the choice whether or not create macvlan interface during the installation. |
| 55 | +* L22, input the cluster network information. Use semicolon(;) to seperate between items. eg, item *node01,eth0.2,10.0.2.8,10.0.2.1,24* tells install script creating a MacVLAN interface with the master *eth0.2* at the node whose hostname is *node01*, the additional info including IP of the master(*eth0.2* here), the gateway and mask of the subnet(10.0.2.1 and 24). You CAN have Multiple items for each node. |
| 56 | + |
| 57 | +Save the change and run: |
| 58 | + |
| 59 | +```shell |
| 60 | +kubectl apply -f anchor.yaml |
| 61 | +``` |
| 62 | + |
| 63 | +Wait for installation to complete, it will create a daemonset named anchor, a service account named anchor, a cluster role and a cluster role binding if RBAC enabled. |
| 64 | + |
| 65 | +There are several works done by the pod which created by the daemonset on each node: |
| 66 | + |
| 67 | +* Copy binary files named *anchor* and *octopus* to the node |
| 68 | +* Config and write a CNI config file named 10-anchor.conf to the node |
| 69 | +* Create MacVLAN interface(s) on the node, the interfaces created here will be removed on node restart, but when the node rejoin the k8s cluster, the daemonset recreates a pod and it will recrete the interfaces. |
| 70 | + |
| 71 | +## Example |
| 72 | + |
| 73 | +**Preparation** |
| 74 | + |
| 75 | +There are three k-v stores used by the anchor ipam, they are: |
| 76 | + |
| 77 | +| KV | Example | Explanation | |
| 78 | +|:----:|:---------:|:-------------:| |
| 79 | +| Namespace -> IPs | /anchor/ns/default -> 10.0.1.[2-9],10.0.2.8 | IPs are reserved and can be used by the namespace in the key | |
| 80 | +| Subnet -> Gateway | /anchor/gw/10.0.1.0/24 -> 10.0.1.1 | The map between subnet and its gateway | |
| 81 | +| Container -> IP | /anchor/cn/212b... -> 10.0.1.2 | The IP binding with the ContainerID | |
| 82 | + |
| 83 | +At the beginning, the stores are empty, so just input some data according to the environment. |
| 84 | + |
| 85 | +I have created a WebUI named [Powder monkey](https://github.com/hainesc/powder) to display and operate the k-v stores. The frontend is written in Angular and the backend written in Golang. |
| 86 | + |
| 87 | +**Run example** |
| 88 | + |
| 89 | +```shell |
| 90 | +curl -O https://raw.githubusercontent.com/hainesc/anchor/master/examples/anchor-2048.yaml |
| 91 | +``` |
| 92 | + |
| 93 | +Edit L14 and choose a subnet for it, then Run: |
| 94 | + |
| 95 | +```shell |
| 96 | +kubectl apply -f anchor-2048.yaml |
| 97 | +``` |
| 98 | + |
| 99 | +Wait for the installation to complete, it will create a deployment named anchor-2048 and the service named anchor-2048. |
| 100 | + |
| 101 | +```shell |
| 102 | +kubectl get pods -n default -o wide |
| 103 | +``` |
| 104 | +will displays the IP binding with the Pod. Open you browser and enjoy the game via the IP of the pod. |
| 105 | + |
| 106 | +Please describe the Pod if some errors, and refer to the log of kubelet for more details. |
| 107 | + |
| 108 | +## Customized |
| 109 | + |
| 110 | +Anchor uses *annotations* written in the yaml for passing customized config as you see in the example. |
| 111 | + |
| 112 | + |
| 113 | +| Key | Value | Explanation | |
| 114 | +|:-----:|:-------:|:-------------:| |
| 115 | +| cni.anchor.org/subnet | 10.0.1.0/24 | The Pod should be allocated an IP in the subnet | |
| 116 | +| cni.anchor.org/gateway | 10.0.1.254 | The gateway of the pod is overwritten by the customized one | |
| 117 | +| cni.anchor.org/routes | 10.88.0.0/16,10.0.1.5;10.99.1.0/24,10.0.1.7 | Add customized routes for the pod | |
| 118 | + |
| 119 | +The *cni.anchor.org/subnet* is **mandatory** since anchor cannot guess an IP if it don't know which VLAN the pod in. |
| 120 | + |
| 121 | +## TODO |
| 122 | + |
| 123 | +* IPv6 support |
| 124 | +* K-V store redesign |
| 125 | +* Powder monkey improvement |
| 126 | + |
| 127 | +## Donation |
| 128 | + |
| 129 | +If you find anchor is useful and helpful, please consider making a donation. |
| 130 | + |
| 131 | +* Bitcoin: 1HainescwMrDfay9cU1yWC6vc8ThZj8uAQ |
| 132 | + |
| 133 | + |
| 134 | + |
0 commit comments