diff --git a/docs/modules/ROOT/pages/references/vsphere/architecture.adoc b/docs/modules/ROOT/pages/references/vsphere/architecture.adoc index de974995..4cc34a89 100644 --- a/docs/modules/ROOT/pages/references/vsphere/architecture.adoc +++ b/docs/modules/ROOT/pages/references/vsphere/architecture.adoc @@ -1,40 +1,33 @@ -= APPUiO Managed OpenShift 4 on VMWare vSphere +:infra-type: VMWare vSphere +:infra-svg: ocp4-architecture-vsphere.svg += APPUiO Managed OpenShift 4 on {infra-type} == Architecture overview -The APPUiO Managed OpenShift 4 architecture on VMWare vSphere is based on the xref:references/architecture/index.adoc[generic APPUiO Managed OpenShift 4 architecture]. -We expect that readers of this document are familiar with the xref:references/architecture/index.adoc[generic APPUiO Managed OpenShift 4 architecture] and the overall Kubernetes and OpenShift 4 architecture. +include::partial$architecture/overview.adoc[] -This architecture document extends upon the generic architecture by defining how the APPUiO Managed OpenShift 4 cluster is embedded into the VMWare vSphere environment. -The diagram below shows a detailed view of how an APPUiO Managed OpenShift 4 cluster is embedded into an existing VMWare vSphere environment. +== {infra-type} requirements -.APPUiO Managed OpenShift4 on vSphere architecture -image::ocp4-architecture-vsphere.svg[alt=OCP4 vSphere architecture, width=900] +Red Hat OpenShift 4 imposes version requirements on the VMWare virtual hardware version, {infra-type} ESXi and vCenter. +See the https://docs.openshift.com/container-platform/latest/installing/installing_vsphere/installing-vsphere-installer-provisioned.html#installation-vsphere-infrastructure_installing-vsphere-installer-provisioned[upstream documentation] for the specific version requirements as well as further details on required {infra-type} requirements. -The following sections of this document provide detailed descriptions for the elements shown in the architecture diagram. +APPUiO Managed OpenShift 4 needs credentials to access the {infra-type} API for three main reasons: -== vSphere requirements - -Red Hat OpenShift 4 imposes version requirements on the VMWare virtual hardware version, vSphere ESXi and vCenter. -See the https://docs.openshift.com/container-platform/latest/installing/installing_vsphere/installing-vsphere-installer-provisioned.html#installation-vsphere-infrastructure_installing-vsphere-installer-provisioned[upstream documentation] for the specific version requirements as well as further details on required VMWare vSphere requirements. - -APPUiO Managed OpenShift 4 needs credentials to access the vSphere API for three main reasons: - -1. The OpenShift 4 installer needs access to vSphere to setup the OpenShift 4 cluster +1. The OpenShift 4 installer needs access to {infra-type} to setup the OpenShift 4 cluster + NOTE: The installer also needs to be able to access at least one ESXi host to upload the VM template from which all the cluster VMs will be created. 2. OpenShift 4 manages the VMs making up the cluster from within the cluster. -3. The vSphere CSI driver manages additional block devices that can be used by applications +3. The {infra-type} CSI driver manages additional block devices that can be used by applications -The upstream OpenShift 4 documentation has detailed informations about the https://docs.openshift.com/container-platform/latest/installing/installing_vsphere/installing-vsphere-installer-provisioned.html#installation-vsphere-installer-infra-requirements_installing-vsphere-installer-provisioned[required permissions on vSphere]. +The upstream OpenShift 4 documentation has detailed informations about the https://docs.openshift.com/container-platform/latest/installing/installing_vsphere/installing-vsphere-installer-provisioned.html#installation-vsphere-installer-infra-requirements_installing-vsphere-installer-provisioned[required permissions on {infra-type}]. -NOTE: Entries for "VMWare vSphere CSI Driver Operator" are required. +NOTE: Entries for "{infra-type} CSI Driver Operator" are required. == Networking === Bastion host -To deploy an APPUiO Managed OpenShift 4 cluster on VMWare vSphere, a bastion host inside the customer's premise is required. +To deploy an APPUiO Managed OpenShift 4 cluster on {infra-type}, a bastion host inside the customer's premise is required. The bastion host: * must be accessible via SSH from a management system operated by VSHN @@ -44,18 +37,11 @@ The bastion host: * must run a recent Ubuntu version The bastion host is used to run the installer from, and for troubleshooting access to both the cluster and the vCenter. -The bastion host must be provided by the vSphere infrastructure operator, but VSHN can handle management and maintenance. +The bastion host must be provided by the {infra-type} infrastructure operator, but VSHN can handle management and maintenance. === Machine network -Each APPUiO Managed OpenShift 4 cluster is deployed into a /24 "cluster machine network" (sometimes also "cluster network" or "machine network") -This network must be provided by the vSphere infrastructure operator. -DHCP is mandatory for this network, but a number of IPs must be reserved to be used as Virtual IPs for the cluster. - -Traffic inside this network shouldn't be restricted. - -VMs in this network must be able to reach various services on the internet. -See below for a detailed list of external systems that must be reachable. +include::partial$architecture/networking-cluster.adoc[] === Virtual IPs @@ -73,29 +59,11 @@ These additional VIPs will be managed by `keepalived` instances on the cluster. === Pod and service networks -APPUiO Managed Openshift 4 uses https://cilium.io/[Cilium] to provide in-cluster networking. -Cilium allocates two cluster-internal networks: - -1. The pod network: every pod on the cluster will get an IP address from this network. -This network enables basic in-cluster connectivity. -APPUiO Managed OpenShift 4 uses `10.128.0.0/14` as the pod network. -Each node in the cluster is assigned a `/23` from this range. -Pods on a node are always assigned an IP from the range allocated to that node. -2. Service network: used for service discovery. -Traffic to IPs in this network is forwarded to the appropriate pods by Cilium. -APPUiO Managed OpenShift 4 uses `172.30.0.0/16` as the service network. - -Both of these networks are interanl to the OpenShift 4 cluster. -Therefore, the IP CIDRs for these networks must not be routable from the outside. -Additionally, the same IP CIDRs can be reused for multiple OpenShift 4 clusters. - -However, the chosen CIDRs shouldn't overlap with existing networks allocated by the customer. -If there are overlaps, external systems in the overlapping ranges won't be accessible from within the OpenShift 4 cluster. -The pod and service network CIDRs can be customized if and only if there are conflicts. +include::partial$architecture/networking-pods.adoc[] === Exposing the cluster -The vSphere infrastructure operator must provide some form of ingress and egress gateway for the cluster. +The {infra-type} infrastructure operator must provide some form of ingress and egress gateway for the cluster. The ingress gateway must expose two public IPs: 1. A public IP for the API. @@ -112,88 +80,15 @@ Additional DNS records can be pointed to this IP by the customer. === External services -APPUiO Managed OpenShift 4 requires various external services. - -==== VSHN services - -APPUiO Managed OpenShift 4 requires access to VSHN's https://syn.tools[Project Syn] infrastructure. -The Project Syn infrastructure components that must be reachable are - -* the Project Syn API at `\https://api.syn.vshn.net` -* the Project Syn Vault at `\https://vault-prod.syn.vshn.net` -* VSHN's GitLab instance at `ssh://git@git.vshn.net` -* VSHN's acme-dns instance at `\https://acme-dns-api.vshn.net` - -Additionally, APPUiO Managed OpenShift 4 requires access to VSHN's identity management: - -* VSHN LDAP at `ldaps://ldap.vshn.net:636` -* VSHN SSO at `\https://id.vshn.net` - -Finally, APPUiO Managed OpenShift 4 requires access to VSHN's central metrics storage at `\https://metrics-receive.appuio.net` - -==== Red Hat services - -See the https://docs.openshift.com/container-platform/4.14/installing/install_config/configuring-firewall.html#configuring-firewall_configuring-firewall[upstream documentation] for the full list of services. - -The most important services for APPUiO Managed OpenShift 4 are - -* the Red Hat container registries at `registry.redhat.io` and `registry.access.redhat.com`. -* the OpenShift Update Service (OSUS) at `\https://api.openshift.com`. - -==== 3rd party services - -Finally, APPUiO Managed OpenShift 4 requires access to a number of third party services: - -* OpsGenie at `\https://api.opsgenie.com` -* Passbolt at `\https://cloud.passbolt.com/vshn` -* Let's Encrypt at `\https://acme-v02.api.letsencrypt.com` and `\https://acme-staging-v02.api.letsencrypt.com` -* Various container registries -** GitHub at `ghcr.io` -** Quay at `quay.io` -** DockerHub at `docker.io` -** Google container registry at `gcr.io` -** Kubernetes container registry at `registry.k8s.io` +include::partial$architecture/networking-external.adoc[] == Storage -APPUiO managed OpenShift 4 requires 3 different types of storage: - -1. Root disks -2. Persistent volumes -3. S3 compatible object storage - -=== Root disks - -Root disks are virtual block devices (100 GiB) which are attached to the VMs which make up the APPUiO Managed OpenShift 4 cluster. -The root disks are allocated and attached to the VM when the VM is created. -They hold the operating system and temporary data. -They're ephemeral (no application data is stored on them), and don't need to be backed up. -Finally, root disks are deleted when the VM to which they're attached is deleted. - -=== Persistent volumes - -Persistent volumes are virtual block devices with arbitrary sizes. -They're allocated dynamically based on requests from workloads (applications or infrastructure components) within the cluster. -These block devices are automatically attached to the VM hosting the application container. -They're deleted when the corresponding Kubernetes `PersistentVolume` resource is deleted. - -The VMWare vSphere CSI driver is the in-cluster component which is responsible for allocating, attaching and deleting the persistent volume block devices. - -These devices hold application data, but backups are usually done from within the cluster. - -=== S3 compatible object storage - -Various OpenShift components, such as the integrated image registry, the logging stack and backups, require S3 compatible object storage. -The customer or vSphere infrastructure operator must provide S3 compatible object storage. -Most modern storage solutions offer some object storage functionality. - -If https://products.vshn.ch/appcat/index.html[VSHN's Application Catalog (AppCat)] offering is required on the cluster, the object storage must support automatic bucket creation via an AppCat-supported provisioner. - -NOTE: If no object storage is available, we can use external object storage as a fallback. +include::partial$architecture/storage.adoc[] == Glossary -=== Components +=== Components vSphere [cols="1,3,1"] |=== @@ -213,24 +108,15 @@ a|A simple Ubuntu VM which is used by VSHN to bootstrap the cluster(s) and for e ** access to the vSphere API ** access to at least one ESXi host to allow the initial VM template upload -|vSphere infrastructure operator - -|Installer -|A CLI tool that bootstraps an OpenShift 4 cluster based on a configuration file. -|VSHN / Red Hat +|{infra-type} infrastructure operator -|Bootstrap Node -|A temporary VM in the cluster machine network which is provisioned by the installer to facilitate the initial setup of the cluster. -This VM is decommissioned by the installer once the cluster installation is completed. -| VSHN / Installer - -|vSphere & vCenter +|{infra-type} & vCenter a|VMWare virtualization platform. See the upstream documentation for https://docs.openshift.com/container-platform/latest/installing/installing_vsphere/installing-vsphere-installer-provisioned.html#installation-vsphere-infrastructure_installing-vsphere-installer-provisioned[supported versions], https://docs.openshift.com/container-platform/latest/installing/installing_vsphere/installing-vsphere-installer-provisioned.html#installation-vsphere-installer-network-requirements_installing-vsphere-installer-provisioned[network connectivity] and https://docs.openshift.com/container-platform/latest/installing/installing_vsphere/installing-vsphere-installer-provisioned.html#installation-vsphere-installer-infra-requirements_installing-vsphere-installer-provisioned[required permissions]. -Entries for "VMWare vSphere CSI Driver Operator" are required. -|vSphere infrastructure operator +Entries for "{infra-type} CSI Driver Operator" are required. +|{infra-type} infrastructure operator |Cluster machine network (sometimes "cluster network" or "machine network") a|An internal subnet, usually a `/24`, in which the OpenShift 4 cluster will be placed. @@ -247,27 +133,7 @@ OpenShift manages the floating IPs with VRRP. At minimum two IPs must be allocated as floating IPs. These two IPs are used for the Kubernetes API and the ingress router. -|vSphere infrastructure operator - -|Pod network -a|A subnet that's internal to the Openshift 4 cluster. -This subnet shouldn't be routable from outside the cluster. - -This subnet is managed by Cilium and is implemented with VXLAN traffic between the cluster VMs - -APPUiO Managed OpenShift 4 uses `10.128.0.0/14` as the pod network. -If the pod network IP range conflicts with existing subnets, the pod network IP range can be adjusted. -| VSHN / Cilium - -|Service network -a|A subnet that's internal to the OpenShift 4 cluster. -This subnet shouldn't be routable from outside the cluster. - -This subnet is managed by Cilium and is implemented with eBPF rules on the cluster VMs. - -APPUiO Managed OpenShift 4 uses `172.30.0.0/16` as the service network. -If the service network IP range conflicts with existing subnets, the service network IP range can be adjusted. -| VSHN / Cilium +|{infra-type} infrastructure operator |S3 compatible storage a|Various OpenShift components require S3 compatible storage. @@ -278,7 +144,7 @@ The main APPUiO Managed OpenShift 4 components that use object storage are * OpenShift integrated image registry * OpenShift logging stack * APPUiO Managed cluster backups -|Customer / vSphere infrastructure provider +|Customer / {infra-type} infrastructure provider |Access gateway |To access the OpenShift API and applications deployed on the cluster, two public IPs are required. @@ -287,30 +153,14 @@ The following forwarding is required: * For the ingress public IP, ports `80/tcp` and `443/tcp` must be forwarded to the "Ingress VIP" in the machine network. * For the API public IP, port `6443/tcp` must be forwarded to the "API VIP" in the machine network. -|Customer / vSphere infrastructure provider - -|DNS -|The APPUiO Managed OpenShift 4 cluster's base DNS records are defined and managed by VSHN. -All records must be publicly resolvable. -To expose applications under a customer domain, a CNAME target is provided. -| VSHN +|Customer / {infra-type} infrastructure provider |=== -=== Other terms +=== Components General -[cols="1,4"] -|=== -|Name|Description - -|Node -|A virtual machine that's part of an OpenShift 4 cluster - -|Control plane -a|A collection of components that +include::partial$architecture/glossary-general.adoc[] -* facilitate the management of the container platform -* manage the virtual hardware making up the cluster -* manage the applications running on the cluster +=== Other terms -|=== +include::partial$architecture/glossary-others.adoc[] diff --git a/docs/modules/ROOT/partials/architecture/glossary-general.adoc b/docs/modules/ROOT/partials/architecture/glossary-general.adoc new file mode 100644 index 00000000..becd3b1e --- /dev/null +++ b/docs/modules/ROOT/partials/architecture/glossary-general.adoc @@ -0,0 +1,40 @@ +[cols="1,3,1"] +|=== +|Name|Description|provided by + +|Installer +|A CLI tool that bootstraps an OpenShift 4 cluster based on a configuration file. +|VSHN / Red Hat + +|Bootstrap Node +|A temporary VM in the cluster machine network which is provisioned by the installer to facilitate the initial setup of the cluster. +This VM is decommissioned by the installer once the cluster installation is completed. +| VSHN / Installer + +|Pod network +a|A subnet that's internal to the Openshift 4 cluster. +This subnet shouldn't be routable from outside the cluster. + +This subnet is managed by Cilium and is implemented with VXLAN traffic between the cluster VMs + +APPUiO Managed OpenShift 4 uses `10.128.0.0/14` as the pod network. +If the pod network IP range conflicts with existing subnets, the pod network IP range can be adjusted. +| VSHN / Cilium + +|Service network +a|A subnet that's internal to the OpenShift 4 cluster. +This subnet shouldn't be routable from outside the cluster. + +This subnet is managed by Cilium and is implemented with eBPF rules on the cluster VMs. + +APPUiO Managed OpenShift 4 uses `172.30.0.0/16` as the service network. +If the service network IP range conflicts with existing subnets, the service network IP range can be adjusted. +| VSHN / Cilium + +|DNS +|The APPUiO Managed OpenShift 4 cluster's base DNS records are defined and managed by VSHN. +All records must be publicly resolvable. +To expose applications under a customer domain, a CNAME target is provided. +| VSHN + +|=== diff --git a/docs/modules/ROOT/partials/architecture/glossary-others.adoc b/docs/modules/ROOT/partials/architecture/glossary-others.adoc new file mode 100644 index 00000000..51f60e35 --- /dev/null +++ b/docs/modules/ROOT/partials/architecture/glossary-others.adoc @@ -0,0 +1,15 @@ +[cols="1,4"] +|=== +|Name|Description + +|Node +|A virtual machine that's part of an OpenShift 4 cluster + +|Control plane +a|A collection of components that + +* facilitate the management of the container platform +* manage the virtual hardware making up the cluster +* manage the applications running on the cluster + +|=== diff --git a/docs/modules/ROOT/partials/architecture/networking-cluster.adoc b/docs/modules/ROOT/partials/architecture/networking-cluster.adoc new file mode 100644 index 00000000..14ec45cf --- /dev/null +++ b/docs/modules/ROOT/partials/architecture/networking-cluster.adoc @@ -0,0 +1,8 @@ +Each APPUiO Managed OpenShift 4 cluster is deployed into a /24 "cluster machine network" (sometimes also "cluster network" or "machine network") +This network must be provided by the {infra-type} infrastructure operator. +DHCP is mandatory for this network, but a number of IPs must be reserved to be used as Virtual IPs for the cluster. + +Traffic inside this network shouldn't be restricted. + +VMs in this network must be able to reach various services on the internet. +See below for a detailed list of external systems that must be reachable. diff --git a/docs/modules/ROOT/partials/architecture/networking-external.adoc b/docs/modules/ROOT/partials/architecture/networking-external.adoc new file mode 100644 index 00000000..11c622e8 --- /dev/null +++ b/docs/modules/ROOT/partials/architecture/networking-external.adoc @@ -0,0 +1,41 @@ +APPUiO Managed OpenShift 4 requires various external services. + +==== VSHN services + +APPUiO Managed OpenShift 4 requires access to VSHN's https://syn.tools[Project Syn] infrastructure. +The Project Syn infrastructure components that must be reachable are + +* the Project Syn API at `\https://api.syn.vshn.net` +* the Project Syn Vault at `\https://vault-prod.syn.vshn.net` +* VSHN's GitLab instance at `ssh://git@git.vshn.net` +* VSHN's acme-dns instance at `\https://acme-dns-api.vshn.net` + +Additionally, APPUiO Managed OpenShift 4 requires access to VSHN's identity management: + +* VSHN LDAP at `ldaps://ldap.vshn.net:636` +* VSHN SSO at `\https://id.vshn.net` + +Finally, APPUiO Managed OpenShift 4 requires access to VSHN's central metrics storage at `\https://metrics-receive.appuio.net` + +==== Red Hat services + +See the https://docs.openshift.com/container-platform/4.14/installing/install_config/configuring-firewall.html#configuring-firewall_configuring-firewall[upstream documentation] for the full list of services. + +The most important services for APPUiO Managed OpenShift 4 are + +* the Red Hat container registries at `registry.redhat.io` and `registry.access.redhat.com`. +* the OpenShift Update Service (OSUS) at `\https://api.openshift.com`. + +==== 3rd party services + +Finally, APPUiO Managed OpenShift 4 requires access to a number of third party services: + +* OpsGenie at `\https://api.opsgenie.com` +* Passbolt at `\https://cloud.passbolt.com/vshn` +* Let's Encrypt at `\https://acme-v02.api.letsencrypt.com` and `\https://acme-staging-v02.api.letsencrypt.com` +* Various container registries +** GitHub at `ghcr.io` +** Quay at `quay.io` +** DockerHub at `docker.io` +** Google container registry at `gcr.io` +** Kubernetes container registry at `registry.k8s.io` diff --git a/docs/modules/ROOT/partials/architecture/networking-pods.adoc b/docs/modules/ROOT/partials/architecture/networking-pods.adoc new file mode 100644 index 00000000..1e8adcba --- /dev/null +++ b/docs/modules/ROOT/partials/architecture/networking-pods.adoc @@ -0,0 +1,19 @@ +APPUiO Managed Openshift 4 uses https://cilium.io/[Cilium] to provide in-cluster networking. +Cilium allocates two cluster-internal networks: + +1. The pod network: every pod on the cluster will get an IP address from this network. +This network enables basic in-cluster connectivity. +APPUiO Managed OpenShift 4 uses `10.128.0.0/14` as the pod network. +Each node in the cluster is assigned a `/23` from this range. +Pods on a node are always assigned an IP from the range allocated to that node. +2. Service network: used for service discovery. +Traffic to IPs in this network is forwarded to the appropriate pods by Cilium. +APPUiO Managed OpenShift 4 uses `172.30.0.0/16` as the service network. + +Both of these networks are interanl to the OpenShift 4 cluster. +Therefore, the IP CIDRs for these networks must not be routable from the outside. +Additionally, the same IP CIDRs can be reused for multiple OpenShift 4 clusters. + +However, the chosen CIDRs shouldn't overlap with existing networks allocated by the customer. +If there are overlaps, external systems in the overlapping ranges won't be accessible from within the OpenShift 4 cluster. +The pod and service network CIDRs can be customized if and only if there are conflicts. diff --git a/docs/modules/ROOT/partials/architecture/overview.adoc b/docs/modules/ROOT/partials/architecture/overview.adoc new file mode 100644 index 00000000..ebe482bd --- /dev/null +++ b/docs/modules/ROOT/partials/architecture/overview.adoc @@ -0,0 +1,10 @@ +The APPUiO Managed OpenShift 4 architecture on {infra-type} is based on the xref:references/architecture/index.adoc[generic APPUiO Managed OpenShift 4 architecture]. +We expect that readers of this document are familiar with the xref:references/architecture/index.adoc[generic APPUiO Managed OpenShift 4 architecture] and the overall Kubernetes and OpenShift 4 architecture. + +This architecture document extends upon the generic architecture by defining how the APPUiO Managed OpenShift 4 cluster is embedded into the {infra-type} environment. +The diagram below shows a detailed view of how an APPUiO Managed OpenShift 4 cluster is embedded into the {infra-type} environment. + +.APPUiO Managed OpenShift4 on {infra-type} architecture +image::{infra-svg}[alt=OCP4 {infra-type} architecture, width=900] + +The following sections of this document provide detailed descriptions for the elements shown in the architecture diagram. diff --git a/docs/modules/ROOT/partials/architecture/storage.adoc b/docs/modules/ROOT/partials/architecture/storage.adoc new file mode 100644 index 00000000..5a35120e --- /dev/null +++ b/docs/modules/ROOT/partials/architecture/storage.adoc @@ -0,0 +1,34 @@ +APPUiO managed OpenShift 4 requires 3 different types of storage: + +1. Root disks +2. Persistent volumes +3. S3 compatible object storage + +=== Root disks + +Root disks are virtual block devices (100 GiB) which are attached to the VMs which make up the APPUiO Managed OpenShift 4 cluster. +The root disks are allocated and attached to the VM when the VM is created. +They hold the operating system and temporary data. +They're ephemeral (no application data is stored on them), and don't need to be backed up. +Finally, root disks are deleted when the VM to which they're attached is deleted. + +=== Persistent volumes + +Persistent volumes are virtual block devices with arbitrary sizes. +They're allocated dynamically based on requests from workloads (applications or infrastructure components) within the cluster. +These block devices are automatically attached to the VM hosting the application container. +They're deleted when the corresponding Kubernetes `PersistentVolume` resource is deleted. + +The {infra-type} CSI driver is the in-cluster component which is responsible for allocating, attaching and deleting the persistent volume block devices. + +These devices hold application data, but backups are usually done from within the cluster. + +=== S3 compatible object storage + +Various OpenShift components, such as the integrated image registry, the logging stack and backups, require S3 compatible object storage. +The customer or vSphere infrastructure operator must provide S3 compatible object storage. +Most modern storage solutions offer some object storage functionality. + +If https://products.vshn.ch/appcat/index.html[VSHN's Application Catalog (AppCat)] offering is required on the cluster, the object storage must support automatic bucket creation via an AppCat-supported provisioner. + +NOTE: If no object storage is available, we can use external object storage as a fallback.