Deploying CaaSP CAP on ECP

I spent many days testing different deployment configurations on ECP, because ever since the introduction of manually configuring pod security policies (PSPs) I have not had a successful deployment, and we doc peeps are getting conflicting information. The basis for my testing is Setup CAP on CaaSP on ECP. (Refer to this page if you have any questions about the steps in this document.) Prabal's scripts automate creating an NFS storage class and applying PSPs. I forked SUSE/cf-ci to test various PSP configurations. The steps on this page create a successful deployment using the original SUSE/cf-ci configurations, but the PSPs do not make sense to me as it seems the end result is akin to mode 0777.

I tried upgrading CaaSP 3.0 (transactional-update up), and my CAP deployments failed. I do not know why, and have not had time yet to test different PSP configurations. So the following steps are on the stock CaaSP 3.0 GMC image in ECP.

This is all fragile, and fixing a damaged deployment is difficult, so the CAP Guides must present exact perfect steps for customers to get it right the first time.

1. Create a CaaSP cluster on ECP:

$ git clone https://github.com/prabalsharma/automation.git
$ cd automation/caasp-openstack-heat

Edit heat-environment.yaml.example with your DNS server and desired internal Kube cluster network range. Do not overlap with the CaaSP defaults of 172.16.0.0/13 and 172.24.0.0/16. Current usable DNS servers are 10.84.2.20, 10.84.2.21, and 10.84.100.100.

This is my heat-environment.yaml.example file:

---
parameters:
  root_password: password
  admin_flavor: m1.large
  master_flavor: m1.xlarge
  worker_flavor: m1.xlarge
  external_net: floating
  internal_net_cidr: 172.24.8.0/24
  dns_nameserver: 10.84.100.100
  worker_num_volumes: 0
  worker_volume_size: 60

Create your cluster with this command:

./caasp-openstack --build -m 1 -w 3 --openrc <path to your ECP openrc.sh> --image CaaSP-3.0.0-GMC --name <your stack name>

2. When you see Velum started!, open a Web browser to the floating IP address assigned to the admin node + omg.howdoi.website, e.g. https://10.86.2.234.omg.howdoi.website. Use the internal IP address for the internal dashboard location address, and check the box to install Tiller. Continue through the screens for selecting nodes, and on the screen for configuring the External Kubernetes API FQDN and External Dashboard FQDN, use the master and admin floating IP addresses + omg.howdoi.website, then bootstrap the cluster.

3. After the new CaaSP cluster has bootstrapped, download your Kubeconfig file and verify that you can connect to the cluster with kubectl get nodes. Then apply the SUSE/cf-ci scripts to set up PSPs and create an NFS storage class on the worker nodes by opening an SSH session to your master node, cloning the cf-ci repo, and running the cluster prep script:

ssh root@<master-ip>
password: password
git clone https://github.com/SUSE/cf-ci.git
bash cf-ci/automation-scripts/prep-new-cluster.sh

Run kubectl get storageclass and kubectl get psp to verify.

4. Configuring the CAP deployment In ECP, attach a floating IP address to one of your worker nodes. In a real production deployment there would be a load balancer or ingress controller. For quick testing and not hassling with DNS, use a worker node. Then use this as your domain address in your scf-config-value.yaml file.

After months of confusion, I finally figured out a working configuring for the external_ips:. This is the internal IP addresses that expose services externally. Use the internal IP addresses of the worker nodes, and also enter your domain IP address, which is required for Stratos.

env:    
    DOMAIN: _10.86.1.7_.omg.howdoi.website
    UAA_HOST: uaa._10.86.1.7_.omg.howdoi.website
    UAA_PORT: 2793
    
kube:
    external_ips: ["_10.86.1.7_", "_172.24.8.6_", "_172.24.8.24_", "_172.24.8.15_"]
    
    storage_class: 
        persistent: "persistent"
        shared: "shared"
    
    registry: 
        hostname: "registry.suse.com"
        username: ""
        password: ""
    organization: "cap"
    
    auth: rbac    
    psp:
        privileged: "suse.cap.psp"
    
secrets:
    # Create a password for your CAP cluster
    CLUSTER_ADMIN_PASSWORD: password
    
    # Create a password for your UAA client secret
    UAA_ADMIN_CLIENT_SECRET: password