devops-zerodowntime-manual.adoc

Zero-Downtime Deployment to Production

In this lab you will get familiar with production deployments without requiring any downtime or service windows.

Background

Although you have automated the build and deployment of the Cart service all the way from development to production, every time the pipeline performs the deployment in production the service is unavailable for the period it takes for the Cart container to start up. This might be acceptable for certain services however an unavailable service for an online shop equalis lost orders and revenue specially during shopping seasons like Christmas.

Zero downtime deployments refers to a number of deployment pattern that allow deploying to production environment during work hours without a need for service windows and without causing any disruption for the live services. Blue-Green and Canary Release are two popular patterns that are often used to achieve zero downtime deployments.

Canary Release refers to deploying a new version of your app into production parallel to the live version and redirectly only a fraction of production traffic to the new version. After verifying that the canary deployment works correctly, incremental more production traffic is switched to the canary version until it serves 100% of the production traffic. Canary Release allows you to test the application directly in the production with subset of users so that in case of failure, only a small user set will be affected.

In Blue-Green deployment approach you also have two parallel production environments but at any point in time only one of them is active and receives the production traffic, for example the blue one. When you deploy the new version of the application in production, you deploy into the inactive environment, green in this example, and after ensuring that the new version works correctly in production you switch the router so that all incoming production traffic goes to the green environment, making the blue one idle.

Blue-green deployment also enables you to be able to quickly rollback to the previous version. if anything goes wrong with the new version after the switch, you can switch back the router the blue version. You still need to manage transactions and database changes when deploying new versions of your application however there are other patterns to specifically deal with those issues separately for example depending on your environment and application design, you might be able to feed transactions to both environments so that the blue environment acts as a backup or run the application in read-only mode before doing a switch.

Blue-Green Deployment

In this lab, you will perform a Blue-Green deployment to push a code change into production without requiring any downtime. For a Blue-Green deployments, you need two production environments running in parallel. Since the CI/CD pipeline in this lab builds and deploys the Cart service, instead of two Prod environment for the entire CoolStore application, you can run two parallel containers for the Cart service: Cart Blue and Cart Green.

Start with tagging the cart:prod image into a blue and green version so that we can update one without affecting the other one and the deployed container that is based on that:

Caution

Replace {{PROJECT_SUFFIX}} with the number provided to you by the instructor.

$ oc project prod-{{PROJECT_SUFFIX}}
$ oc tag cart:prod cart:prod-blue
$ oc tag cart:prod cart:prod-green

You need to create two deployment configs as well, one based on the cart:prod-blue and another based on cart:prod-green image. Also two services that send traffic to the containers deployed by each of the Blue and Green deployment configs. These services and deployment configs are identical to the currently deployed cart service however use the newly tagged images instead. The Cart route will then point to the Cart Blue or Cart Green service in order to send the production traffic to the appointed container.

In order to create the blue and green Cart service and deployment configs, we can export the current Cart service and deployment configs as YAML or JSON, modify them and then deploy the modified definitions. Use the oc export CLI command to export the Cart service and deploymentconfig:

$ oc get svc/cart dc/cart -o yaml --export > ~/cart-blue.yaml

Open the cart-blue.yaml file in a text editor and replace all occurrences of cart with cart-blue. After replacement, make sure the referenced image tag is cart:prod-blue which you created in the previous steps:

- imageChangeParams:
    automatic: true
    containerNames:
    - cart-blue
    from:
      kind: ImageStreamTag
      name: cart:prod-blue
      namespace: prod-{{PROJECT_SUFFIX}}
  type: ImageChange

Alternatively if you are a bash aficionados, use the sed command for the replacement:

Linux

$ sed -i "s/cart$/cart-blue/g;s/cart:prod/cart:prod-blue/g" ~/cart-blue.yaml

Mac

$ sed -i .bak "s/cart$/cart-blue/g;s/cart:prod/cart:prod-blue/g" ~/cart-blue.yaml

The Cart blue service and deployment config definition is ready. You can create the Cart green service and deployment config definitions by duplicating ~/cart-blue.yaml to ~/cart-green.yaml and replacing all occurrences of blue with green:

$ sed "s/blue/green/g" ~/cart-blue.yaml > ~/cart-green.yaml

You are all set to create the blue and green services and deployment configs for the Cart service:

$ oc create -f ~/cart-blue.yaml
$ oc create -f ~/cart-green.yaml

Now that the Cart Blue and Cart Green deployment and services are defined, you can deploy those containers in the the Prod environment. Click on Applications → Deployments and then on cart-blue. Deploy the Cart Blue service by clicking on the Deploy button. Go back to the list of deployments and click on cart-green, and then Deploy button to also deploy the Cart Green service.

The Cart Blue and Green containers will be deployed alongside the previous Cart container.

You created the Cart blue and green services but the route is still pointing to the old container instead of either blue or the green container. Click on Applications → Routes on the left sidebar menu and then click on cart route. Notice that it is currently pointing at the cart service. Click on Actions → Edit from the top-right menu. You can create a blue/green route by clicking on Split traffic across multiple services and adding cart-blue and cart-green services with 100 and 0 weight respectively to send all traffic to the cart-blue service and then click on Save. You can {{OPENSHIFT_DOCS_BASE}}/dev_guide/routes.html#routes-load-balancing-for-AB-testing[split the traffic] coming to a route across multiple backend services via weighting however in this lab you only need the cart-blue and cart-green services.

Split routes in OpenShift enables other deployment patterns such as Canary Release and A/B testing as well by specifying other weight ratios on the services for example:

• 10/90: to send 10% of production traffic to the new container for Canary Release

• 50/50: to send 50% of traffic to the new container for A/B Testing

The blue/green deployment setup is complete now. Clean up the Prod environment by removing the Cart service and deploymentconfig since you will be using the green and blue versions instead.

$ oc delete svc/cart dc/cart

You can also expose the cart-blue and cart-green services as their own dedicated routes in order to facilitate testing even though these routes won’t be used by the CoolStore application.

$ oc expose svc/cart-blue
$ oc expose svc/cart-green
$ oc get routes

Verify the routes are working using the curl command:

Caution

Replace the urls with routes in your project

$ curl http://{{CART_ROUTE}}/api/cart/dummy
$ curl http://{{CART_BLUE_ROUTE}}/api/cart/dummy
$ curl http://{{CART_GREEN_ROUTE}}/api/cart/dummy

Showtime! Let’s perform your first blue/green deployment. Point your browser to Git server web URL and log in (you should be already logged in by now!) with your credentials:

• Git server: http://{{GIT_SERVER_URL}}

• Username: {{GIT_USER}}

• Password: {{GIT_PASSWORD}}

Click on cart-service under My Repositories to go to the repository overview. Browse to the promotion service at src/main/java/com/redhat/coolstore/service/PromoService.java which is responsible for calculating order promotions depending on the total amount of products a customer has added to their shopping cart.

public void applyShippingPromotions(ShoppingCart shoppingCart) {
    if ( shoppingCart != null ) {
        //PROMO: if cart total is greater than 75, free shipping
        if ( shoppingCart.getCartItemTotal() >= 75) {
            shoppingCart.setShippingPromoSavings(shoppingCart.getShippingTotal() * -1);
            shoppingCart.setShippingTotal(0);
        }
    }
}

Notice that the minimum order for free shipping is currently 75$. In order to match a competitor website, you want to change the promotion rules and reduce the minimum to 40$. Click on the pencil icon to open PromoService.java in the web-based editor.

Change the minium order to 40$. The PromoService.java should look like this after the edit:

        //PROMO: if cart total is greater than 40, free shipping
        if ( shoppingCart.getCartItemTotal() >= 40) {
            ...
        }

Click on Commit Changes button to commit the new shipping promotion rule to the Git repository. As soon as the change is committed to the Git repository, the Cart CI/CD pipeline gets automatically triggered via the webhook you defined in the previous lab and starts building the change and pushing it through Dev and Prod environments. Click on Builds → Pipelines on the left sidebar menu and wait until pipeline pauses at the Approve stage. The Cart pipeline promotes the Cart container image to the Prod environment by tagging it as cart:prod. However you have changed the production topology in Prod environment to blue/green deployment and therefore the application promotion process also needs adapt the new topology which you will do in the next lab.

In this lab, you can manually promote the image to Prod environment instead of the automatic promotion via the CI/CD pipeline. Click on Input Required and then click on No button to reject the deployment. Use oc tag CLI command to promote the newly built Cart container image to the green Cart deployment which is currently inactive and does not receive any production traffic.

$ oc tag dev-{{PROJECT_SUFFIX}}/cart:latest prod-{{PROJECT_SUFFIX}}/cart:prod-green

Go to CoolStore Prod project in OpenShift Web Console and notice a new deployment is taking place:

After deployment is ready, verify that the new minimum order for free shipping is working correctly in the green container by adding 2 smart watches to the test shopping cart:

Caution

Replace the Cart route with routes in your project

$ curl -X POST http://{{CART_GREEN_ROUTE}}/api/cart/FOO/444434/2
{"cartItemTotal":48.0,"cartItemPromoSavings":0.0,"shippingTotal":0.0,"shippingPromoSavings":-4.99,"cartTotal":52.99,"shoppingCartItemList":[{"price":24.0,"quantity":2,"promoSavings":0.0,"product":{"itemId":"444434","name":"Pebble Smart Watch","desc":"Smart glasses and smart watches are perhaps two of the most exciting developments in recent years. ","price":24.0}}]}

Notice that the shipping cost is zero since the total order is above the 40$ minimum order. However the change is still not live and invoking the production endpoints for the Cart service should still have the 75$ minimum order. Click on the Web UI route url and add 2 Pebble Smart Watches to your shopping cart. As expected, the shipping cost is not zero.

Now that the new minimum order rule is verified in production, you can switch the router to send 100% of traffic to the Cart green service which includes the new promotion rules and 0% traffic to the Cart blue service.

$ oc set route-backends cart cart-green=100 cart-blue=0

Go back to CoolStore add 2 Pebble Smart Watches to your shopping cart and then go to the shopping cart. Happy Shopping! Shipping is free now for this order.

When using blue/green deployments, rolling back to a previous version is as simple as changing the traffic weights to send 100% of traffic to the previous version (blue service in this case) and 0% to the new version (green service in this case).