Merge pull request #87 from aws-samples/3.0.0

3.0.0

CHANGELOG.MD

@@ -2,6 +2,27 @@
 # Change Log
 All notable changes to this project will be documented in this file.
 
+## [3.0.0] - 2021-05-10
+
+### Upgrade Notes: 
+- the Big Blue Button version defaults now to 2.3.0. if you need to stay on 2.2.X please make sure to alter the BBB version parameter to xenial-22 and the Ubuntu version parameter to xenial-16.04. 
+- the deployment now defaults to "Serverless" deployments for ECS and Aurora. this will optimize costs especially in small/test environments. switching from instanced to serverless will resetup the database means you will have to restore the DB from a backup afterwards or start fresh with testings. to prevent this, ensure you're not just using defaults for your environment.
+- now you should have docker locally installed when planning a scalable deployment. this is due our approach to mirror the needed images to ensure availability when using scaling of the frontend and Scalelite LBs
+
+### Fixed
+- #75: Serverless Aurora did not deploy due a missing !IF condition for the parametergroup 
+- #81: Single deployments failed due a missing condition
+
+### Changed
+- #84: Default Big Blue Button version defaults to 2.3.0 (stable since 7 Days)
+- #77: SES automation is now optional and can be overwritten by using the parameter
+- #77: an existing ACM Certificate ARN can be added to reuse an existing or imported SSL certificate
+- #76: Cloudformation can now validate ACM certificates natively, so we do not need the custom resource anymore.
+- #85: we're now utilizing IAM Roles when mounting the Shared Storage. Also the Storage now limits access to the APs utilizing these Roles. 
+- #86: adding ECR as container registry to prevent failures due dockerhub limitations
+- Versions of Greenlight and Scalelite are now current and reflect the new default Big Blue Button Version
+- Fargate deployment version of Scalelite was broken, this is now fixed. 
+
 ## [2.3.0] - 2021-03-10
 
 ### Upgrade Notes: 

README.md

@@ -49,6 +49,7 @@ To run the automated, [setup script](./setup.sh) based deployment you need to ha
 - an [installed and configured ](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) aws-cli
 - [a named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) at the aws-cli configuration reflecting the account you are planning to use for the deployment
 - [jq](https://stedolan.github.io/jq/)
+- [docker](https://www.docker.com/) 
 
 To run the step-by-step setup: 
 
@@ -125,6 +126,8 @@ The deployment parameters are placed into the bbb-on-aws-param.json or to be set
 | BBBScaleliteMemory | 2048 | Memory limit for the Scalelite tasks | setting per task for all inheritated containers
 | BBBScaleliteCPU | 1024 | vCPU limit for the Scalelite tasks if deployed | setting once per task for all containers
 | BBBSesRegion| - | Region of the SES Service to be used | if the setup is planned to be deployed in a Region w/o Amazon SES, choose a proper region here. 
+| BBBSESValidated| false | controls if a pre validated SES domain is used | set to true if you setup the SES domain outside of this deployment 
+| BBBACMCertArn | - | existing SSL/TLS Certificate ARN for HTTPS | add your Certificate ARN here. e.g. if you imported your own Cert into ACM. 
 
 # Deployment
 
@@ -335,7 +338,6 @@ Be sure NOT to use the main branch if you want to be sure not pulling potential
     - [Scalelite](https://github.com/blindsidenetworks/scalelite)
     - [Coturn](https://github.com/coturn/coturn)
     - [cfn-ses-provider](https://github.com/binxio/cfn-ses-provider)
-    - [cfn-certificate-provider](https://github.com/binxio/cfn-certificate-provider)
     - [cli53](https://github.com/barnybug/cli53)
 
 # Security
@@ -356,6 +358,5 @@ Please be aware of the deviating licenses of the deployed open-source software c
 - cli53: [MIT License](https://github.com/barnybug/cli53/blob/master/LICENSE)
 - Coturn: [New BSD License](https://github.com/coturn/coturn/blob/master/LICENSE)
 - cfn-ses-provider - [Apache-2.0 License](https://github.com/binxio/cfn-ses-provider/blob/master/LICENSE)
-- cfn-certificate-provider - [Apache-2.0 License](https://github.com/binxio/cfn-certificate-provider/blob/master/LICENSE)
 
 Please note that running this code will cause software that is licensed under [AGPL-3.0](https://www.gnu.org/licenses/agpl-3.0.de.html) to be deployed in the user’s account.

bbb-on-aws-master.template.yaml

@@ -107,7 +107,7 @@ Parameters:
   BBBECSInstanceType:
     Description: Instance type for ECS Cluster worker nodes
     Type: String
-    Default: t3a.large
+    Default: fargate
     AllowedValues:
       - fargate
       - t3a.small
@@ -321,12 +321,13 @@ Parameters:
       - db.r4.4xlarge
       - db.r4.8xlarge
       - db.r4.16xlarge
+      - db.t3.micro
       - db.t3.small
       - db.t3.medium
       - db.t3.large
       - db.t3.xlarge
       - db.t3.2xlarge
-    Default: db.t3.medium
+    Default: serverless
   BBBServerlessAuroraMinCapacity:
     Description: The minimum capacity for the Amazon Aurora Serverless Cluster.
     Type: Number
@@ -394,6 +395,10 @@ Parameters:
     Description: Scalelite Importer docker image identifier
     Type: String
     Default: "blindsidenetwks/scalelite:v1-recording-importer"
+  BBBECRRegistry: 
+    Description: ECR Registry to be used. 
+    Type: String
+    Default: ""
   BBBCacheAZMode:
     Description: Redis Cache AZ Mode
     Type: String
@@ -455,6 +460,15 @@ Parameters:
     Description: Set the Region for the SES Service if deployment in a non-SES Region
     Type: String
     Default: "AWS::NoValue"
+  BBBSESValidated: 
+    Description: If the SES Domain is already validated manually we do not need the automation
+    Type: String
+    Default: false
+    AllowedValues: [ true, false ]
+  BBBACMCertArn: 
+    Description: Set this parameter if you want to reuse an existing ACM certificate for your deployment
+    Type: String
+    Default: "AWS::NoValue"
 
 Conditions:
   BBBProdEnvironment: !Equals [!Ref BBBEnvironmentStage, prod]
@@ -463,6 +477,7 @@ Conditions:
   BBBSingleEnvironment: !Equals [!Ref BBBEnvironmentType, single]
   BBBExistingVPC: !And [ !Not [ !Equals [ !Ref BBBVPCs, "" ]], !Equals [ !Select [ 0, !Split [ vpc-, !Ref BBBVPCs ]], "" ]]
   BBBNewVPC: !Not [ Condition: BBBExistingVPC ]
+  BBBSESNotValidated: !Equals [false, !Ref BBBSESValidated]
 
 Resources:
 
@@ -486,28 +501,6 @@ Resources:
       LogGroupName: !Join ["", ["/", !Ref "AWS::StackName", "/applicationlogs"]]
       RetentionInDays: 7
 
-  BBBACMCertStack:
-    Type: AWS::CloudFormation::Stack
-    Condition: BBBScalableEnvironment
-    Properties:
-      TemplateURL:
-        Fn::Join:
-          - ""
-          - - "https://s3.amazonaws.com/"
-            - Fn::ImportValue: !Sub "${BBBStackBucketStack}-BBBStackBucket"
-            - "/bbb-on-aws-acmcert.template.yaml"
-      Parameters:
-        BBBEnvironmentStage:
-          Ref: BBBEnvironmentStage
-      Tags:
-        - Key: Name
-          Value:
-            Fn::Sub: BBBACMCertStack-${BBBEnvironmentName}
-        - Key: Environment
-          Value: !Ref BBBEnvironmentName
-        - Key: Stage
-          Value: !Ref BBBEnvironmentStage
-
   BBBAMIFinderStack:
     Type: AWS::CloudFormation::Stack
     Properties:
@@ -1090,6 +1083,8 @@ Resources:
             - Outputs.BBBRDSDBConnectionSecret
         BBBEnvironmentStage:
           Ref: BBBEnvironmentStage
+        BBBSESValidated: 
+          Ref: BBBSESValidated
         BBBECSCluster:
           Fn::GetAtt:
             - BBBECSStack
@@ -1111,10 +1106,8 @@ Resources:
           Fn::GetAtt:
             - BBBSecurityGroupStack
             - Outputs.BBBScaleliteELBSecurityGroup
-        BBBACMCertProviderArn: 
-          Fn::GetAtt:
-            - BBBACMCertStack
-            - Outputs.BBBACMCertProviderArn
+        BBBACMCertArn:
+          Ref: BBBACMCertArn
         BBBSESProviderArn: 
           Fn::GetAtt:
             - BBBSESProviderStack
@@ -1135,6 +1128,8 @@ Resources:
           Ref: BBBScalelitePollerImage
         BBBScaleliteImporterImage:
           Ref: BBBScaleliteImporterImage
+        BBBECRRegistry: 
+          Ref: BBBECRRegistry
         BBBCacheDBAddress:
           Fn::GetAtt:
             - BBBCacheDBStack

bbb-on-aws-param.json

@@ -1,14 +1,14 @@
 {
   "Parameters" : {
-    "BBBApplicationVersion": "xenial-22",
-    "BBBApplicationInstanceOSVersion": "xenial-16.04",
+    "BBBApplicationVersion": "bionic-230",
+    "BBBApplicationInstanceOSVersion": "bionic-18.04",
     "BBBTurnInstanceOSVersion": "focal-20.04",
-    "BBBECSInstanceType": "t3a.medium",
+    "BBBECSInstanceType": "fargate",
     "BBBApplicationInstanceType": "t3a.medium",
     "BBBApplicationDataVolumeSize": 50,
     "BBBApplicationRootVolumeSize": 20,
     "BBBTurnInstanceType": "t3a.micro",
-    "BBBDBInstanceType": "db.t3.medium",
+    "BBBDBInstanceType": "serverless",
     "BBBServerlessAuroraMinCapacity": 2,
     "BBBServerlessAuroraMaxCapacity": 4,
     "BBBCACHEDBInstanceType": "cache.t3.micro",
@@ -27,19 +27,21 @@
     "BBBTurnMinInstances": 1,
     "BBBTurnDesiredInstances": 1,
     "BBBDBName": "frontendapp",
-    "BBBDBEngineVersion": "12.4",
+    "BBBDBEngineVersion": "10.12",
     "BBBEnvironmentStage": "dev",
     "BBBEnvironmentName": "bbbonaws",
     "BBBEnvironmentType": "scalable",
-    "BBBgreenlightImage": "bigbluebutton/greenlight:release-2.7.20",
-    "BBBScaleliteApiImage": "blindsidenetwks/scalelite:v1.0.7-api",
-    "BBBScaleliteNginxImage": "blindsidenetwks/scalelite:v1.0.7-nginx",
-    "BBBScalelitePollerImage": "blindsidenetwks/scalelite:v1.0.7-poller",
-    "BBBScaleliteImporterImage": "blindsidenetwks/scalelite:v1.0.7-recording-importer",
+    "BBBgreenlightImage": "bigbluebutton/greenlight:v2",
+    "BBBScaleliteApiImage": "blindsidenetwks/scalelite:v1.1-beta.4-api",
+    "BBBScaleliteNginxImage": "blindsidenetwks/scalelite:v1.1-beta.4-nginx",
+    "BBBScalelitePollerImage": "blindsidenetwks/scalelite:v1.1-beta.4-poller",
+    "BBBScaleliteImporterImage": "blindsidenetwks/scalelite:v1.1-beta.4-recording-importer",
     "BBBCacheAZMode": "cross-az",
     "BBBGreenlightMemory": 1024,
     "BBBGreenlightCPU": 512,
     "BBBScaleliteMemory": 2048,
     "BBBScaleliteCPU": 1024
   }
 }
+
+

destroy.sh

@@ -62,6 +62,9 @@ echo "##################################################"
 aws cloudformation delete-stack --stack-name $BBBPREPSTACK --profile=$BBBPROFILE
 aws cloudformation wait stack-delete-complete --profile=$BBBPROFILE --stack-name $BBBPREPSTACK
 
+aws cloudformation delete-stack --stack-name $BBBECRSTACK --profile=$BBBPROFILE
+aws cloudformation wait stack-delete-complete --profile=$BBBPROFILE --stack-name $BBBECRSTACK
+
 echo "##################################################"
 echo "Deletion done"
 

setup.sh

@@ -98,11 +98,57 @@ fi
 echo "##################################################"
 echo "File Copy finished"
 
+ENVIRONMENTTYPE=$(jq -r ".Parameters.BBBEnvironmentType" bbb-on-aws-param.json)
+
+if [ "$ENVIRONMENTTYPE" == 'scalable' ]
+then 
+  BBBECRStack="${BBBSTACK}-registry"
+  aws cloudformation deploy --profile=$BBBPROFILE --stack-name $BBBECRStack  \
+      --parameter-overrides $PARAMETERS \
+      $(jq -r '.Parameters | to_entries | map("\(.key)=\(.value)") | join(" ")' bbb-on-aws-param.json) \
+      --template ./templates/bbb-on-aws-registry.template.yaml
+
+  GREENLIGHTREGISTRY=`aws cloudformation describe-stacks --profile=$BBBPROFILE --query "Stacks[0].Outputs[0].OutputValue" --stack-name $BBBECRStack`
+  GREENLIGHTREGISTRY=`echo "${GREENLIGHTREGISTRYREGISTRY//\"}"`
+  SCALEILITEREGISTRY=`aws cloudformation describe-stacks --profile=$BBBPROFILE --query "Stacks[0].Outputs[1].OutputValue" --stack-name $BBBECRStack`
+  SCALEILITEREGISTRY=`echo "${SCALEILITEREGISTRY//\"}"`
+
+  # we will mirror the needed images from dockerhub and push towards ECR
+  echo "##################################################"
+  echo "Mirror docker images to ECR for further usage"
+  echo "##################################################"
+
+  IMAGES=( BBBgreenlightImage BBBScaleliteNginxImage BBBScaleliteApiImage BBBScalelitePollerImage BBBScaleliteImporterImage )
+
+  ACCOUNTID=$(aws sts get-caller-identity --query Account --output text --profile=$BBBPROFILE)
+  REGION=$(aws configure get region --profile=$BBBPROFILE)
+  REGISTRY=$ACCOUNTID.dkr.ecr.$REGION.amazonaws.com
+  SCALEILITEREGISTRY=$ACCOUNTID.dkr.ecr.$REGION.amazonaws.com/$SCALEILITEREGISTRY  
+  GREENLIGHTREGISTRY=$ACCOUNTID.dkr.ecr.$REGION.amazonaws.com/$GREENLIGHTREGISTRY  
+
+  aws ecr get-login-password --profile=$BBBPROFILE | docker login --username AWS --password-stdin $SCALEILITEREGISTRY
+  aws ecr get-login-password --profile=$BBBPROFILE | docker login --username AWS --password-stdin $GREENLIGHTREGISTRY
+
+  for IMAGE in "${IMAGES[@]}"
+  do
+    IMAGE=$(jq -r ".Parameters.$IMAGE" bbb-on-aws-param.json)
+    docker pull $IMAGE
+    docker tag $IMAGE $REGISTRY/$IMAGE
+    docker push $REGISTRY/$IMAGE
+  done
+
+  echo "##################################################"
+  echo "Registry Preperation finished"
+else
+  REGISTRY="Dockerhub"
+fi
+
 # Setting the dynamic Parameters for the Deployment
 PARAMETERS=" BBBOperatorEMail=$OPERATOREMAIL \
              BBBStackBucketStack=$BBBSTACK-Sources \
              BBBDomainName=$DOMAIN \
-             BBBHostedZone=$HOSTEDZONE"
+             BBBHostedZone=$HOSTEDZONE \
+             BBBECRRegistry=$REGISTRY"
 
 # Deploy the BBB infrastructure. 
 echo "Building the BBB Environment"
@@ -116,4 +162,4 @@ aws cloudformation deploy --profile=$BBBPROFILE --stack-name $BBBSTACK \
 echo "##################################################"
 echo "Deployment finished"
 
-exit 0 
+exit 0