-
Notifications
You must be signed in to change notification settings - Fork 122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Abstract some ec2 CI logic and add support for c7g #1220
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
31 changes: 31 additions & 0 deletions
31
tests/ci/cdk/cdk/codebuild/ec2_test_framework_omnibus.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 OR ISC | ||
|
|
||
| version: 0.2 | ||
|
|
||
| # Doc for batch https://docs.aws.amazon.com/codebuild/latest/userguide/batch-build-buildspec.html#build-spec.batch.build-list | ||
| batch: | ||
| build-list: | ||
| # Actual tests are ran on an m1 ec2 instance via SSM Commands. | ||
| # MacOS is special since the ec2 instance can't be spun up and closed on the fly. | ||
| # TODO: Migrate this to Github Actions when https://github.com/actions/runner-images/issues/2187 is resolved. | ||
| - identifier: macos_arm_tests | ||
| buildspec: ./tests/ci/codebuild/macos-aarch/run_m1_tests.yml | ||
| env: | ||
| type: LINUX_CONTAINER | ||
| privileged-mode: true | ||
| compute-type: BUILD_GENERAL1_LARGE | ||
| image: 620771051181.dkr.ecr.us-west-2.amazonaws.com/aws-lc-docker-images-linux-x86:ubuntu-20.04_clang-7x-bm-framework_latest | ||
|
|
||
| # Actual tests are ran on an Graviton3 ec2 instance via SSM Commands. | ||
| - identifier: graviton3_tests | ||
| buildspec: ./tests/ci/codebuild/common/run_ec2_target.yml | ||
| env: | ||
| type: LINUX_CONTAINER | ||
| privileged-mode: true | ||
| compute-type: BUILD_GENERAL1_LARGE | ||
| image: 620771051181.dkr.ecr.us-west-2.amazonaws.com/aws-lc-docker-images-linux-x86:ubuntu-20.04_clang-7x-bm-framework_latest | ||
| variables: | ||
| EC2_AMI: "ami-0c79a55dda52434da" | ||
| EC2_INSTANCE_TYPE: "c7g.2xlarge" | ||
| ECR_DOCKER_TAG: "amazonlinux-2023_clang-15x_sanitizer" |
16 changes: 0 additions & 16 deletions
16
tests/ci/cdk/cdk/codebuild/github_ci_macos_m1_omnibus.yaml
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 OR ISC | ||
|
|
||
| --- | ||
| schemaVersion: '2.2' | ||
| description: aws-lc:bmFrameworkEc2Benchmark | ||
| mainSteps: | ||
| - action: aws:runShellScript | ||
| name: runShellScript | ||
| inputs: | ||
| timeoutSeconds: '7200' | ||
| runCommand: | ||
| - sudo -i | ||
| - export DEBIAN_FRONTEND=noninteractive | ||
| - export CPU_TYPE=$(dpkg --print-architecture) | ||
| # if we have a cpu type of x86, we want linux-x86 | ||
| - if [ "${CPU_TYPE}" = amd64 ]; then export CPU_ARCH=linux-x86; export AWS_CLI_PREFIX=x86_; sudo sh -c "echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo"; fi | ||
| # if we have a cpu type of arm, we want linux-aarch | ||
| - if [ "${CPU_TYPE}" = arm64 ]; then export CPU_ARCH=linux-aarch; export AWS_CLI_PREFIX=aarch; fi | ||
| # install aws-cli | ||
| - apt-get -y install unzip | ||
| - curl "https://awscli.amazonaws.com/awscli-exe-linux-${AWS_CLI_PREFIX}64.zip" -o "awscliv2.zip" | ||
| - unzip awscliv2.zip | ||
| - ./aws/install | ||
| - git clone {GITHUB_REPO} aws-lc-pr | ||
| - cd aws-lc-pr | ||
| - git checkout {COMMIT_ID} | ||
| # install docker if its not already installed | ||
| - chmod +x ./tests/ci/benchmark_framework/install_docker.sh | ||
| - ./tests/ci/benchmark_framework/install_docker.sh | ||
| # log into docker and get needed docker image from ecr | ||
| - export ECR_REPO="{AWS_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com/aws-lc-docker-images-${CPU_ARCH}" | ||
| - aws ecr get-login-password --region us-west-2 | docker login -u AWS --password-stdin "${ECR_REPO}" | ||
| - docker pull "${ECR_REPO}:{ECR_DOCKER_TAG}" | ||
| - # start the container and run the bm script | ||
| - exec_docker="docker run --env AWS_ACCOUNT_ID={AWS_ACCOUNT_ID} --env PR_NUM={PR_NUM} --env COMMIT_ID={COMMIT_ID} --env CPU_TYPE=${CPU_TYPE} -v `pwd`:`pwd` -w `pwd` ${ECR_REPO}:{ECR_DOCKER_TAG}_latest" | ||
| - chmod +x ./tests/ci/run_posix_sanitizers.sh | ||
skmcgrail marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - $exec_docker ./tests/ci/run_posix_sanitizers.sh | ||
| - chmod +x ./tests/ci/run_fips_tests.sh | ||
| - $exec_docker ./tests/ci/run_fips_tests.sh | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 OR ISC | ||
|
|
||
| version: 0.2 | ||
|
|
||
| env: | ||
| variables: | ||
| GOPROXY: https://proxy.golang.org,direct | ||
|
|
||
| phases: | ||
| build: | ||
| commands: | ||
| - ./tests/ci/run_ec2_test_framework.sh "${EC2_AMI}" "${EC2_INSTANCE_TYPE}" "${ECR_DOCKER_TAG}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| #!/usr/bin/env bash | ||
| set -exo pipefail | ||
| # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 OR ISC | ||
|
|
||
| # Please run from project root folder! | ||
| # You'll want to set the codebuild env variables set if running locally | ||
| source tests/ci/common_ssm_setup.sh | ||
|
|
||
| AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) | ||
|
|
||
| # Cleanup AWS resources. | ||
| cleanup() { | ||
| set +e | ||
| aws ec2 terminate-instances --instance-ids "${instance_id}" | ||
| aws ssm delete-document --name "${ssm_doc_name}" | ||
| } | ||
|
|
||
| generate_ssm_document_file() { | ||
| # use sed to replace placeholder values inside preexisting document | ||
| sed -e "s,{AWS_ACCOUNT_ID},${AWS_ACCOUNT_ID},g" \ | ||
| -e "s,{PR_NUM},${CODEBUILD_WEBHOOK_TRIGGER},g" \ | ||
| -e "s,{COMMIT_ID},${CODEBUILD_SOURCE_VERSION},g" \ | ||
| -e "s,{GITHUB_REPO},${CODEBUILD_SOURCE_REPO_URL},g" \ | ||
| -e "s,{ECR_DOCKER_TAG},${ecr_docker_tag},g" \ | ||
| tests/ci/cdk/cdk/ssm/general_test_run_ssm_document.yaml \ | ||
| > "tests/ci/cdk/cdk/ssm/${ec2_ami_id}_ssm_document.yaml" | ||
| } | ||
|
|
||
| #$1 for ami, $2 for instance-type, echos the instance id so we can capture the output | ||
| create_ec2_instances() { | ||
| local instance_id | ||
| instance_id="$(aws ec2 run-instances --image-id "$1" --count 1 \ | ||
| --instance-type "$2" --security-group-ids "${sg_id}" --subnet-id "${subnet_id}" \ | ||
| --block-device-mappings 'DeviceName="/dev/sda1",Ebs={DeleteOnTermination=True,VolumeSize=200}' \ | ||
| --tag-specifications 'ResourceType="instance",Tags=[{Key="aws-lc",Value="aws-lc-ci-ec2-test-framework-ec2-x86-instance"}]' \ | ||
| --iam-instance-profile Name=aws-lc-ci-ec2-test-framework-ec2-profile \ | ||
| --placement 'AvailabilityZone=us-west-2a' \ | ||
| --query Instances[*].InstanceId --output text)" | ||
| echo "${instance_id}" | ||
| } | ||
|
|
||
| trap cleanup EXIT | ||
|
|
||
| # print some information for reference | ||
| echo GitHub PR Number: "${CODEBUILD_WEBHOOK_TRIGGER}" | ||
| echo GitHub Commit Version: "${CODEBUILD_SOURCE_VERSION}" | ||
| echo AWS Account ID: "${AWS_ACCOUNT_ID}" | ||
| echo GitHub Repo Link: "${CODEBUILD_SOURCE_REPO_URL}" | ||
| export ec2_ami_id="$1" | ||
| export ec2_instance_type="$2" | ||
| export ecr_docker_tag="$3" | ||
| export s3_bucket_name="aws-lc-codebuild" | ||
|
|
||
| # Get resources for ec2 instances. These were created with the cdk script. | ||
| vpc_id="$(aws ec2 describe-vpcs --filter Name=tag:Name,Values=aws-lc-ci-ec2-test-framework/aws-lc-ci-ec2-test-framework-ec2-vpc --query Vpcs[*].VpcId --output text)" | ||
| sg_id="$(aws ec2 describe-security-groups --filter Name=vpc-id,Values="${vpc_id}" --filter Name=group-name,Values=codebuild_ec2_sg --query SecurityGroups[*].GroupId --output text)" | ||
| subnet_id="$(aws ec2 describe-subnets --filter Name=vpc-id,Values="${vpc_id}" --filter Name=state,Values=available --filter Name=tag:Name,Values=aws-lc-ci-ec2-test-framework/aws-lc-ci-ec2-test-framework-ec2-vpc/PrivateSubnet1 --query Subnets[*].SubnetId --output text)" | ||
|
|
||
| # create the ssm documents that will be used for the various ssm commands | ||
| generate_ssm_document_file | ||
|
|
||
| # create ec2 instances | ||
| instance_id=$(create_ec2_instances "${ec2_ami_id}" "${ec2_instance_type}") | ||
| if [[ -z "${instance_id}" ]]; then | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Give a few minutes for the ec2 instance to be ready | ||
| sleep 60 | ||
| for i in {1..30}; do | ||
| status=$(aws ssm describe-instance-information --filter Key="InstanceIds",Values="${instance_id}" \ | ||
| --query InstanceInformationList[*].PingStatus --output text) | ||
| if [ "${status}" == Online ]; then | ||
| break | ||
| fi | ||
| echo "Wait for instances to be able to run the SSM commands" | ||
|
|
||
| # if we've hit the 30 minute mark and still aren't ready, then something has gone wrong | ||
| if [ "${i}" = 30 ]; then exit 1; fi | ||
| sleep 60 | ||
| done | ||
|
|
||
|
|
||
| # Create, and run ssm command. | ||
| ssm_doc_name=$(create_ssm_document "${ec2_ami_id}") | ||
|
|
||
| cloudwatch_group_name="aws-lc-ci-ec2-test-framework-cw-logs" | ||
| ec2_test_ssm_command_id=$(run_ssm_command "${ssm_doc_name}" "${instance_id}" ${cloudwatch_group_name}) | ||
|
|
||
| run_url="https://${AWS_REGION}.console.aws.amazon.com/cloudwatch/home?region=${AWS_REGION}\ | ||
| #logsV2:log-groups/log-group/${cloudwatch_group_name}/log-events/\ | ||
| ${ec2_test_ssm_command_id}\$252F${instance_id}\$252FrunShellScript\$252Fstdout" | ||
|
|
||
| echo "Actual Run in EC2 can be observered at CloudWatch URL: ${run_url}" | ||
|
|
||
| # Give some time for the commands to run | ||
| done=false | ||
| success=false | ||
| for i in {1..45}; do | ||
| echo "${i}: Continue to wait 2 min for SSM commands to finish." | ||
| sleep 120 | ||
|
|
||
| ssm_command_status="$(aws ssm list-commands --command-id "${ec2_test_ssm_command_id}" --query Commands[*].Status --output text)" | ||
| ssm_target_count="$(aws ssm list-commands --command-id "${ec2_test_ssm_command_id}" --query Commands[*].TargetCount --output text)" | ||
| ssm_completed_count="$(aws ssm list-commands --command-id "${ec2_test_ssm_command_id}" --query Commands[*].CompletedCount --output text)" | ||
| if [[ ${ssm_command_status} == 'Success' && ${ssm_completed_count} == "${ssm_target_count}" ]]; then | ||
| echo "SSM command ${ec2_test_ssm_command_id} finished successfully." | ||
| success=true | ||
| done=true | ||
| elif [[ ${ssm_command_status} == 'Failed' && ${ssm_completed_count} == "${ssm_target_count}" ]]; then | ||
| echo "SSM command ${ec2_test_ssm_command_id} failed." | ||
| done=true | ||
| else | ||
| # Still running. | ||
| done=false | ||
| fi | ||
|
|
||
| # if after the loop finish and done is still true, then we're done | ||
| if [ "${done}" = true ]; then | ||
| echo "EC2 SSM command has finished." | ||
|
|
||
| # if success is still true here, then none of the commands failed | ||
| if [ "${success}" == true ]; then | ||
| echo "EC2 SSM command succeeded!" | ||
| exit 0 | ||
| else | ||
| echo "EC2 SSM command failed!" | ||
| exit 1 | ||
| fi | ||
| break | ||
| fi | ||
| done | ||
| exit 1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't remember, but I think we will need to remember to manually terminate this stack. It looks like the resources created in it are all unique I believe? Don't believe CDK will handle this for us in this instance.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah good call. I had to delete the original stack in my local fork, so the same should be applicable here as well. Also might have to apply this change to the FIPS branches as well or they'll still be calling the old stack.