Skip to content

Commit

Permalink
Added "Stack" to lunch AWS-Batch
Browse files Browse the repository at this point in the history
  • Loading branch information
hadiparsianNIH authored Feb 12, 2025
1 parent 1c686c8 commit 1dd26ac
Showing 1 changed file with 5 additions and 74 deletions.
79 changes: 5 additions & 74 deletions AWS/tutorial_4-methylseq2.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -73,85 +73,16 @@
"id": "dff8af02",
"metadata": {},
"source": [
"### AWS Batch Setup"
]
},
{
"cell_type": "markdown",
"id": "049af895",
"metadata": {},
"source": [
"#### **0-Before You Begin**\n",
"\n",
"- Ensure you have a VPC, subnets, and security group set up before continuing the AWS Batch setup.\n",
"- Attach following policies to the SageMaker notebook instance:\n",
"\n",
"![](images/batch-policis.PNG)"
]
},
{
"cell_type": "markdown",
"id": "653e1967",
"metadata": {},
"source": [
"#### **1-Creating a Compute Environment**\n",
"\n",
"Naviate to **'AWS Batch'** in the console. Within AWS Batch on the left side menu navigate to **'Compute environments'**, select **'Create'**. \n",
"For this tutorial we will be working with EC2, select 'EC2'.\n",
"- Select \\\"Managed\\\" for Orchestration type.\n",
"- Enter a name for your compute environment\n",
"- Under service role select \"AWSServiceRoleForBatch\"\n",
"- Under Instance role select \"AWSServiceRoleForBatch\"\n",
"- Click 'Next'\n",
"\n",
"![](images/batch-compute1.PNG)\n",
"\n",
"We have set the fields in the image below to default. Click Next.\n",
"\n",
"![](images/batch-compute2.PNG)\n",
"### AWS Batch Setup\n",
"\n",
"For Network Configuration select your VPC, subnets, and security groups you would like to utilize. This will allow AWS Batch to create instances that can communicate with each other and have access to acceptable networks.\n",
"AWS Batch will create the needed permissions, roles and resources to run Nextflow in a serverless manner. You can set up AWS Batch manually or deploy it **automatically** with a stack template. The Launch Stack button below will take you to the cloud formation create stack webpage with the template with required resources already linked. \n",
"\n",
"![](images/batch-compute3.PNG)\n",
"\n",
"The last step is to review all the configuration made to your compute environment. Once you are satisfied click \"Create compute environment\".\n",
"\n",
"![](images/batch-compute4.PNG)"
]
},
{
"cell_type": "markdown",
"id": "82979a36",
"metadata": {},
"source": [
"#### **3-Creating a Job Queue**\n",
"\n",
"Now that we have created a compute environment lets create a job queue. Job queues help Batch to stay organized by holding jobs until they can be scheduled to run in a compute environment.\n",
"\n",
"In the AWS Batch console, go to the left side menu, click \"Job queues\" and click \"Create\".\n",
"\n",
"- Set orchestration type to EC2\n",
"- Give your queue a name and set its priority. for this tutorial we have se it to '1000' to have the highest priority\n",
"- Associate the compute environment you created in the pervious step.\n",
"- Review and create the job queue.\n",
"\n",
"![](images/batch-queue1.PNG)"
]
},
{
"cell_type": "markdown",
"id": "44e7fc55",
"metadata": {},
"source": [
"#### **4-Applying Permissions**\n",
"If you prefer to skip manual deployment and deploy automatically in the cloud, click the Launch Stack button below. For a walkthrough of the screens during automatic deployment please click [here](https://github.com/NIGMS/NIGMS-Sandbox/blob/main/docs/HowToLaunchAWSBatch.md). The deployment should take ~5 min and then the resources will be ready for use. \n",
"\n",
"For this step we are enabling AWS Batch permissions on EC2 clusters.\n",
"[![Launch Stack](images/LaunchStack.jpg)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=aws-batch-nigms&templateURL=https://nigms-sandbox.s3.us-east-1.amazonaws.com/cf-templates/AWSBatch_template.yaml )\n",
"\n",
"- On the left side menu under 'Control settings' click 'Permissions'.\n",
"- Next to Container insights click 'Edit'.\n",
"- Using the toggles select with compute environment should have these permissions and click 'Save changes'.\n",
"\n",
"![](images/batch-permission1.PNG)"
"Before begining this tutorial, if you do not have required roles, policies, permissions or compute environment and would like to **manually** set those up please click [here](https://github.com/NIGMS/NIGMS-Sandbox/blob/main/docs/AWS-Batch-Setup.md) to set that up."
]
},
{
Expand Down

0 comments on commit 1dd26ac

Please sign in to comment.