add next two episodes

config.yaml

@@ -67,6 +67,8 @@ contact: '[email protected]'
 # Order of episodes in your lesson
 episodes: 
 - Data-storage-setting-up-S3.md
+- SageMaker-notebooks-as-controllers.md
+- Accessing-S3-via-SageMaker-notebooks.md
 
 # Information for Learners
 learners: 

episodes/Acessing-S3-via-SageMaker-notebooks.md

@@ -0,0 +1,163 @@
+---
+title: "Accessing and Managing Data in S3 with SageMaker Notebooks"
+teaching: 20
+exercises: 10
+---
+
+:::::::::::::::::::::::::::::::::::::: questions 
+
+- How can I load data from S3 into a SageMaker notebook?
+- How do I monitor storage usage and costs for my S3 bucket?
+- What steps are involved in pushing new data back to S3 from a notebook?
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+::::::::::::::::::::::::::::::::::::: objectives
+
+- Read data directly from an S3 bucket into memory in a SageMaker notebook.
+- Check storage usage and estimate costs for data in an S3 bucket.
+- Upload new files from the SageMaker environment back to the S3 bucket.
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+## 1A. Read Data from S3 into Memory
+
+Our data is stored in an S3 bucket called `titanic-dataset-test`. This approach reads data directly from S3 into memory within the Jupyter notebook environment without creating a local copy of `train.csv`.
+
+```python
+import boto3
+import pandas as pd
+import sagemaker
+from sagemaker import get_execution_role
+
+# Define the SageMaker role and session
+role = sagemaker.get_execution_role()
+session = sagemaker.Session()
+s3 = boto3.client('s3')
+
+# Define the S3 bucket and object key
+bucket = 'titanic-dataset-test'  # replace with your S3 bucket name
+key = 'data/titanic_train.csv'   # replace with your object key
+response = s3.get_object(Bucket=bucket, Key=key)
+
+# Load the data into a pandas DataFrame
+train_data = pd.read_csv(response['Body'])
+print(train_data.shape)
+train_data.head()
+
+::::::::::::::::::::::::::::::::::::: callout 
+
+### Why Direct Reading?
+
+Directly reading from S3 into memory minimizes storage requirements on the notebook instance and can handle large datasets without local storage limitations.
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+## 1B. Download Data as a Local Copy
+
+For smaller datasets, it can be convenient to have a local copy within the notebook’s environment. However, if your dataset is large (>1GB), consider skipping this step.
+
+### Steps to Download Data from S3
+
+```python
+# Define the S3 bucket and file location
+file_key = "data/titanic_train.csv"  # Path to your file in the S3 bucket
+local_file_path = "./titanic_train.csv"  # Local path to save the file
+
+# Download the file from S3
+s3.download_file(bucket, file_key, local_file_path)
+print("File downloaded:", local_file_path)
+```
+
+:::::::::::::::::::::::::::::::::::::: callout 
+
+## Resolving Permission Issues
+
+If you encounter permission issues when downloading from S3:
+- Ensure your IAM role has appropriate policies for S3 access.
+- Verify the bucket policy allows access.
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+## 2. Check Current Size and Storage Costs of the Bucket
+
+It’s useful to periodically check the storage usage and associated costs of your S3 bucket. Using the **Boto3** library, you can calculate the total size of objects within a specified bucket.
+
+```python
+# Initialize the total size counter
+total_size_bytes = 0
+
+# List and sum the size of all objects in the bucket
+paginator = s3.get_paginator('list_objects_v2')
+for page in paginator.paginate(Bucket=bucket):
+    for obj in page.get('Contents', []):
+        total_size_bytes += obj['Size']
+
+# Convert the total size to megabytes for readability
+total_size_mb = total_size_bytes / (1024 ** 2)
+print(f"Total size of bucket '{bucket}': {total_size_mb:.2f} MB")
+```
+
+::::::::::::::::::::::::::::::::::::: callout 
+
+### Explanation
+
+1. **Paginator**: Handles large listings in S3 buckets.
+2. **Size Calculation**: Sums the `Size` attribute of each object.
+3. **Unit Conversion**: Size is converted from bytes to megabytes for readability.
+
+> **Tip**: For large buckets, consider filtering by folder or object prefix to calculate size for specific directories.
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+## 3. Estimate Storage Costs
+
+AWS S3 costs are based on data size, region, and storage class. The example below estimates costs for the **S3 Standard** storage class in **US East (N. Virginia)**.
+
+```python
+# Pricing per GB for different storage tiers
+first_50_tb_price_per_gb = 0.023
+next_450_tb_price_per_gb = 0.022
+over_500_tb_price_per_gb = 0.021
+
+# Calculate the cost based on the size
+total_size_gb = total_size_bytes / (1024 ** 3)
+if total_size_gb <= 50 * 1024:
+    cost = total_size_gb * first_50_tb_price_per_gb
+elif total_size_gb <= 500 * 1024:
+    cost = (50 * 1024 * first_50_tb_price_per_gb) + \
+           ((total_size_gb - 50 * 1024) * next_450_tb_price_per_gb)
+else:
+    cost = (50 * 1024 * first_50_tb_price_per_gb) + \
+           (450 * 1024 * next_450_tb_price_per_gb) + \
+           ((total_size_gb - 500 * 1024) * over_500_tb_price_per_gb)
+
+print(f"Estimated monthly storage cost: ${cost:.4f}")
+```
+
+> For up-to-date pricing details, refer to the [AWS S3 Pricing page](https://aws.amazon.com/s3/pricing/).
+
+## Important Considerations:
+
+- **Pricing Tiers**: S3 has tiered pricing, so costs vary with data size.
+- **Region and Storage Class**: Prices differ by AWS region and storage class.
+- **Additional Costs**: Consider other costs for requests, retrievals, and data transfer.
+
+## 4. Upload New Files from Notebook to S3
+
+As your analysis generates new files, you may need to upload them to your S3 bucket. Here’s how to upload a file from the notebook environment to S3.
+
+```python
+# Define the S3 bucket name and file paths
+train_file_path = "results.txt"  # File to upload
+s3.upload_file(train_file_path, bucket, "results/results.txt")
+print("Files uploaded successfully.")
+```
+
+:::::::::::::::::::::::::::::::::::::: keypoints 
+
+- Load data from S3 into memory for efficient storage and processing.
+- Periodically check storage usage and costs to manage S3 budgets.
+- Use SageMaker to upload analysis results and maintain an organized workflow.
+
+::::::::::::::::::::::::::::::::::::::::::::::::

episodes/SageMaker-notebooks-as-controllers.md

@@ -0,0 +1,95 @@
+---
+title: "Notebooks as Controllers"
+teaching: 20
+exercises: 10
+---
+
+:::::::::::::::::::::::::::::::::::::: questions 
+
+- How do you set up and use SageMaker Notebooks for machine learning tasks?
+- How can you manage compute resources efficiently using SageMaker's controller notebook approach?
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+::::::::::::::::::::::::::::::::::::: objectives
+
+- Describe how to use SageMaker Notebooks for ML workflows.
+- Set up a Jupyter notebook instance as a controller to manage compute tasks.
+- Use SageMaker SDK to launch training and tuning jobs on scalable instances.
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+## Step 2: Running Python Code with SageMaker Notebooks
+
+Amazon SageMaker provides a managed environment to simplify the process of building, training, and deploying machine learning models. By using SageMaker, you can focus on model development without needing to manually provision resources or set up environments. Here, we'll guide you through setting up a Jupyter notebook instance, loading data, training a model, and performing optional hyperparameter tuning using the Titanic dataset in S3.
+
+> **Note**: We’ll use SageMaker notebook instances directly (instead of SageMaker Studio) for easier instance monitoring across users and streamlined resource management.
+
+## Using the Notebook as a Controller
+
+In this setup, the notebook instance functions as a **controller** to manage more resource-intensive compute tasks. By selecting a minimal instance (e.g., `ml.t3.medium`) for the notebook, you can perform lightweight operations and leverage the **SageMaker Python SDK** to launch more powerful, scalable compute instances when needed for model training, batch processing, or hyperparameter tuning. This approach minimizes costs by keeping your controller instance lightweight while accessing the full power of SageMaker for demanding tasks.
+
+## Summary of Key Steps
+1. Navigate to SageMaker in AWS.
+2. Create a Jupyter notebook instance as a controller.
+3. Set up the Python environment within the notebook.
+4. Load the Titanic dataset from S3.
+5. Use SageMaker SDK to launch training and tuning jobs on powerful instances.
+6. View and monitor training/tuning progress.
+
+## Detailed Procedure
+
+### 1. Navigate to SageMaker
+- In the AWS Console, search for **SageMaker** and select **SageMaker - Build, Train, and Deploy Models**.
+- Click **Set up for single user** (if prompted) and wait for the SageMaker domain to spin up.
+- Under **S3 Resource Configurations**, select the S3 bucket you created earlier containing your dataset.
+
+### 2. Create a New Notebook Instance
+- In the SageMaker menu, go to **Notebooks > Notebook instances**, then click **Create notebook instance**.
+- **Notebook Name**: Enter a name (e.g., `Titanic-ML-Notebook`).
+- **Instance Type**: Start with a small instance type, such as `ml.t3.medium`. You can scale up later as needed for intensive tasks, which will be managed by launching separate training jobs from this notebook.
+- **Permissions and Encryption**:
+   - **IAM Role**: Choose an existing role or create a new one. The role should include the `AmazonSageMakerFullAccess` policy to enable access to AWS services like S3.
+   - **Root Access**: Choose to enable or disable root access. If you’re comfortable with managing privileges, enabling root access allows for additional flexibility in package installation.
+   - **Encryption Key** (Optional): Specify a KMS key for encrypting data at rest if needed. Otherwise, leave it blank.
+- **Network (Optional)**: Networking settings are optional. Configure them if you’re working within a specific VPC or need network customization.
+- **Git Repositories Configuration (Optional)**: Connect a GitHub repository to automatically clone it into your notebook. Note that larger repositories consume more disk space, so manage storage to minimize costs.
+   - **Tips to Manage Storage**:
+     - Use **S3** for large files or datasets instead of storing them in the repository.
+     - Keep Git repositories small (code and small files only).
+     - Monitor storage with the following command in a terminal to check disk usage:
+       ```bash
+       du -sh *
+       ```
+- **Tags (Optional)**: Adding tags helps track and organize resources for billing and management.
+   - Example: `Key: Job, Value: Titanic-Analysis-Notebook`
+
+Click **Create notebook instance**. It may take a few minutes for the instance to start. Once its status is **InService**, you can open the notebook instance and start coding.
+
+::::::::::::::::::::::::::::::::::::: callout
+
+### Tags for Billing Management
+Adding tags to your notebook instance helps track costs over time. This is particularly useful when you need to break down expenses by project, task, or team. We recommend using tags like `Job`, `Project`, or `Team` to help with future cost analysis.
+
+::::::::::::::::::::::::::::::::::::::::::::::::
+
+### Managing Training and Tuning with the Controller Notebook
+
+After setting up the controller notebook, use the **SageMaker Python SDK** within the notebook to launch compute-heavy tasks on more powerful instances as needed. Examples of tasks to launch include:
+
+- **Training a Model**: Use the SDK to submit a training job, specifying a higher-powered instance (e.g., `ml.p2.xlarge` or `ml.m5.4xlarge`) based on your model’s resource requirements.
+- **Hyperparameter Tuning**: Configure and launch tuning jobs, allowing SageMaker to automatically manage multiple powerful instances for optimal tuning.
+- **Batch Processing**: Offload batch data processing tasks to a larger instance if needed.
+
+This setup allows you to control costs by keeping the notebook instance minimal and only incurring costs for larger instances when they are actively training or tuning models.
+
+For more details, refer to the [SageMaker Python SDK documentation](https://sagemaker.readthedocs.io/) for example code on launching and managing remote training jobs.
+
+::::::::::::::::::::::::::::::::::::: keypoints 
+
+- Use a minimal SageMaker notebook instance as a controller to manage larger, resource-intensive tasks.
+- Launch training, tuning, or batch processing jobs on scalable instances using the SageMaker SDK.
+- Tags can help track costs effectively, especially in multi-project or team settings.
+- Use the SageMaker SDK documentation to explore additional options for managing compute resources in AWS.
+
+::::::::::::::::::::::::::::::::::::::::::::::::