Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update prefect-aws docs #16639

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 122 additions & 25 deletions docs/integrations/prefect-aws/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: prefect-aws
---

The prefect-aws library makes it easy to leverage the capabilities of AWS in your workflows.
The `prefect-aws` integration makes it easy to leverage the capabilities of AWS in your workflows.
For example, you can retrieve secrets using AWS Secrets Manager, read and write objects with AWS S3, and deploy your flows on AWS ECS.

## Getting started
Expand All @@ -11,10 +11,9 @@ For example, you can retrieve secrets using AWS Secrets Manager, read and write

- An [AWS account](https://aws.amazon.com/account/) and the necessary permissions to access desired services.

### Install `prefect-aws`
### Installation

The following command will install a version of `prefect-aws` compatible with your installed version of `prefect`.
If you don't already have `prefect` installed, it will install the newest version of `prefect` as well.
Install `prefect-aws` as a dependency of Prefect. If you don't already have `prefect` installed, it will install the newest version of `prefect` as well.

```bash
pip install "prefect[aws]"
Expand All @@ -28,29 +27,18 @@ pip install -U "prefect[aws]"

### Register newly installed block types

Register the block types in the `prefect-aws` module to make them available for use.
Register the block types in `prefect-aws` to make them available for use.

```bash
prefect block register -m prefect_aws
```

## Examples

### Run flows on AWS ECS

Run flows on [AWS Elastic Container Service (ECS)](https://aws.amazon.com/ecs/) to dynamically scale your infrastructure.

See the [ECS work pool docs](/integrations/prefect-aws/ecs_guide) for a walkthrough of using ECS in a hybrid work pool.

If you're using Prefect Cloud, [ECS push work pools](/v3/deploy/infrastructure-examples/serverless) provide all the benefits of ECS with a quick setup and no worker needed.

In the examples below, you create blocks with Python code.
Alternatively, each block can be created through the Prefect UI.
## Blocks setup

### Save credentials to an AWS Credentials block
### Credentials

Use of most AWS services requires an authenticated session.
Prefect makes it simple to provide credentials via a AWS Credentials block.
Most AWS services requires an authenticated session.
Prefect makes it simple to provide credentials via AWS Credentials blocks.

Steps:

Expand All @@ -70,22 +58,61 @@ AwsCredentials(
).save("BLOCK-NAME-PLACEHOLDER")
```

Prefect is using the Boto3 library under the hood.
Prefect uses the Boto3 library under the hood.
To find credentials for authentication, any data not provided to the block are sourced at runtime in the order shown in the [Boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
Prefect creates the session object using the values in the block and then, any missing values follow the sequence in the Boto3 docs.

See an example of using the `AwsCredentials` block with [AWS Secrets Manager](#aws-secrets-manager) with third-party services without storing credentials in the block itself in the [docs](/v3/develop/secrets).
### S3

Create a block for reading and writing files to S3.

```python
from prefect_aws import AwsCredentials
from prefect_aws.s3 import S3Bucket

S3Bucket(
bucket_name="BUCKET-NAME-PLACEHOLDER",
credentials=aws_credentials
).save("S3-BLOCK-NAME-PLACEHOLDER")
```

### Lambda
Invoke AWS Lambdas, synchronously or asynchronously.

Here's how to load the saved credentials:
```python
from prefect_aws.lambda_function import LambdaFunction
from prefect_aws.credentials import AwsCredentials

LambdaFunction(
function_name="test_lambda_function",
aws_credentials=credentials,
).save("LAMBDA-BLOCK-NAME-PLACEHOLDER")
```

### Secret Manager
Create a block to read, write, and delete AWS Secret Manager secrets.

```python
from prefect_aws import AwsCredentials
from prefect_aws.secrets_manager import AwsSecret

AwsSecret(
secret_name="test_secret_name",
aws_credentials=credentials,
).save("AWS-SECRET-BLOCK-NAME-PLACEHOLDER")

AwsCredentials.load("BLOCK-NAME-PLACEHOLDER")
```

The AWS Credentials block is often nested within other blocks, such as `S3Bucket` or `AwsSecret`, and provides authentication for those services.

## Run flows on AWS ECS

Run flows on [AWS Elastic Container Service (ECS)](https://aws.amazon.com/ecs/) to dynamically scale your infrastructure.

Prefect Cloud offers [ECS push work pools](/v3/deploy/infrastructure-examples/serverless). Push work pools submit runs directly to ECS, instead of requiring a worker to actively poll for flow runs to execute.

See the [ECS work pool guide](/integrations/prefect-aws/ecs_guide) for a walkthrough of using ECS in a hybrid work pool.

## Examples

### Read and write files to AWS S3

Expand Down Expand Up @@ -144,8 +171,78 @@ if __name__ == "__main__":
secrets_manager_flow()
```

### Invoke lambdas

```python
from prefect_aws.lambda_function import LambdaFunction
from prefect_aws.credentials import AwsCredentials

credentials = AwsCredentials()
lambda_function = LambdaFunction(
function_name="test_lambda_function",
aws_credentials=credentials,
)
response = lambda_function.invoke(
payload={"foo": "bar"},
invocation_type="RequestResponse",
)
response["Payload"].read()
```

### Submit AWS Glue jobs

```python
from prefect import flow
from prefect_aws import AwsCredentials
from prefect_aws.glue_job import GlueJobBlock


@flow
def example_run_glue_job():
aws_credentials = AwsCredentials(
aws_access_key_id="your_access_key_id",
aws_secret_access_key="your_secret_access_key"
)
glue_job_run = GlueJobBlock(
job_name="your_glue_job_name",
arguments={"--YOUR_EXTRA_ARGUMENT": "YOUR_EXTRA_ARGUMENT_VALUE"},
).trigger()

return glue_job_run.wait_for_completion()

seanpwlms marked this conversation as resolved.
Show resolved Hide resolved

example_run_glue_job()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wadda ya think about and if name = main block here and later?

```

### Submit AWS Batch jobs

```python
from prefect import flow
from prefect_aws import AwsCredentials
from prefect_aws.batch import batch_submit


@flow
def example_batch_submit_flow():
aws_credentials = AwsCredentials(
aws_access_key_id="acccess_key_id",
aws_secret_access_key="secret_access_key"
)
job_id = batch_submit(
"job_name",
"job_queue",
"job_definition",
aws_credentials
)
return job_id

seanpwlms marked this conversation as resolved.
Show resolved Hide resolved
example_batch_submit_flow()
```

## Resources

For assistance using AWS, consult the [AWS documentation](https://docs.aws.amazon.com/) and, in particular, the [Boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html).

Refer to the `prefect-aws` SDK documentation linked in the sidebar to explore all the capabilities of the `prefect-aws` library.

Refer to the [secrets documentation](/v3/develop/secrets) to see an example of using the `AwsCredentials` block with a third-party service like Snowflake. The example does not require you to save your AWS credentials as part of the Snowflake block.
2 changes: 1 addition & 1 deletion docs/v3/develop/blocks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -433,7 +433,7 @@ If a block type is not available in the UI, you can [register it](#register-bloc
| S3 Bucket | `s3-bucket` | [prefect-aws](/integrations/prefect-aws/) |
| Azure Blob Storage Credentials | `azure-blob-storage-credentials` | [prefect-azure](/integrations/prefect-azure/) |
| Azure Container Instance Credentials | `azure-container-instance-credentials` | [prefect-azure](/integrations/prefect-azure/) |
| Azure Container Instance Job] | `azure-container-instance-job` | [prefect-azure](/integrations/prefect-azure/) |
| Azure Container Instance Job | `azure-container-instance-job` | [prefect-azure](/integrations/prefect-azure/) |
| Azure Cosmos DB Credentials | `azure-cosmos-db-credentials` | [prefect-azure](/integrations/prefect-azure/) |
| AzureML Credentials | `azureml-credentials` | [prefect-azure](/integrations/prefect-azure/) |
| BitBucket Credentials | `bitbucket-credentials` | [prefect-bitbucket](/integrations/prefect-bitbucket/) |
Expand Down
Loading