Skip to content

Latest commit

 

History

History
261 lines (154 loc) · 7.2 KB

README.md

File metadata and controls

261 lines (154 loc) · 7.2 KB
Raw
title has_children nav_order nav_exclude
Infrastructure Components
true
3
false

DataOps Infrastructure Components

These components define the technical building blocks which enable advanced, ready-to-deploy data solutions in the Infrastructure Catalog.

Contents

  1. AWS Components

  2. Azure Components

    • (Coming soon)

  3. GCP Components

    • (Coming soon)

AWS Components

AWS EC2

Overview

EC2 is the virtual machine layer of the AWS platform. This module allows you to pass your own startup scripts, and it streamlines the creation and usage of credentials (passwords and/or SSH keypairs) needed to connect to the instances.

Documentation

AWS ECR

Overview

ECR (Elastic Compute Repository) is the private-hosted AWS equivalent of DockerHub. ECR allows you to securely publish docker images which should not be accessible to external users.

Documentation

AWS ECR-Image

Overview

ECR (Elastic Compute Repository) is the private-hosted AWS equivalent of DockerHub. ECR allows you to securely publish docker images which should not be accessible to external users.

Documentation

AWS ECS-Cluster

Overview

ECS, or EC2 Container Service, is able to run docker containers natively in AWS cloud. While the module can support classic EC2-based and Fargate, features, this module generally prefers "ECS Fargete", which allows dynamic launching of docker containers with no always-on cost and no servers to manage or pay for when tasks are not running.

Use in combination with the ECS-Task component.

Documentation

AWS ECS-Task

Overview

ECS, or EC2 Container Service, is able to run docker containers natively in AWS cloud. While the module can support classic EC2-based and Fargate, features, this module generally prefers "ECS Fargete", which allows dynamic launching of docker containers with no always-on cost and no servers to manage or pay for when tasks are not running.

Use in combination with the ECS-Cluster component.

Documentation

AWS Glue-Crawler

Overview

Glue is AWS's fully managed extract, transform, and load (ETL) service. A Glue crawler is used to access a data store and create table definitions. This can be used in conjuction with Amazon Athena to query flat files in S3 buckets using SQL.

Documentation

AWS Glue-Job

Overview

Glue is AWS's fully managed extract, transform, and load (ETL) service. A Glue job can be used job to run ETL Python scripts.

Documentation

AWS Lambda-Python

Overview

AWS Lambda is a platform which enables serverless execution of arbitrary functions. This module specifically focuses on the Python implementatin of Lambda functions. Given a path to a folder of one or more python fyles, this module takes care of packaging the python code into a zip and uploading to a new Lambda Function in AWS. The module can also be configured with S3-based triggers, to run the function automatically whenever a file is landed in a specific S3 path.

Documentation

AWS RDS

Overview

Deploys an RDS-backed database. RDS currently supports the following database engines:

  • Aurora
  • MySQL
  • PostgreSQL
  • Oracle
  • SQL Server

Each engine type has it's own required configuration. For already-configured database configurations, see the catalog modules: catalog/aws/mysql and catalog/aws/postgres which are built on top of this component module.

  • NOTE: Requires AWS policy 'AmazonRDSFullAccess' on the terraform account

Documentation

AWS Redshift

Overview

This is the underlying technical component which supports the Redshift catalog module.

NOTE: Requires AWS policy 'AmazonRedshiftFullAccess' on the terraform account

Documentation

AWS Secrets-Manager

Overview

This module takes as input a set of maps from variable names to secrets locations (in YAML or JSON). The module uploads those secrets to AWS Secrets Manager and returns the same map pointing to the IDs of new AWS Secrets manager locations. Those IDs (aka ARNs) can then safely be handed on to other resources which required access to those secrets.

Usage Notes:

  • Any secrets locations which are already pointing to AWS secrets will simply be passed back through to the output with no changes.
  • For security reasons, this module does not accept inputs for secrets using the clear text of the secrets themselves. To properly use this module, first save the secrets to a YAML or JSON file which is excluded from source control.

Documentation

AWS Step-Functions

Overview

AWS Step Functions is a service provided by Amazon Web Services that makes it easier to orchestrate multiple AWS services to accomplish tasks. Step Functions allows you to create steps in a process where the output of one step becomes the input for another step.

Documentation

AWS VPC

Overview

The VPC module creates a number of network services which support other key AWS functions.

Included automatically when creating this module:

  • 1 VPC which contains the following:
    • 2 private subnets (for resources which do not need a public IP address)
    • 2 public subnets (for resources which do need a public IP address)
    • 1 NAT gateway (allows private subnet resources to reach the outside world)
    • 1 Intenet gateway (allows resources in public and private subnets to reach the internet)
    • route tables and routes to connect all of the above

Documentation

Azure Components

(Coming soon)

GCP Components

(Coming soon)

NOTE: This documentation was auto-generated using terraform-docs. Please do not attempt to manually update this file.