From c05af5d9cc51c497d6fd6d8c69677ab326090d5a Mon Sep 17 00:00:00 2001 From: Aaron Steers <18150651+aaronsteers@users.noreply.github.com> Date: Mon, 23 Mar 2020 19:20:01 -0700 Subject: [PATCH] Auto-docs v3 (#80) * move index to catalog/components root * updated all readme docs * update contributions guide --- README.md | 2 +- docs/catalog_index.md => catalog/README.md | 276 ++++++------- catalog/aws/airflow/README.md | 10 + catalog/aws/data-lake/README.md | 10 + catalog/aws/dbt/README.md | 10 + catalog/aws/environment/README.md | 10 + catalog/aws/mysql/README.md | 10 + catalog/aws/postgres/README.md | 10 + catalog/aws/redshift/README.md | 10 + catalog/aws/singer-taps/README.md | 26 +- catalog/aws/tableau-server/README.md | 10 + .../README.md | 364 +++++++++--------- components/aws/ec2/README.md | 10 + components/aws/ecr/README.md | 10 + components/aws/ecs-cluster/README.md | 11 + components/aws/ecs-task/README.md | 13 + components/aws/lambda-python/README.md | 12 + components/aws/rds/README.md | 10 + components/aws/redshift/README.md | 10 + components/aws/secrets-manager/README.md | 10 + components/aws/step-functions/README.md | 11 + components/aws/vpc/README.md | 10 + docs/CONTRIBUTING.md | 27 +- docs/README.md | 13 +- docs/build.py | 35 +- samples/airflow-on-aws/README.md | 10 + samples/dbt-and-singer-on-aws/README.md | 11 + samples/mysql-on-aws/README.md | 9 + samples/postgres-on-aws/README.md | 9 + samples/redshift-dw/README.md | 10 + samples/s3-lambda-trigger/README.md | 9 + samples/secrets-manager-on-aws/README.md | 9 + samples/tableau-on-aws/README.md | 10 + 33 files changed, 646 insertions(+), 361 deletions(-) rename docs/catalog_index.md => catalog/README.md (94%) rename docs/components_index.md => components/README.md (96%) diff --git a/README.md b/README.md index 217a68dd..4bace3b4 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ ## Catalog Documentation -Catalog documentation is available from the [Infrastructure Catalog Index](docs/catalog_index.md). +Catalog documentation is available from the [Infrastructure Catalog Index](catalog/README.md). ## Contributions Guide diff --git a/docs/catalog_index.md b/catalog/README.md similarity index 94% rename from docs/catalog_index.md rename to catalog/README.md index e84e431b..7f2ef1ea 100644 --- a/docs/catalog_index.md +++ b/catalog/README.md @@ -1,138 +1,138 @@ - -# DataOps Infrastructure Catalog - -The Infrastructure Catalog contains ready-to-deploy terraform modules for a variety of production data project use cases and POCs. For information about the technical building blocks used in these modules, please see the catalog [components index](components_index.md). - -## Contents - -1. [AWS Catalog](#aws-catalog) - - [AWS Airflow](#aws-airflow) - - [AWS Data-Lake](#aws-data-lake) - - [AWS DBT](#aws-dbt) - - [AWS Environment](#aws-environment) - - [AWS MySQL](#aws-mysql) - - [AWS Postgres](#aws-postgres) - - [AWS Redshift](#aws-redshift) - - [AWS Singer-Taps](#aws-singer-taps) - - [AWS Tableau-Server](#aws-tableau-server) - -2. Azure Catalog - * _(Coming soon)_ -2. GCP Catalog - * _(Coming soon)_ - -## AWS Catalog - -### [AWS Airflow](../catalog/aws/airflow/README.md) - -Airflow is an open source platform to programmatically author, schedule and monitor workflows. More information here: [airflow.apache.org](https://airflow.apache.org/) - - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/airflow?ref=master` -* See the [AWS Airflow Readme](../catalog/aws/airflow/README.md) for input/output specs and additional info. - -------------------- - -### [AWS Data-Lake](../catalog/aws/data-lake/README.md) - -This data lake implementation creates three buckets, one each for data, logging, and metadata. The data lake also supports lambda functions which can -trigger automatically when new content is added. - - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/data-lake?ref=master` -* See the [AWS Data-Lake Readme](../catalog/aws/data-lake/README.md) for input/output specs and additional info. - -------------------- - -### [AWS DBT](../catalog/aws/dbt/README.md) - -DBT (Data Built Tool) is a CI/CD and DevOps-friendly platform for automating data transformations. More info at [www.getdbt.com](https://www.getdbt.com). - - - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/dbt?ref=master` -* See the [AWS DBT Readme](../catalog/aws/dbt/README.md) for input/output specs and additional info. - -------------------- - -### [AWS Environment](../catalog/aws/environment/README.md) - -The environment module sets up common infrastrcuture like VPCs and network subnets. The `envrionment` output -from this module is designed to be passed easily to downstream modules, streamlining the reuse of these core components. - - - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/environment?ref=master` -* See the [AWS Environment Readme](../catalog/aws/environment/README.md) for input/output specs and additional info. - -------------------- - -### [AWS MySQL](../catalog/aws/mysql/README.md) - -Deploys a MySQL server running on RDS. - -* NOTE: Requires AWS policy 'AmazonRDSFullAccess' on the terraform account - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/mysql?ref=master` -* See the [AWS MySQL Readme](../catalog/aws/mysql/README.md) for input/output specs and additional info. - -------------------- - -### [AWS Postgres](../catalog/aws/postgres/README.md) - -Deploys a Postgres server running on RDS. - -* NOTE: Requires AWS policy 'AmazonRDSFullAccess' on the terraform account - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/postgres?ref=master` -* See the [AWS Postgres Readme](../catalog/aws/postgres/README.md) for input/output specs and additional info. - -------------------- - -### [AWS Redshift](../catalog/aws/redshift/README.md) - -Redshift is an AWS database platform which applies MPP (Massively-Parallel-Processing) principles to big data workloads in the cloud. - - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/redshift?ref=master` -* See the [AWS Redshift Readme](../catalog/aws/redshift/README.md) for input/output specs and additional info. - -------------------- - -### [AWS Singer-Taps](../catalog/aws/singer-taps/README.md) - -The Singer Taps platform is the open source stack which powers the [Stitcher](https://www.stitcher.com) ELT platform. For more information, see [singer.io](https://singer.io) - - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/singer-taps?ref=master` -* See the [AWS Singer-Taps Readme](../catalog/aws/singer-taps/README.md) for input/output specs and additional info. - -------------------- - -### [AWS Tableau-Server](../catalog/aws/tableau-server/README.md) - -This module securely deploys one or more Tableau Servers, which can then be used to host reports in production or POC environments. -The module supports both Linux and Windows versions of the Tableau Server Software. - - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/tableau-server?ref=master` -* See the [AWS Tableau-Server Readme](../catalog/aws/tableau-server/README.md) for input/output specs and additional info. - -------------------- - - - -## Azure Catalog - -_(Coming soon)_ - -## GCP Catalog - -_(Coming soon)_ - -------------------- - -_**NOTE:** This documentation was [auto-generated](build.py) using -`terraform-docs` and `s-infra` from `slalom.dataops`. -Please do not attempt to manually update this file._ - + +# DataOps Infrastructure Catalog + +The Infrastructure Catalog contains ready-to-deploy terraform modules for a variety of production data project use cases and POCs. For information about the technical building blocks used in these modules, please see the catalog [components index](../components/README.md). + +## Contents + +1. [AWS Catalog](#aws-catalog) + - [AWS Airflow](#aws-airflow) + - [AWS Data-Lake](#aws-data-lake) + - [AWS DBT](#aws-dbt) + - [AWS Environment](#aws-environment) + - [AWS MySQL](#aws-mysql) + - [AWS Postgres](#aws-postgres) + - [AWS Redshift](#aws-redshift) + - [AWS Singer-Taps](#aws-singer-taps) + - [AWS Tableau-Server](#aws-tableau-server) + +2. Azure Catalog + * _(Coming soon)_ +2. GCP Catalog + * _(Coming soon)_ + +## AWS Catalog + +### [AWS Airflow](../catalog/aws/airflow/README.md) + +Airflow is an open source platform to programmatically author, schedule and monitor workflows. More information here: [airflow.apache.org](https://airflow.apache.org/) + + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/airflow?ref=master` +* See the [AWS Airflow Readme](../catalog/aws/airflow/README.md) for input/output specs and additional info. + +------------------- + +### [AWS Data-Lake](../catalog/aws/data-lake/README.md) + +This data lake implementation creates three buckets, one each for data, logging, and metadata. The data lake also supports lambda functions which can +trigger automatically when new content is added. + + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/data-lake?ref=master` +* See the [AWS Data-Lake Readme](../catalog/aws/data-lake/README.md) for input/output specs and additional info. + +------------------- + +### [AWS DBT](../catalog/aws/dbt/README.md) + +DBT (Data Built Tool) is a CI/CD and DevOps-friendly platform for automating data transformations. More info at [www.getdbt.com](https://www.getdbt.com). + + + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/dbt?ref=master` +* See the [AWS DBT Readme](../catalog/aws/dbt/README.md) for input/output specs and additional info. + +------------------- + +### [AWS Environment](../catalog/aws/environment/README.md) + +The environment module sets up common infrastrcuture like VPCs and network subnets. The `envrionment` output +from this module is designed to be passed easily to downstream modules, streamlining the reuse of these core components. + + + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/environment?ref=master` +* See the [AWS Environment Readme](../catalog/aws/environment/README.md) for input/output specs and additional info. + +------------------- + +### [AWS MySQL](../catalog/aws/mysql/README.md) + +Deploys a MySQL server running on RDS. + +* NOTE: Requires AWS policy 'AmazonRDSFullAccess' on the terraform account + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/mysql?ref=master` +* See the [AWS MySQL Readme](../catalog/aws/mysql/README.md) for input/output specs and additional info. + +------------------- + +### [AWS Postgres](../catalog/aws/postgres/README.md) + +Deploys a Postgres server running on RDS. + +* NOTE: Requires AWS policy 'AmazonRDSFullAccess' on the terraform account + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/postgres?ref=master` +* See the [AWS Postgres Readme](../catalog/aws/postgres/README.md) for input/output specs and additional info. + +------------------- + +### [AWS Redshift](../catalog/aws/redshift/README.md) + +Redshift is an AWS database platform which applies MPP (Massively-Parallel-Processing) principles to big data workloads in the cloud. + + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/redshift?ref=master` +* See the [AWS Redshift Readme](../catalog/aws/redshift/README.md) for input/output specs and additional info. + +------------------- + +### [AWS Singer-Taps](../catalog/aws/singer-taps/README.md) + +The Singer Taps platform is the open source stack which powers the [Stitcher](https://www.stitcher.com) ELT platform. For more information, see [singer.io](https://singer.io) + + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/singer-taps?ref=master` +* See the [AWS Singer-Taps Readme](../catalog/aws/singer-taps/README.md) for input/output specs and additional info. + +------------------- + +### [AWS Tableau-Server](../catalog/aws/tableau-server/README.md) + +This module securely deploys one or more Tableau Servers, which can then be used to host reports in production or POC environments. +The module supports both Linux and Windows versions of the Tableau Server Software. + + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//catalog/aws/tableau-server?ref=master` +* See the [AWS Tableau-Server Readme](../catalog/aws/tableau-server/README.md) for input/output specs and additional info. + +------------------- + + + +## Azure Catalog + +_(Coming soon)_ + +## GCP Catalog + +_(Coming soon)_ + +------------------- + +_**NOTE:** This documentation was [auto-generated](../docs/build.py) using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ + diff --git a/catalog/aws/airflow/README.md b/catalog/aws/airflow/README.md index 5f87d714..8a79c0f8 100644 --- a/catalog/aws/airflow/README.md +++ b/catalog/aws/airflow/README.md @@ -34,6 +34,16 @@ Airflow is an open source platform to programmatically author, schedule and moni --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/catalog/aws/data-lake/README.md b/catalog/aws/data-lake/README.md index e9805ec3..9a8865ce 100644 --- a/catalog/aws/data-lake/README.md +++ b/catalog/aws/data-lake/README.md @@ -31,6 +31,16 @@ trigger automatically when new content is added. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/catalog/aws/dbt/README.md b/catalog/aws/dbt/README.md index aa28aa57..8700202f 100644 --- a/catalog/aws/dbt/README.md +++ b/catalog/aws/dbt/README.md @@ -36,6 +36,16 @@ DBT (Data Built Tool) is a CI/CD and DevOps-friendly platform for automating dat --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/catalog/aws/environment/README.md b/catalog/aws/environment/README.md index 55a3adf5..3432ca70 100644 --- a/catalog/aws/environment/README.md +++ b/catalog/aws/environment/README.md @@ -35,6 +35,16 @@ from this module is designed to be passed easily to downstream modules, streamli --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/catalog/aws/mysql/README.md b/catalog/aws/mysql/README.md index 60b626d2..aeb19d0c 100644 --- a/catalog/aws/mysql/README.md +++ b/catalog/aws/mysql/README.md @@ -36,6 +36,16 @@ Deploys a MySQL server running on RDS. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/catalog/aws/postgres/README.md b/catalog/aws/postgres/README.md index 5007d521..352fdd6d 100644 --- a/catalog/aws/postgres/README.md +++ b/catalog/aws/postgres/README.md @@ -39,6 +39,16 @@ Deploys a Postgres server running on RDS. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/catalog/aws/redshift/README.md b/catalog/aws/redshift/README.md index 693fa75a..e17f0934 100644 --- a/catalog/aws/redshift/README.md +++ b/catalog/aws/redshift/README.md @@ -34,6 +34,16 @@ Redshift is an AWS database platform which applies MPP (Massively-Parallel-Proce --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/catalog/aws/singer-taps/README.md b/catalog/aws/singer-taps/README.md index 9e3eb52c..a6698b82 100644 --- a/catalog/aws/singer-taps/README.md +++ b/catalog/aws/singer-taps/README.md @@ -15,18 +15,21 @@ The Singer Taps platform is the open source stack which powers the [Stitcher](ht | container\_command | n/a | `any` | n/a | yes | | container\_entrypoint | n/a | `any` | n/a | yes | | container\_image | n/a | `any` | n/a | yes | +| data\_lake\_metadata\_path | The remote folder for storing tap definitions files.
Currently only S3 paths (s3://...) are supported. | `string` | n/a | yes | +| data\_lake\_storage\_path | The path to where files should be stored in the data lake.
Note: - currently only S3 paths (S3://...) are supported.data - You must specify `target` or `data_lake_storage_path` but not both. | `string` | n/a | yes | +| data\_lake\_type | Specify `S3` if loading to an S3 data lake, otherwise leave blank. | `any` | n/a | yes | | environment | Standard `environment` module input. | 
object({
    vpc_id          = string
    aws_region      = string
    public_subnets  = list(string)
    private_subnets = list(string)
  })
| n/a | yes | +| local\_metadata\_path | The local folder which countains tap definitions files: `data.select` and `plan-*.yml` | `string` | n/a | yes | | name\_prefix | Standard `name_prefix` module input. | `string` | n/a | yes | | resource\_tags | Standard `resource_tags` module input. | `map(string)` | n/a | yes | -| source\_code\_folder | n/a | `string` | n/a | yes | -| source\_code\_s3\_bucket | n/a | `string` | n/a | yes | | taps | n/a | 
list(object({
    id       = string
    settings = map(string)
    secrets  = map(string)
  }))
| n/a | yes | +| target | The definition of which target to load data into.
Note: You must specify `target` or `data_lake_storage_path` but not both. | 
object({
    id       = string
    settings = map(string)
    secrets  = map(string)
  })
| n/a | yes | | container\_num\_cores | n/a | `number` | `0.5` | no | | container\_ram\_gb | n/a | `number` | `1` | no | -| scheduled\_sync\_times | A list of schedule strings in 4 digit format: HHMM | `list(string)` | `[]` | no | -| scheduled\_timezone | n/a | `string` | `"PT"` | no | -| source\_code\_s3\_path | n/a | `string` | `"code/taps"` | no | -| target | n/a | 
object({
    id       = string
    settings = map(string)
    secrets  = map(string)
  })
| 
{
  "id": "s3-csv",
  "secrets": {},
  "settings": {
    "s3_key_prefix": "data/raw/{tap}/{table}/{version}/"
  }
}
| no | +| data\_file\_naming\_scheme | n/a | `string` | `"{tap}/{table}/v{version}/{file}"` | no | +| scheduled\_sync\_times | A list of one or more daily sync times in `HHMM` format. E.g.: `0400` for 4am, `1600` for 4pm | `list(string)` | `[]` | no | +| scheduled\_timezone | The timezone used in scheduling.
Currently the following codes are supported: PST, EST, UTC | `string` | `"PT"` | no | +| state\_file\_naming\_scheme | n/a | `string` | `"{tap}/{table}/state/{tap}-{table}-v{version}-state.json"` | no | ## Outputs @@ -36,6 +39,17 @@ The Singer Taps platform is the open source stack which powers the [Stitcher](ht --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [s3-upload.tf](s3-upload.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/catalog/aws/tableau-server/README.md b/catalog/aws/tableau-server/README.md index 49ff15bc..08914a00 100644 --- a/catalog/aws/tableau-server/README.md +++ b/catalog/aws/tableau-server/README.md @@ -44,6 +44,16 @@ The module supports both Linux and Windows versions of the Tableau Server Softwa --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/docs/components_index.md b/components/README.md similarity index 96% rename from docs/components_index.md rename to components/README.md index 58047a54..a6d56299 100644 --- a/docs/components_index.md +++ b/components/README.md @@ -1,182 +1,182 @@ - -# DataOps Infrastructure Components - -These components define the technical building blocks which enable advanced, ready-to-deploy data solutions in the [Infrastructure Catalog](catalog_index.md). - -## Contents - -1. [AWS Components](#aws-components) - - [AWS EC2](#aws-ec2) - - [AWS ECR](#aws-ecr) - - [AWS ECS-Cluster](#aws-ecs-cluster) - - [AWS ECS-Task](#aws-ecs-task) - - [AWS Lambda-Python](#aws-lambda-python) - - [AWS RDS](#aws-rds) - - [AWS Redshift](#aws-redshift) - - [AWS Secrets-Manager](#aws-secrets-manager) - - [AWS Step-Functions](#aws-step-functions) - - [AWS VPC](#aws-vpc) - -2. Azure Components - * _(Coming soon)_ -2. GCP Components - * _(Coming soon)_ - -## AWS Components - -### [AWS EC2](../components/aws/ec2/README.md) - -EC2 is the virtual machine layer of the AWS platform. This module allows you to pass your own startup scripts, and it streamlines the creation and usage of -credentials (passwords and/or SSH keypairs) needed to connect to the instances. - - - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/ec2?ref=master` -* See the [AWS EC2 Readme](../components/aws/ec2/README.md) for input/output specs and additional info. - -------------------- - -### [AWS ECR](../components/aws/ecr/README.md) - -ECR (Elastic Compute Repository) is the private-hosted AWS equivalent of DockerHub. ECR allows you to securely publish docker images which -should not be accessible to external users. - - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/ecr?ref=master` -* See the [AWS ECR Readme](../components/aws/ecr/README.md) for input/output specs and additional info. - -------------------- - -### [AWS ECS-Cluster](../components/aws/ecs-cluster/README.md) - -ECS, or EC2 Container Service, is able to run docker containers natively in AWS cloud. While the module can support classic EC2-based and Fargate, -features, this module generally prefers "ECS Fargete", which allows dynamic launching of docker containers with no always-on cost and no servers -to manage or pay for when tasks are not running. - -Use in combination with the `ECS-Task` component. - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/ecs-cluster?ref=master` -* See the [AWS ECS-Cluster Readme](../components/aws/ecs-cluster/README.md) for input/output specs and additional info. - -------------------- - -### [AWS ECS-Task](../components/aws/ecs-task/README.md) - -ECS, or EC2 Container Service, is able to run docker containers natively in AWS cloud. While the module can support classic EC2-based and Fargate, -features, this module generally prefers "ECS Fargete", which allows dynamic launching of docker containers with no always-on cost and no servers -to manage or pay for when tasks are not running. - -Use in combination with the `ECS-Cluster` component. - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/ecs-task?ref=master` -* See the [AWS ECS-Task Readme](../components/aws/ecs-task/README.md) for input/output specs and additional info. - -------------------- - -### [AWS Lambda-Python](../components/aws/lambda-python/README.md) - -AWS Lambda is a platform which enables serverless execution of arbitrary functions. This module specifically focuses on the -Python implementatin of Lambda functions. Given a path to a folder of one or more python fyles, this module takes care of -packaging the python code into a zip and uploading to a new Lambda Function in AWS. The module can also be configured with -S3-based triggers, to run the function automatically whenever a file is landed in a specific S3 path. - - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/lambda-python?ref=master` -* See the [AWS Lambda-Python Readme](../components/aws/lambda-python/README.md) for input/output specs and additional info. - -------------------- - -### [AWS RDS](../components/aws/rds/README.md) - -Deploys an RDS-backed database. RDS currently supports the following database engines: -* Aurora -* MySQL -* PostgreSQL -* Oracle -* SQL Server - -Each engine type has it's own required configuration. For already-configured database -configurations, see the catalog modules: `catalog/aws/mysql` and `catalog/aws/postgres` -which are built on top of this component module. - -* NOTE: Requires AWS policy 'AmazonRDSFullAccess' on the terraform account - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/rds?ref=master` -* See the [AWS RDS Readme](../components/aws/rds/README.md) for input/output specs and additional info. - -------------------- - -### [AWS Redshift](../components/aws/redshift/README.md) - -This is the underlying technical component which supports the Redshift catalog module. - -NOTE: Requires AWS policy 'AmazonRedshiftFullAccess' on the terraform account - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/redshift?ref=master` -* See the [AWS Redshift Readme](../components/aws/redshift/README.md) for input/output specs and additional info. - -------------------- - -### [AWS Secrets-Manager](../components/aws/secrets-manager/README.md) - -This module takes as input a set of maps from variable names to secrets locations (in YAML or -JSON). The module uploads those secrets to AWS Secrets Manager and returns the same map pointing -to the IDs of new AWS Secrets manager locations. Those IDs (aka ARNs) can then safely be handed -on to other resources which required access to those secrets. - -**Usage Notes:** - -* Any secrets locations which are already pointing to AWS secrets will simply be passed back through to the output with no changes. -* For security reasons, this module does not accept inputs for secrets using the clear text of the secrets themselves. To properly use this module, first save the secrets to a YAML or JSON file which is excluded from source control. - - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/secrets-manager?ref=master` -* See the [AWS Secrets-Manager Readme](../components/aws/secrets-manager/README.md) for input/output specs and additional info. - -------------------- - -### [AWS Step-Functions](../components/aws/step-functions/README.md) - -AWS Step Functions is a service provided by Amazon Web Services that makes it easier to orchestrate multiple AWS services -to accomplish tasks. Step Functions allows you to create steps in a process where the output of one step becomes the input -for another step. - - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/step-functions?ref=master` -* See the [AWS Step-Functions Readme](../components/aws/step-functions/README.md) for input/output specs and additional info. - -------------------- - -### [AWS VPC](../components/aws/vpc/README.md) - -The VPC module creates a number of network services which support other key AWS functions. - -Included automatically when creating this module: -* 1 VPC which contains the following: - * 2 private subnets (for resources which **do not** need a public IP address) - * 2 public subnets (for resources which do need a public IP address) - * 1 NAT gateway (allows private sugnet resources to reach the outside world) - * 1 Intenet gateway (allows resources in public and private subnets to reach the internet) - * route tables and routes to connect all of the above - -* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/vpc?ref=master` -* See the [AWS VPC Readme](../components/aws/vpc/README.md) for input/output specs and additional info. - -------------------- - - - -## Azure Components - -_(Coming soon)_ - -## GCP Components - -_(Coming soon)_ - -------------------- - -_**NOTE:** This documentation was [auto-generated](build.py) using -`terraform-docs` and `s-infra` from `slalom.dataops`. -Please do not attempt to manually update this file._ - + +# DataOps Infrastructure Components + +These components define the technical building blocks which enable advanced, ready-to-deploy data solutions in the [Infrastructure Catalog](../catalog/README.md). + +## Contents + +1. [AWS Components](#aws-components) + - [AWS EC2](#aws-ec2) + - [AWS ECR](#aws-ecr) + - [AWS ECS-Cluster](#aws-ecs-cluster) + - [AWS ECS-Task](#aws-ecs-task) + - [AWS Lambda-Python](#aws-lambda-python) + - [AWS RDS](#aws-rds) + - [AWS Redshift](#aws-redshift) + - [AWS Secrets-Manager](#aws-secrets-manager) + - [AWS Step-Functions](#aws-step-functions) + - [AWS VPC](#aws-vpc) + +2. Azure Components + * _(Coming soon)_ +2. GCP Components + * _(Coming soon)_ + +## AWS Components + +### [AWS EC2](../components/aws/ec2/README.md) + +EC2 is the virtual machine layer of the AWS platform. This module allows you to pass your own startup scripts, and it streamlines the creation and usage of +credentials (passwords and/or SSH keypairs) needed to connect to the instances. + + + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/ec2?ref=master` +* See the [AWS EC2 Readme](../components/aws/ec2/README.md) for input/output specs and additional info. + +------------------- + +### [AWS ECR](../components/aws/ecr/README.md) + +ECR (Elastic Compute Repository) is the private-hosted AWS equivalent of DockerHub. ECR allows you to securely publish docker images which +should not be accessible to external users. + + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/ecr?ref=master` +* See the [AWS ECR Readme](../components/aws/ecr/README.md) for input/output specs and additional info. + +------------------- + +### [AWS ECS-Cluster](../components/aws/ecs-cluster/README.md) + +ECS, or EC2 Container Service, is able to run docker containers natively in AWS cloud. While the module can support classic EC2-based and Fargate, +features, this module generally prefers "ECS Fargete", which allows dynamic launching of docker containers with no always-on cost and no servers +to manage or pay for when tasks are not running. + +Use in combination with the `ECS-Task` component. + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/ecs-cluster?ref=master` +* See the [AWS ECS-Cluster Readme](../components/aws/ecs-cluster/README.md) for input/output specs and additional info. + +------------------- + +### [AWS ECS-Task](../components/aws/ecs-task/README.md) + +ECS, or EC2 Container Service, is able to run docker containers natively in AWS cloud. While the module can support classic EC2-based and Fargate, +features, this module generally prefers "ECS Fargete", which allows dynamic launching of docker containers with no always-on cost and no servers +to manage or pay for when tasks are not running. + +Use in combination with the `ECS-Cluster` component. + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/ecs-task?ref=master` +* See the [AWS ECS-Task Readme](../components/aws/ecs-task/README.md) for input/output specs and additional info. + +------------------- + +### [AWS Lambda-Python](../components/aws/lambda-python/README.md) + +AWS Lambda is a platform which enables serverless execution of arbitrary functions. This module specifically focuses on the +Python implementatin of Lambda functions. Given a path to a folder of one or more python fyles, this module takes care of +packaging the python code into a zip and uploading to a new Lambda Function in AWS. The module can also be configured with +S3-based triggers, to run the function automatically whenever a file is landed in a specific S3 path. + + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/lambda-python?ref=master` +* See the [AWS Lambda-Python Readme](../components/aws/lambda-python/README.md) for input/output specs and additional info. + +------------------- + +### [AWS RDS](../components/aws/rds/README.md) + +Deploys an RDS-backed database. RDS currently supports the following database engines: +* Aurora +* MySQL +* PostgreSQL +* Oracle +* SQL Server + +Each engine type has it's own required configuration. For already-configured database +configurations, see the catalog modules: `catalog/aws/mysql` and `catalog/aws/postgres` +which are built on top of this component module. + +* NOTE: Requires AWS policy 'AmazonRDSFullAccess' on the terraform account + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/rds?ref=master` +* See the [AWS RDS Readme](../components/aws/rds/README.md) for input/output specs and additional info. + +------------------- + +### [AWS Redshift](../components/aws/redshift/README.md) + +This is the underlying technical component which supports the Redshift catalog module. + +NOTE: Requires AWS policy 'AmazonRedshiftFullAccess' on the terraform account + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/redshift?ref=master` +* See the [AWS Redshift Readme](../components/aws/redshift/README.md) for input/output specs and additional info. + +------------------- + +### [AWS Secrets-Manager](../components/aws/secrets-manager/README.md) + +This module takes as input a set of maps from variable names to secrets locations (in YAML or +JSON). The module uploads those secrets to AWS Secrets Manager and returns the same map pointing +to the IDs of new AWS Secrets manager locations. Those IDs (aka ARNs) can then safely be handed +on to other resources which required access to those secrets. + +**Usage Notes:** + +* Any secrets locations which are already pointing to AWS secrets will simply be passed back through to the output with no changes. +* For security reasons, this module does not accept inputs for secrets using the clear text of the secrets themselves. To properly use this module, first save the secrets to a YAML or JSON file which is excluded from source control. + + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/secrets-manager?ref=master` +* See the [AWS Secrets-Manager Readme](../components/aws/secrets-manager/README.md) for input/output specs and additional info. + +------------------- + +### [AWS Step-Functions](../components/aws/step-functions/README.md) + +AWS Step Functions is a service provided by Amazon Web Services that makes it easier to orchestrate multiple AWS services +to accomplish tasks. Step Functions allows you to create steps in a process where the output of one step becomes the input +for another step. + + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/step-functions?ref=master` +* See the [AWS Step-Functions Readme](../components/aws/step-functions/README.md) for input/output specs and additional info. + +------------------- + +### [AWS VPC](../components/aws/vpc/README.md) + +The VPC module creates a number of network services which support other key AWS functions. + +Included automatically when creating this module: +* 1 VPC which contains the following: + * 2 private subnets (for resources which **do not** need a public IP address) + * 2 public subnets (for resources which do need a public IP address) + * 1 NAT gateway (allows private sugnet resources to reach the outside world) + * 1 Intenet gateway (allows resources in public and private subnets to reach the internet) + * route tables and routes to connect all of the above + +* Source: `git::https://github.com/slalom-ggp/dataops-infra//components/aws/vpc?ref=master` +* See the [AWS VPC Readme](../components/aws/vpc/README.md) for input/output specs and additional info. + +------------------- + + + +## Azure Components + +_(Coming soon)_ + +## GCP Components + +_(Coming soon)_ + +------------------- + +_**NOTE:** This documentation was [auto-generated](../docs/build.py) using +`terraform-docs` and `s-infra` from `slalom.dataops`. +Please do not attempt to manually update this file._ + diff --git a/components/aws/ec2/README.md b/components/aws/ec2/README.md index e6daeb1d..40c5e560 100644 --- a/components/aws/ec2/README.md +++ b/components/aws/ec2/README.md @@ -52,6 +52,16 @@ credentials (passwords and/or SSH keypairs) needed to connect to the instances. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/components/aws/ecr/README.md b/components/aws/ecr/README.md index d227d384..e14f5802 100644 --- a/components/aws/ecr/README.md +++ b/components/aws/ecr/README.md @@ -29,6 +29,16 @@ should not be accessible to external users. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/components/aws/ecs-cluster/README.md b/components/aws/ecs-cluster/README.md index ffacccef..83c451a3 100644 --- a/components/aws/ecs-cluster/README.md +++ b/components/aws/ecs-cluster/README.md @@ -32,6 +32,17 @@ Use in combination with the `ECS-Task` component. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [iam.tf](iam.tf) +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/components/aws/ecs-task/README.md b/components/aws/ecs-task/README.md index 7e307d12..e7862273 100644 --- a/components/aws/ecs-task/README.md +++ b/components/aws/ecs-task/README.md @@ -23,6 +23,7 @@ Use in combination with the `ECS-Cluster` component. | environment | Standard `environment` module input. | 
object({
    vpc_id          = string
    aws_region      = string
    public_subnets  = list(string)
    private_subnets = list(string)
  })
| n/a | yes | | load\_balancer\_arn | n/a | `string` | n/a | yes | | name\_prefix | Standard `name_prefix` module input. | `string` | n/a | yes | +| permitted\_s3\_buckets | A list of bucket names, to which the ECS task will be granted read/write access. | `list(string)` | n/a | yes | | resource\_tags | Standard `resource_tags` module input. | `map(string)` | n/a | yes | | secrets\_manager\_kms\_key\_id | n/a | `string` | n/a | yes | | use\_fargate | n/a | `bool` | n/a | yes | @@ -53,6 +54,18 @@ Use in combination with the `ECS-Cluster` component. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [alb.tf](alb.tf) +* [iam.tf](iam.tf) +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/components/aws/lambda-python/README.md b/components/aws/lambda-python/README.md index 955b6bd6..61cc1a5b 100644 --- a/components/aws/lambda-python/README.md +++ b/components/aws/lambda-python/README.md @@ -34,6 +34,18 @@ S3-based triggers, to run the function automatically whenever a file is landed i --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [iam.tf](iam.tf) +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [python-zip.tf](python-zip.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/components/aws/rds/README.md b/components/aws/rds/README.md index fcc24056..bab0b9ad 100644 --- a/components/aws/rds/README.md +++ b/components/aws/rds/README.md @@ -46,6 +46,16 @@ which are built on top of this component module. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/components/aws/redshift/README.md b/components/aws/redshift/README.md index b346fd56..0f5ba3fa 100644 --- a/components/aws/redshift/README.md +++ b/components/aws/redshift/README.md @@ -37,6 +37,16 @@ NOTE: Requires AWS policy 'AmazonRedshiftFullAccess' on the terraform account --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/components/aws/secrets-manager/README.md b/components/aws/secrets-manager/README.md index 85d41e53..f3d4f426 100644 --- a/components/aws/secrets-manager/README.md +++ b/components/aws/secrets-manager/README.md @@ -70,6 +70,16 @@ on to other resources which required access to those secrets. ``` +--------------------- + +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + --------------------- _**NOTE:** This documentation was auto-generated using diff --git a/components/aws/step-functions/README.md b/components/aws/step-functions/README.md index 88c0af71..7032f1c8 100644 --- a/components/aws/step-functions/README.md +++ b/components/aws/step-functions/README.md @@ -29,6 +29,17 @@ for another step. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [iam.tf](iam.tf) +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/components/aws/vpc/README.md b/components/aws/vpc/README.md index a03ab961..43e1123e 100644 --- a/components/aws/vpc/README.md +++ b/components/aws/vpc/README.md @@ -38,6 +38,16 @@ Included automatically when creating this module: --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [main.tf](main.tf) +* [outputs.tf](outputs.tf) +* [variables.tf](variables.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index fb6b78ef..e0fa2319 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -11,7 +11,7 @@ 1. [Common Input and Output Variables](#common-input-and-output-variables) 2. [Show Usage in Samples](#show-usage-in-samples) 3. [Adhere to Formatting Standards](#adhere-to-formatting-standards) - 4. [Be Self-documenting](#be-self-documenting) + 4. [Create Self-documenting Modules](#create-self-documenting-modules) 5. [Follow Purpose-Driven Design Patterns](#follow-purpose-driven-design-patterns) ## Feature Requests @@ -42,18 +42,14 @@ When making changes to the terraform modules, please observe the following 3-tie > _Solution modules should be configurable as simply as possible to align with the stated business requirements of the module._ - ### Catalog Modules (Layer 2) > _**Catalog Modules (Layer 2)** should combine modules from Layer 3 in a method targeted to specific functional or business requirements._ - ### Component Modules (Layer 3) > _**Component Modules (Layer 3)** define the technical solutions as mapped to individual cloud product offerings (EC2, EC2, Azure Functions, etc.), meeting a technical requirement from one or more catalog modules which reference them._ - - ## Coding Standards Submitted PRs should meet the following code standards before being merged into `master`: @@ -73,24 +69,24 @@ In addition to custom variables, each AWS catalog and component module should su * As a rule, each component and category module should be referenced by at least one solution module in the `samples` folder, and the sample should demonstrate how to utilize the core functionality. - * In addition to providing an easy on-ramp for new users to learn how to use your module, your sample module is also an entrypoint for the CI/CD pipeline to perform automated tests. - * If your sample code ever stops working, the automated tests will catch this, giving us a means to catch and fix the breakages before they can impact users. + * In addition to providing an easy on-ramp for new users to learn how to use your module, your sample module is also an entrypoint for the CI/CD pipeline to perform automated tests. + * If your sample code ever stops working, the automated tests will catch this, giving us a means to catch and fix the breakages before they can impact users. ### Adhere to Formatting Standards * Terraform makes it very easy way to auto-format modules, which in turn ensures a consistent experience when reviewing code across multiple authors. - * If you use VS Code, the defaults specified in `settings.json` should automatically apply formatting on each file save. - * Formatting is checked automatically after each commit by the CI/CD pipeline. - * If you receive failures related to Terraform formatting, simply run `terraform fmt -recursive` from the root of the repo. This command will auto format the entire repo and then you can simply commit the resulting changes. + * If you use VS Code, the defaults specified in `settings.json` should automatically apply formatting on each file save. + * Formatting is checked automatically after each commit by the CI/CD pipeline. + * If you receive failures related to Terraform formatting, simply run `terraform fmt -recursive` from the root of the repo. This command will auto format the entire repo and then you can simply commit the resulting changes. -### Be Self-documenting +### Create Self-documenting Modules * In order for components to be effectively used by a broad audience, each module must be self-documenting and should be included in the Catalog auto-document tool. - * Make sure each input and output variable has it's `description` field set. - * Make sure each module has a `main.tf` file and that the file contains a header comment with a paragraph description of the basic module functions. See [components/aws/secrets-manager/main.tf](../components/aws/secrets-manager/main.tf) for a sample. - * All input variables should be stored in `variables.tf` and all output variables should be stored in `outputs.tf`. - * After the above are met, update the project docs by navigating to the `docs` directory and running `build.py`. This command will update all module README files as well as [catalog_index.md](catalog_index.md) and [components_index.md](components_index.md). + * Make sure each input and output variable has it's `description` field set. + * Make sure each module has a `main.tf` file and that the file contains a header comment with a paragraph description of the basic module functions. See [components/aws/secrets-manager/main.tf](../components/aws/secrets-manager/main.tf) for a sample. + * All input variables should be stored in `variables.tf` and all output variables should be stored in `outputs.tf`. + * After the above are met, update the project docs by navigating to the `docs` directory and running `build.py` (more details [here](README.md)). This command will update all module README files as well as [catalog/README.md](../catalog/README.md) and [components/README.md](../components/README.md). ### Follow Purpose-Driven Design Patterns @@ -98,4 +94,3 @@ In addition to custom variables, each AWS catalog and component module should su * There is no expectation that modules should be fully generic or meet every use case. * Opinionated and purpose-driven approaches are preferred versus trying to build modules that are one-size-fits-all. - diff --git a/docs/README.md b/docs/README.md index a1009e32..1f2fd5e4 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,17 @@ # `dataops-infra` Docs -These documents are published automatically to [infra.dataops.tk](https://infra.dataops.tk) using [Github Pages](https://help.github.com/en/github/working-with-github-pages) and [Jekyll](https://jekyllrb.com/docs/). +_These documents are published automatically to [infra.dataops.tk](https://infra.dataops.tk) using [Github Pages](https://help.github.com/en/github/working-with-github-pages) and [Jekyll](https://jekyllrb.com/docs/)._ + +## Rebuilding Docs + +To rebuild or update all module README docs, run the following command on your dev machine: + +```bash +cd docs +python build.py +``` + +To provide additional documentation for a module, you may optionally publish additional documentation into a `USAGE.md` or `NOTES.md` file. If one or both of these files exist within a module folder, the content from the additional file(s) will be automatically added into the auto-generated README.md. ## Local Testing diff --git a/docs/build.py b/docs/build.py index ab15cef9..7835275b 100644 --- a/docs/build.py +++ b/docs/build.py @@ -1,30 +1,31 @@ -from slalom.dataops import io, jobs, infra import os +from slalom.dataops import io, jobs, infra + SPECIAL_CASE_WORDS = ["AWS", "ECR", "ECS", "IAM", "VPC", "DBT", "EC2", "RDS", "MySQL"] CATALOG_TEMPLATE = """ -# DataOps Infrastructure {type} +# DataOps Infrastructure {index_type} {overview} ## Contents {toc} -2. Azure {type} +2. Azure {index_type} * _(Coming soon)_ -2. GCP {type} +2. GCP {index_type} * _(Coming soon)_ -## AWS {type} +## AWS {index_type} {aws} -## Azure {type} +## Azure {index_type} {azure} -## GCP {type} +## GCP {index_type} {gcp} @@ -48,12 +49,12 @@ def build_catalog_index(): build_index( "Catalog", tf_dir="../catalog", - output_file="catalog_index.md", + output_file="../catalog/README.md", overview_desc=( "The Infrastructure Catalog contains ready-to-deploy terraform modules for " "a variety of production data project use cases and POCs. For information " "about the technical building blocks used in these modules, please see the " - "catalog [components index](components_index.md)." + "catalog [components index](../components/README.md)." ), ) @@ -62,11 +63,11 @@ def build_component_index(): build_index( "Components", tf_dir="../components", - output_file="components_index.md", + output_file="../components/README.md", overview_desc=( "These components define the technical building blocks which enable " "advanced, ready-to-deploy data solutions in the " - "[Infrastructure Catalog](catalog_index.md)." + "[Infrastructure Catalog](../catalog/README.md)." ), ) @@ -90,13 +91,13 @@ def _proper(str, title_case=True): return " ".join(new_words) -def build_index(type: str, tf_dir: str, output_file: str, overview_desc: str): +def build_index(index_type: str, tf_dir: str, output_file: str, overview_desc: str): """ - Update the Documentation index. + Update the documentation index. Parameters: ---------- - type (str): Either 'Components' or 'Catalog'. + index_type (str): Either 'Components' or 'Catalog'. tf_dir (str): The directory to scan for modules. output_file (str): The location of the documentation file to create. overview_dsc (str): The overview that will appear at the top of the index. @@ -113,8 +114,8 @@ def build_index(type: str, tf_dir: str, output_file: str, overview_desc: str): toc_str = "" for platform_i, platform in enumerate(content_metadata.keys(), start=1): toc_str += ( - f"{platform_i}. [{_proper(platform)} {type}]" - f"(#{platform.lower()}-{type.lower()})\n" + f"{platform_i}. [{_proper(platform)} {index_type}]" + f"(#{platform.lower()}-{index_type.lower()})\n" ) print(f"Exploring platform '{platform}'") @@ -142,7 +143,7 @@ def build_index(type: str, tf_dir: str, output_file: str, overview_desc: str): aws=content_metadata["aws"], azure="_(Coming soon)_", gcp="_(Coming soon)_", - type=type, + index_type=index_type, overview=overview_desc, ) io.create_text_file(output_file, content) diff --git a/samples/airflow-on-aws/README.md b/samples/airflow-on-aws/README.md index bdb05566..dd50c2cd 100644 --- a/samples/airflow-on-aws/README.md +++ b/samples/airflow-on-aws/README.md @@ -20,6 +20,16 @@ No input. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [00_environment.tf](00_environment.tf) +* [01_data-lake.tf](01_data-lake.tf) +* [02_airflow.tf](02_airflow.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/samples/dbt-and-singer-on-aws/README.md b/samples/dbt-and-singer-on-aws/README.md index 3f8cb688..9da193ff 100644 --- a/samples/dbt-and-singer-on-aws/README.md +++ b/samples/dbt-and-singer-on-aws/README.md @@ -21,6 +21,17 @@ No input. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [00_environment.tf](00_environment.tf) +* [01_data-lake.tf](01_data-lake.tf) +* [02_singer-taps.tf](02_singer-taps.tf) +* [03_data-build-tool.tf](03_data-build-tool.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/samples/mysql-on-aws/README.md b/samples/mysql-on-aws/README.md index 09ac5897..c74127e2 100644 --- a/samples/mysql-on-aws/README.md +++ b/samples/mysql-on-aws/README.md @@ -19,6 +19,15 @@ No input. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [00_environment.tf](00_environment.tf) +* [01_rds_mysql.tf](01_rds_mysql.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/samples/postgres-on-aws/README.md b/samples/postgres-on-aws/README.md index d45c606a..48ff3e28 100644 --- a/samples/postgres-on-aws/README.md +++ b/samples/postgres-on-aws/README.md @@ -19,6 +19,15 @@ No input. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [00_environment.tf](00_environment.tf) +* [01_rds_postgres.tf](01_rds_postgres.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/samples/redshift-dw/README.md b/samples/redshift-dw/README.md index 999f1d01..974dc882 100644 --- a/samples/redshift-dw/README.md +++ b/samples/redshift-dw/README.md @@ -19,6 +19,16 @@ No input. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [00_environment.tf](00_environment.tf) +* [01_data-lake.tf](01_data-lake.tf) +* [02_redshift.tf](02_redshift.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/samples/s3-lambda-trigger/README.md b/samples/s3-lambda-trigger/README.md index 82d6e4b3..e3e1b264 100644 --- a/samples/s3-lambda-trigger/README.md +++ b/samples/s3-lambda-trigger/README.md @@ -19,6 +19,15 @@ No input. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [00_environment.tf](00_environment.tf) +* [01_data-lake-w-lambda.tf](01_data-lake-w-lambda.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/samples/secrets-manager-on-aws/README.md b/samples/secrets-manager-on-aws/README.md index a65f456a..8eb294e6 100644 --- a/samples/secrets-manager-on-aws/README.md +++ b/samples/secrets-manager-on-aws/README.md @@ -19,6 +19,15 @@ No input. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [00_environment.tf](00_environment.tf) +* [01_secrets-upload.tf](01_secrets-upload.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._ diff --git a/samples/tableau-on-aws/README.md b/samples/tableau-on-aws/README.md index d8992cf5..c1658628 100644 --- a/samples/tableau-on-aws/README.md +++ b/samples/tableau-on-aws/README.md @@ -19,6 +19,16 @@ No input. --------------------- +## Source Files + +_Source code for this module is available using the links below._ + +* [00_environment.tf](00_environment.tf) +* [01_data-lake.tf](01_data-lake.tf) +* [02_tableau-server.tf](02_tableau-server.tf) + +--------------------- + _**NOTE:** This documentation was auto-generated using `terraform-docs` and `s-infra` from `slalom.dataops`. Please do not attempt to manually update this file._