feat: Added support for VPC Flow Logs in Parquet format (terraform-aws-modules#700)

drewmullen · web-flow · commit 2423e4ff84e5 · 2021-10-14T21:28:26.000+02:00
diff --git a/.github/contributing.md b/.github/contributing.md
@@ -0,0 +1,34 @@
+# Contributing
+
+When contributing to this repository, please first discuss the change you wish to make via issue,
+email, or any other method with the owners of this repository before making a change.
+
+Please note we have a code of conduct, please follow it in all your interactions with the project.
+
+## Pull Request Process
+
+1. Update the README.md with details of changes including example hcl blocks and [example files](./examples) if appropriate.
+2. Run pre-commit hooks `pre-commit run -a`.
+3. Once all outstanding comments and checklist items have been addressed, your contribution will be merged! Merged PRs will be included in the next release. The terraform-aws-vpc maintainers take care of updating the CHANGELOG as they merge.
+
+## Checklists for contributions
+
+- [ ] Add [semantics prefix](#semantic-pull-requests) to your PR or Commits (at least one of your commit groups)
+- [ ] CI tests are passing
+- [ ] README.md has been updated after any changes to variables and outputs. See https://github.com/terraform-aws-modules/terraform-aws-vpc/#doc-generation
+- [ ] Run pre-commit hooks `pre-commit run -a`
+
+## Semantic Pull Requests
+
+To generate changelog, Pull Requests or Commits must have semantic and must follow conventional specs below:
+
+- `feat:` for new features
+- `fix:` for bug fixes
+- `improvement:` for enhancements
+- `docs:` for documentation and examples
+- `refactor:` for code refactoring
+- `test:` for tests
+- `ci:` for CI purpose
+- `chore:` for chores stuff
+
+The `chore` prefix skipped during changelog generation. It can be used for `chore: update changelog` commit message by example.
diff --git a/README.md b/README.md
@@ -120,6 +120,14 @@ You can add additional tags with `intra_subnet_tags` as with other subnet types.
 
 VPC Flow Log allows to capture IP traffic for a specific network interface (ENI), subnet, or entire VPC. This module supports enabling or disabling VPC Flow Logs for entire VPC. If you need to have VPC Flow Logs for subnet or ENI, you have to manage it outside of this module with [aws_flow_log resource](https://www.terraform.io/docs/providers/aws/r/flow_log.html).
 
+### VPC Flow Log Examples
+
+By default `file_format` is `plain-text`. You can also specify `parquet` to have logs written in Apache Parquet format.
+
+```
+flow_log_file_format = "parquet"
+```
+
 ### Permissions Boundary
 
 If your organization requires a permissions boundary to be attached to the VPC Flow Log role, make sure that you specify an ARN of the permissions boundary policy as `vpc_flow_log_permissions_boundary` argument. Read more about required [IAM policy for publishing flow logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-cwl.html#flow-logs-iam).
@@ -183,19 +191,25 @@ It is possible to integrate this VPC module with [terraform-aws-transit-gateway
 - [Manage Default VPC](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/manage-default-vpc)
 - [Few tests and edge case examples](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/issues)
 
+## Contributing
+
+Report issues/questions/feature requests on in the [issues](https://github.com/terraform-aws-modules/terraform-aws-vpc/issues/new) section.
+
+Full contributing [guidelines are covered here](.github/contributing.md).
+
 <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 ## Requirements
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.38 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.38 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.63 |
 
 ## Modules
 
@@ -370,8 +384,11 @@ No modules.
 | <a name="input_flow_log_cloudwatch_log_group_retention_in_days"></a> [flow\_log\_cloudwatch\_log\_group\_retention\_in\_days](#input\_flow\_log\_cloudwatch\_log\_group\_retention\_in\_days) | Specifies the number of days you want to retain log events in the specified log group for VPC flow logs. | `number` | `null` | no |
 | <a name="input_flow_log_destination_arn"></a> [flow\_log\_destination\_arn](#input\_flow\_log\_destination\_arn) | The ARN of the CloudWatch log group or S3 bucket where VPC Flow Logs will be pushed. If this ARN is a S3 bucket the appropriate permissions need to be set on that bucket's policy. When create\_flow\_log\_cloudwatch\_log\_group is set to false this argument must be provided. | `string` | `""` | no |
 | <a name="input_flow_log_destination_type"></a> [flow\_log\_destination\_type](#input\_flow\_log\_destination\_type) | Type of flow log destination. Can be s3 or cloud-watch-logs. | `string` | `"cloud-watch-logs"` | no |
+| <a name="input_flow_log_file_format"></a> [flow\_log\_file\_format](#input\_flow\_log\_file\_format) | (Optional) The format for the flow log. Valid values: `plain-text`, `parquet`. | `string` | `"plain-text"` | no |
+| <a name="input_flow_log_hive_compatible_partitions"></a> [flow\_log\_hive\_compatible\_partitions](#input\_flow\_log\_hive\_compatible\_partitions) | (Optional) Indicates whether to use Hive-compatible prefixes for flow logs stored in Amazon S3. | `bool` | `false` | no |
 | <a name="input_flow_log_log_format"></a> [flow\_log\_log\_format](#input\_flow\_log\_log\_format) | The fields to include in the flow log record, in the order in which they should appear. | `string` | `null` | no |
 | <a name="input_flow_log_max_aggregation_interval"></a> [flow\_log\_max\_aggregation\_interval](#input\_flow\_log\_max\_aggregation\_interval) | The maximum interval of time during which a flow of packets is captured and aggregated into a flow log record. Valid Values: `60` seconds or `600` seconds. | `number` | `600` | no |
+| <a name="input_flow_log_per_hour_partition"></a> [flow\_log\_per\_hour\_partition](#input\_flow\_log\_per\_hour\_partition) | (Optional) Indicates whether to partition the flow log per hour. This reduces the cost and response time for queries. | `bool` | `false` | no |
 | <a name="input_flow_log_traffic_type"></a> [flow\_log\_traffic\_type](#input\_flow\_log\_traffic\_type) | The type of traffic to capture. Valid values: ACCEPT, REJECT, ALL. | `string` | `"ALL"` | no |
 | <a name="input_igw_tags"></a> [igw\_tags](#input\_igw\_tags) | Additional tags for the internet gateway | `map(string)` | `{}` | no |
 | <a name="input_instance_tenancy"></a> [instance\_tenancy](#input\_instance\_tenancy) | A tenancy option for instances launched into the VPC | `string` | `"default"` | no |
diff --git a/examples/complete-vpc/README.md b/examples/complete-vpc/README.md
@@ -21,14 +21,14 @@ Note that this example may create resources which can cost money (AWS Elastic IP
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.28 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.63 |
 
 ## Modules
 
diff --git a/examples/complete-vpc/versions.tf b/examples/complete-vpc/versions.tf
@@ -1,10 +1,10 @@
 terraform {
-  required_version = ">= 0.12.31"
+  required_version = ">= 0.13.1"
 
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 3.28"
+      version = ">= 3.63"
     }
   }
 }
diff --git a/examples/ipv6/README.md b/examples/ipv6/README.md
@@ -19,8 +19,8 @@ Note that this example may create resources which can cost money (AWS Elastic IP
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
 
 ## Providers
 
diff --git a/examples/ipv6/versions.tf b/examples/ipv6/versions.tf
@@ -1,10 +1,10 @@
 terraform {
-  required_version = ">= 0.12.31"
+  required_version = ">= 0.13.1"
 
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 3.28"
+      version = ">= 3.63"
     }
   }
 }
diff --git a/examples/issues/README.md b/examples/issues/README.md
@@ -24,8 +24,8 @@ Note that this example may create resources which can cost money (AWS Elastic IP
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
 
 ## Providers
 
diff --git a/examples/issues/versions.tf b/examples/issues/versions.tf
@@ -1,10 +1,10 @@
 terraform {
-  required_version = ">= 0.12.31"
+  required_version = ">= 0.13.1"
 
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 3.28"
+      version = ">= 3.63"
     }
   }
 }
diff --git a/examples/manage-default-vpc/README.md b/examples/manage-default-vpc/README.md
@@ -21,8 +21,8 @@ Run `terraform destroy` when you don't need these resources.
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
 
 ## Providers
 
diff --git a/examples/manage-default-vpc/versions.tf b/examples/manage-default-vpc/versions.tf
@@ -1,10 +1,10 @@
 terraform {
-  required_version = ">= 0.12.31"
+  required_version = ">= 0.13.1"
 
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 3.28"
+      version = ">= 3.63"
     }
   }
 }
diff --git a/examples/network-acls/README.md b/examples/network-acls/README.md
@@ -23,8 +23,8 @@ Note that this example may create resources which can cost money (AWS Elastic IP
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
 
 ## Providers
 
diff --git a/examples/network-acls/versions.tf b/examples/network-acls/versions.tf
@@ -1,10 +1,10 @@
 terraform {
-  required_version = ">= 0.12.31"
+  required_version = ">= 0.13.1"
 
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 3.28"
+      version = ">= 3.63"
     }
   }
 }
diff --git a/examples/outpost/README.md b/examples/outpost/README.md
@@ -23,14 +23,14 @@ Note that this example may create resources which can cost money (AWS Elastic IP
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.28 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.63 |
 
 ## Modules
 
diff --git a/examples/outpost/versions.tf b/examples/outpost/versions.tf
@@ -1,10 +1,10 @@
 terraform {
-  required_version = ">= 0.12.31"
+  required_version = ">= 0.13.1"
 
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 3.28"
+      version = ">= 3.63"
     }
   }
 }
diff --git a/examples/secondary-cidr-blocks/README.md b/examples/secondary-cidr-blocks/README.md
@@ -21,8 +21,8 @@ Note that this example may create resources which can cost money (AWS Elastic IP
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
 
 ## Providers
 
diff --git a/examples/secondary-cidr-blocks/versions.tf b/examples/secondary-cidr-blocks/versions.tf
@@ -1,10 +1,10 @@
 terraform {
-  required_version = ">= 0.12.31"
+  required_version = ">= 0.13.1"
 
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 3.28"
+      version = ">= 3.63"
     }
   }
 }
diff --git a/examples/simple-vpc/README.md b/examples/simple-vpc/README.md
@@ -25,8 +25,8 @@ Note that this example may create resources which can cost money (AWS Elastic IP
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
 
 ## Providers
 
diff --git a/examples/simple-vpc/versions.tf b/examples/simple-vpc/versions.tf
@@ -1,10 +1,10 @@
 terraform {
-  required_version = ">= 0.12.31"
+  required_version = ">= 0.13.1"
 
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 3.28"
+      version = ">= 3.63"
     }
   }
 }
diff --git a/examples/vpc-flow-logs/README.md b/examples/vpc-flow-logs/README.md
@@ -23,15 +23,15 @@ Note that this example may create resources which can cost money (AWS Elastic IP
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
 | <a name="requirement_random"></a> [random](#requirement\_random) | >= 2 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.28 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.63 |
 | <a name="provider_random"></a> [random](#provider\_random) | >= 2 |
 
 ## Modules
@@ -42,6 +42,7 @@ Note that this example may create resources which can cost money (AWS Elastic IP
 | <a name="module_vpc_with_flow_logs_cloudwatch_logs"></a> [vpc\_with\_flow\_logs\_cloudwatch\_logs](#module\_vpc\_with\_flow\_logs\_cloudwatch\_logs) | ../../ |  |
 | <a name="module_vpc_with_flow_logs_cloudwatch_logs_default"></a> [vpc\_with\_flow\_logs\_cloudwatch\_logs\_default](#module\_vpc\_with\_flow\_logs\_cloudwatch\_logs\_default) | ../../ |  |
 | <a name="module_vpc_with_flow_logs_s3_bucket"></a> [vpc\_with\_flow\_logs\_s3\_bucket](#module\_vpc\_with\_flow\_logs\_s3\_bucket) | ../../ |  |
+| <a name="module_vpc_with_flow_logs_s3_bucket_parquet"></a> [vpc\_with\_flow\_logs\_s3\_bucket\_parquet](#module\_vpc\_with\_flow\_logs\_s3\_bucket\_parquet) | ../../ |  |
 
 ## Resources
 
diff --git a/examples/vpc-flow-logs/main.tf b/examples/vpc-flow-logs/main.tf
@@ -31,6 +31,25 @@ module "vpc_with_flow_logs_s3_bucket" {
   }
 }
 
+module "vpc_with_flow_logs_s3_bucket_parquet" {
+  source = "../../"
+
+  name = "vpc-flow-logs-s3-bucket"
+  cidr = "10.30.0.0/16"
+
+  azs            = ["${local.region}a"]
+  public_subnets = ["10.30.101.0/24"]
+
+  enable_flow_log           = true
+  flow_log_destination_type = "s3"
+  flow_log_destination_arn  = module.s3_bucket.this_s3_bucket_arn
+  flow_log_file_format      = "parquet"
+
+  vpc_flow_log_tags = {
+    Name = "vpc-flow-logs-s3-bucket"
+  }
+}
+
 # CloudWatch Log Group and IAM role created automatically
 module "vpc_with_flow_logs_cloudwatch_logs_default" {
   source = "../../"
diff --git a/examples/vpc-flow-logs/versions.tf b/examples/vpc-flow-logs/versions.tf
@@ -1,10 +1,10 @@
 terraform {
-  required_version = ">= 0.12.31"
+  required_version = ">= 0.13.1"
 
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 3.28"
+      version = ">= 3.63"
     }
 
     random = {
diff --git a/examples/vpc-separate-private-route-tables/README.md b/examples/vpc-separate-private-route-tables/README.md
@@ -21,8 +21,8 @@ Note that this example may create resources which can cost money (AWS Elastic IP
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
 
 ## Providers
 
diff --git a/examples/vpc-separate-private-route-tables/versions.tf b/examples/vpc-separate-private-route-tables/versions.tf
@@ -1,10 +1,10 @@
 terraform {
-  required_version = ">= 0.12.31"
+  required_version = ">= 0.13.1"
 
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = ">= 3.28"
+      version = ">= 3.63"
     }
   }
 }
diff --git a/modules/vpc-endpoints/README.md b/modules/vpc-endpoints/README.md
@@ -55,7 +55,7 @@ module "endpoints" {
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
 | <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
 
 ## Providers
diff --git a/modules/vpc-endpoints/versions.tf b/modules/vpc-endpoints/versions.tf
diff --git a/variables.tf b/variables.tf
diff --git a/versions.tf b/versions.tf
diff --git a/vpc-flow-logs.tf b/vpc-flow-logs.tf

Original file line number	Diff line number	Diff line change
`@@ -1,10 +1,10 @@`
`1`	`1`	`terraform {`
`2`		`- required_version = ">= 0.12.31"`
	`2`	`+ required_version = ">= 0.13.1"`
`3`	`3`
`4`	`4`	`required_providers {`
`5`	`5`	`aws = {`
`6`	`6`	`source = "hashicorp/aws"`
`7`		`- version = ">= 3.28"`
	`7`	`+ version = ">= 3.63"`
`8`	`8`	`}`
`9`	`9`	`}`
`10`	`10`	`}`