Merge branch 'jk/CUMULUS-3807' of https://github.com/nasa/cumulus int…

…o jk/CUMULUS-3807

CHANGELOG.md

@@ -28,7 +28,7 @@ Please *carefully* review the migration [process documentation](https://nasa.git
 make these updates properly will likely result in deployment failure and/or
 degraded execution table operations.
 
-#### CUMULUS-3449 Please follow instructions before upgrading Cumulus
+#### CUMULUS-3449 Please follow the instructions before upgrading Cumulus
 
 - The updates in CUMULUS-3449 requires manual update to postgres database in
   production environment. Please follow [Update Cumulus_id Type and
@@ -44,6 +44,24 @@ degraded execution table operations.
   - Added `Bulk Execution Delete` migration type to async operations types
 - **CUMULUS-3742**
   - Script for dumping data into postgres database for testing and replicating issues
+- **CUMULUS-3608**
+  - Exposes variables for sqs_message_consumer_watcher messageLimit and timeLimit configurations. Descriptions
+    of the variables [here](tf-modules/ingest/variables.tf) include notes on usage and what users should
+    consider if configuring something other than the default values.
+- **CUMULUS-3449**
+  - Updated the following database columns to BIGINT: executions.cumulus_id, executions.parent_cumulus_id,
+    files.granule_cumulus_id, granules_executions.granule_cumulus_id, granules_executions.execution_cumulus_id
+    and pdrs.execution_cumulus_id
+  - Changed granules table unique constraint to granules_collection_cumulus_id_granule_id_unique
+  - Added indexes granules_granule_id_index and granules_provider_collection_cumulus_id_granule_id_index
+    to granules table
+- **CUMULUS-3700**
+  - Added `volume_type` option to `elasticsearch_config` in the
+    `data-persistance` module to allow configuration of the EBS volume type for
+    Elasticsarch; default remains `gp2`.
+- **CUMULUS-3424**
+  - Exposed `auto_pause` and `seconds_until_auto_pause` variables in
+    `cumulus-rds-tf` module to modify `aws_rds_cluster` scaling_configuration
 
 ### Changed
 
@@ -67,6 +85,9 @@ degraded execution table operations.
 
 - **CUMULUS-3807**
   - Pinned @aws-sdk/client-s3 to 3.614 to address timeout/bug in s3().listObjectsV2
+- **CUMULUS-3785**
+  - Fixed `SftpProviderClient` not awaiting `decryptBase64String` with AWS KMS
+  - Fixed method typo in `@cumulus/api/endpoints/dashboard.js`
 - **CUMULUS-3787**
   - Fixed developer-side bug causing some ts error sto be swallowed in CI
 - **CUMULUS-3320**
@@ -81,12 +102,12 @@ degraded execution table operations.
 #### CUMULUS-3433 Update to node.js v20
 
 The following applies only to users with a custom value configured for
-`async_operation_image`:
+`async-operation`:
 
-- As part of the node v20 update process, a new version (49) of the Core
+- As part of the node v20 update process, a new version (52) of the Core
   async-operation container was published - [cumuluss/async
-  operation](https://hub.docker.com/layers/cumuluss/async-operation)  The
-  default value for `async_operation_image` has been updated in the `cumulus`
+  operation](https://hub.docker.com/layers/cumuluss/async-operation/52/images/sha256-78c05f9809c29707f9da87c0fc380d39a71379669cbebd227378c8481eb11c3a?context=explore)  The
+  default value for `async-operation` has been updated in the `cumulus`
   module, however if you are using an internal image repository such as ECR,
   please make sure to update your deployment configuration with the newly
   provided image.
@@ -170,13 +191,6 @@ to update to at least [cumulus-ecs-task:2.1.0](https://hub.docker.com/layers/cum
 - **CUMULUS-2897**
   - Removed unused Systems Manager AWS SDK client. This change removes the Systems Manager client
     from the `@cumulus/aws-client` package.
-- **CUMULUS-3449**
-  - Updated the following database columns to BIGINT: executions.cumulus_id, executions.parent_cumulus_id,
-    files.granule_cumulus_id, granules_executions.granule_cumulus_id, granules_executions.execution_cumulus_id
-    and pdrs.execution_cumulus_id
-  - Changed granules table unique constraint to granules_collection_cumulus_id_granule_id_unique
-  - Added indexes granules_granule_id_index and granules_provider_collection_cumulus_id_granule_id_index
-    to granules table
 - **CUMULUS-3779**
   - Updates async_operations Docker image to Node v20 and bumps its cumulus dependencies to v18.3.0 to
     support `aws-sdk` v3 changes.
@@ -193,13 +207,6 @@ to update to at least [cumulus-ecs-task:2.1.0](https://hub.docker.com/layers/cum
 - **CUMULUS-3606**
   - Updated  with additional documentation covering tunneling configuration
     using a PKCS11 provider
-- **CUMULUS-3700**
-  - Added `volume_type` option to `elasticsearch_config` in the
-    `data-persistance` module to allow configuration of the EBS volume type for
-    Elasticsarch; default remains `gp2`.
-- **CUMULUS-3424**
-  - Exposed `auto_pause` and `seconds_until_auto_pause` variables in
-    `cumulus-rds-tf` module to modify `aws_rds_cluster` scaling_configuration
 
 ### Changed
 
@@ -282,13 +289,6 @@ to update to at least [cumulus-ecs-task:2.1.0](https://hub.docker.com/layers/cum
 
 ### Fixed
 
-- **CUMULUS-3785**
-  - Fixed `SftpProviderClient` not awaiting `decryptBase64String` with AWS KMS
-  - Fixed method typo in `@cumulus/api/endpoints/dashboard.js`
-- **CUMULUS-3320**
-  - Execution database deletions by `cumulus_id` should have greatly improved
-    performance as a table scan will no longer be required for each record
-    deletion to validate parent-child relationships
 - **CUMULUS-3715**
   - Update `ProvisionUserDatabase` lambda to correctly pass in knex/node debug
     flags to knex custom code

docs/upgrade-notes/update-cumulus_id-type-indexes-CUMULUS-3449.md

@@ -59,10 +59,6 @@ There are many available resources for tmux and the psql client. The following i
 - [PostgreSQL Documentation](https://www.postgresql.org/docs/13/app-psql.html)
 - [Tmux tutorial](https://www.linuxtrainingacademy.com/tmux-tutorial/)
 
-:::note
-The version 9.x psql will be installed. This will work for all our standard SQL but some meta-commands like \d may not work.
-:::
-
 Basic commands for running SQL commands
 
 ```sh
@@ -141,7 +137,8 @@ exit
 3. Install tmux and postgres client
 
     ```sh
-    sudo yum install -y tmux postgresql.x86_64
+    sudo yum install -y tmux
+    sudo amazon-linux-extras install postgresql13
     ```
 
     Once installed, a tmux session is started with two windows. The Cumulus database is connected to in each window
@@ -209,8 +206,7 @@ exit
 
 6. Verify the Updates
 
-     We can verify that the tables are updated successfully by checking the `\d table` results from psql.  We can run the commands
-     outside of EC2 from pgAdmin or a psql client which supports `\d` command.  The following are expected results.
+     We can verify that the tables are updated successfully by checking the `\d table` results from psql.  The following are expected results.
 
     ```sh
     => \d executions;

tf-modules/ingest/cloudwatch-events.tf

@@ -45,8 +45,8 @@ resource "aws_cloudwatch_event_target" "sqs_message_consumer_watcher" {
   rule = aws_cloudwatch_event_rule.sqs_message_consumer_watcher.name
   arn  = aws_lambda_function.sqs_message_consumer.arn
   input = jsonencode({
-    messageLimit = 500
-    timeLimit    = 60
+    messageLimit = var.sqs_message_consumer_watcher_message_limit
+    timeLimit    = var.sqs_message_consumer_watcher_time_limit
   })
 }
 

tf-modules/ingest/variables.tf

@@ -196,3 +196,24 @@ variable "default_log_retention_days" {
   default = 30
   description = "Optional default value that user chooses for their log retention periods"
 }
+
+variable "sqs_message_consumer_watcher_message_limit" {
+  type = number
+  default = 500
+  description = <<EOF
+    Number of messages the SQS message consumer Lambda will attempt to read from SQS in a single execution. 
+    Note that increasing this value may result in a direct increase/decrease in triggered workflows. Users should
+    only adjust this value with the understanding of how it will impact the number of queued workflows in their 
+    system.
+  EOF
+}
+
+variable "sqs_message_consumer_watcher_time_limit" {
+  type = number
+  default = 60
+  description = <<EOF
+    Number of seconds the SQS message consumer Lambda will remain active and polling for new messages. Note that this value 
+    should be less than the overall Lambda invocation timeout or else the Lambda may be terminated while still actively
+    polling SQS. This value should be adjusted in conjunction with sqs_message_consumer_watcher_message_limit.
+  EOF
+}