From 0dfc6621b13cf55428d6b654f45560abfc869a2e Mon Sep 17 00:00:00 2001 From: Paul Pilone Date: Thu, 18 Jul 2024 10:44:11 -0400 Subject: [PATCH 1/4] CUMULUS-3608: Adds variable for sqs message consumer watcher message and time limits (#3728) * Adds variable for sqs message consumer watcher message and time limits * Updates CL * WIP: Adds guidance notes to sqs message consumer variables --- CHANGELOG.md | 4 ++++ tf-modules/ingest/cloudwatch-events.tf | 4 ++-- tf-modules/ingest/variables.tf | 21 +++++++++++++++++++++ 3 files changed, 27 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 82f74a69e73..a3f6a5e2f60 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,10 @@ degraded execution table operations. - Added `Bulk Execution Delete` migration type to async operations types - **CUMULUS-3742** - Script for dumping data into postgres database for testing and replicating issues +- **CUMULUS-3608** + - Exposes variables for sqs_message_consumer_watcher messageLimit and timeLimit configurations. Descriptions + of the variables [here](tf-modules/ingest/variables.tf) include notes on usage and what users should + consider if configuring something other than the default values. ### Changed diff --git a/tf-modules/ingest/cloudwatch-events.tf b/tf-modules/ingest/cloudwatch-events.tf index 3645e8e438d..df420250620 100644 --- a/tf-modules/ingest/cloudwatch-events.tf +++ b/tf-modules/ingest/cloudwatch-events.tf @@ -45,8 +45,8 @@ resource "aws_cloudwatch_event_target" "sqs_message_consumer_watcher" { rule = aws_cloudwatch_event_rule.sqs_message_consumer_watcher.name arn = aws_lambda_function.sqs_message_consumer.arn input = jsonencode({ - messageLimit = 500 - timeLimit = 60 + messageLimit = var.sqs_message_consumer_watcher_message_limit + timeLimit = var.sqs_message_consumer_watcher_time_limit }) } diff --git a/tf-modules/ingest/variables.tf b/tf-modules/ingest/variables.tf index 6276f114834..a21df893325 100644 --- a/tf-modules/ingest/variables.tf +++ b/tf-modules/ingest/variables.tf @@ -196,3 +196,24 @@ variable "default_log_retention_days" { default = 30 description = "Optional default value that user chooses for their log retention periods" } + +variable "sqs_message_consumer_watcher_message_limit" { + type = number + default = 500 + description = < Date: Thu, 18 Jul 2024 11:43:24 -0400 Subject: [PATCH 2/4] Fixes wrong CL section for recent PRs (#3733) --- CHANGELOG.md | 38 +++++++++++++++++--------------------- 1 file changed, 17 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a3f6a5e2f60..c3917655f03 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,6 +48,20 @@ degraded execution table operations. - Exposes variables for sqs_message_consumer_watcher messageLimit and timeLimit configurations. Descriptions of the variables [here](tf-modules/ingest/variables.tf) include notes on usage and what users should consider if configuring something other than the default values. +- **CUMULUS-3449** + - Updated the following database columns to BIGINT: executions.cumulus_id, executions.parent_cumulus_id, + files.granule_cumulus_id, granules_executions.granule_cumulus_id, granules_executions.execution_cumulus_id + and pdrs.execution_cumulus_id + - Changed granules table unique constraint to granules_collection_cumulus_id_granule_id_unique + - Added indexes granules_granule_id_index and granules_provider_collection_cumulus_id_granule_id_index + to granules table +- **CUMULUS-3700** + - Added `volume_type` option to `elasticsearch_config` in the + `data-persistance` module to allow configuration of the EBS volume type for + Elasticsarch; default remains `gp2`. +- **CUMULUS-3424** + - Exposed `auto_pause` and `seconds_until_auto_pause` variables in + `cumulus-rds-tf` module to modify `aws_rds_cluster` scaling_configuration ### Changed @@ -69,6 +83,9 @@ degraded execution table operations. ### Fixed +- **CUMULUS-3785** + - Fixed `SftpProviderClient` not awaiting `decryptBase64String` with AWS KMS + - Fixed method typo in `@cumulus/api/endpoints/dashboard.js` - **CUMULUS-3787** - Fixed developer-side bug causing some ts error sto be swallowed in CI - **CUMULUS-3320** @@ -172,13 +189,6 @@ to update to at least [cumulus-ecs-task:2.1.0](https://hub.docker.com/layers/cum - **CUMULUS-2897** - Removed unused Systems Manager AWS SDK client. This change removes the Systems Manager client from the `@cumulus/aws-client` package. -- **CUMULUS-3449** - - Updated the following database columns to BIGINT: executions.cumulus_id, executions.parent_cumulus_id, - files.granule_cumulus_id, granules_executions.granule_cumulus_id, granules_executions.execution_cumulus_id - and pdrs.execution_cumulus_id - - Changed granules table unique constraint to granules_collection_cumulus_id_granule_id_unique - - Added indexes granules_granule_id_index and granules_provider_collection_cumulus_id_granule_id_index - to granules table - **CUMULUS-3779** - Updates async_operations Docker image to Node v20 and bumps its cumulus dependencies to v18.3.0 to support `aws-sdk` v3 changes. @@ -195,13 +205,6 @@ to update to at least [cumulus-ecs-task:2.1.0](https://hub.docker.com/layers/cum - **CUMULUS-3606** - Updated with additional documentation covering tunneling configuration using a PKCS11 provider -- **CUMULUS-3700** - - Added `volume_type` option to `elasticsearch_config` in the - `data-persistance` module to allow configuration of the EBS volume type for - Elasticsarch; default remains `gp2`. -- **CUMULUS-3424** - - Exposed `auto_pause` and `seconds_until_auto_pause` variables in - `cumulus-rds-tf` module to modify `aws_rds_cluster` scaling_configuration ### Changed @@ -284,13 +287,6 @@ to update to at least [cumulus-ecs-task:2.1.0](https://hub.docker.com/layers/cum ### Fixed -- **CUMULUS-3785** - - Fixed `SftpProviderClient` not awaiting `decryptBase64String` with AWS KMS - - Fixed method typo in `@cumulus/api/endpoints/dashboard.js` -- **CUMULUS-3320** - - Execution database deletions by `cumulus_id` should have greatly improved - performance as a table scan will no longer be required for each record - deletion to validate parent-child relationships - **CUMULUS-3715** - Update `ProvisionUserDatabase` lambda to correctly pass in knex/node debug flags to knex custom code From d0d6a63e1d1b4c02a89df7e4c829770722446371 Mon Sep 17 00:00:00 2001 From: Paul Pilone Date: Thu, 18 Jul 2024 15:53:21 -0400 Subject: [PATCH 3/4] Fixes async-operation image tag for 18.3.1 release (#3730) * Fixes async-operation image tag for 18.3.1 release * Fixes plural typo in async operations --- CHANGELOG.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c3917655f03..69fbbb645bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -100,12 +100,12 @@ degraded execution table operations. #### CUMULUS-3433 Update to node.js v20 The following applies only to users with a custom value configured for -`async_operation_image`: +`async-operation`: -- As part of the node v20 update process, a new version (49) of the Core +- As part of the node v20 update process, a new version (52) of the Core async-operation container was published - [cumuluss/async - operation](https://hub.docker.com/layers/cumuluss/async-operation) The - default value for `async_operation_image` has been updated in the `cumulus` + operation](https://hub.docker.com/layers/cumuluss/async-operation/52/images/sha256-78c05f9809c29707f9da87c0fc380d39a71379669cbebd227378c8481eb11c3a?context=explore) The + default value for `async-operation` has been updated in the `cumulus` module, however if you are using an internal image repository such as ECR, please make sure to update your deployment configuration with the newly provided image. From 5474dfe035c2b2fdfb2073ab65fe5b35c0a18b19 Mon Sep 17 00:00:00 2001 From: jennyhliu <34660846+jennyhliu@users.noreply.github.com> Date: Thu, 18 Jul 2024 18:10:28 -0400 Subject: [PATCH 4/4] CUMULUS-3449-1: Update upgrade instruction to install postgresql13 client (#3729) * CUMULUS-3449: Update doc skip-integration-tests --- CHANGELOG.md | 2 +- .../update-cumulus_id-type-indexes-CUMULUS-3449.md | 10 +++------- 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 69fbbb645bc..0a2b593f004 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,7 +28,7 @@ Please *carefully* review the migration [process documentation](https://nasa.git make these updates properly will likely result in deployment failure and/or degraded execution table operations. -#### CUMULUS-3449 Please follow instructions before upgrading Cumulus +#### CUMULUS-3449 Please follow the instructions before upgrading Cumulus - The updates in CUMULUS-3449 requires manual update to postgres database in production environment. Please follow [Update Cumulus_id Type and diff --git a/docs/upgrade-notes/update-cumulus_id-type-indexes-CUMULUS-3449.md b/docs/upgrade-notes/update-cumulus_id-type-indexes-CUMULUS-3449.md index ecdee9082e6..f27b525797d 100644 --- a/docs/upgrade-notes/update-cumulus_id-type-indexes-CUMULUS-3449.md +++ b/docs/upgrade-notes/update-cumulus_id-type-indexes-CUMULUS-3449.md @@ -59,10 +59,6 @@ There are many available resources for tmux and the psql client. The following i - [PostgreSQL Documentation](https://www.postgresql.org/docs/13/app-psql.html) - [Tmux tutorial](https://www.linuxtrainingacademy.com/tmux-tutorial/) -:::note -The version 9.x psql will be installed. This will work for all our standard SQL but some meta-commands like \d may not work. -::: - Basic commands for running SQL commands ```sh @@ -141,7 +137,8 @@ exit 3. Install tmux and postgres client ```sh - sudo yum install -y tmux postgresql.x86_64 + sudo yum install -y tmux + sudo amazon-linux-extras install postgresql13 ``` Once installed, a tmux session is started with two windows. The Cumulus database is connected to in each window @@ -209,8 +206,7 @@ exit 6. Verify the Updates - We can verify that the tables are updated successfully by checking the `\d table` results from psql. We can run the commands - outside of EC2 from pgAdmin or a psql client which supports `\d` command. The following are expected results. + We can verify that the tables are updated successfully by checking the `\d table` results from psql. The following are expected results. ```sh => \d executions;