Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix spam on ICA rotator (and general upgrade) #411

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions cdk/apps/ica_credentials/.mypy.ini
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
[mypy]
strict = True

[mypy-aws_cdk.*]
ignore_missing_imports = True

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was only here because cdk used to be missing these

[mypy-boto3.*]
ignore_missing_imports = True

Expand Down
65 changes: 65 additions & 0 deletions cdk/apps/ica_credentials/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
EXECUTABLES = python3 uv npx
K := $(foreach exec,$(EXECUTABLES), $(if $(shell which $(exec)),some string,$(error "No $(exec) in PATH")))

#
# by default we do everything to set up for running python and check the source is good
#

all: setup-python-cdk typecheck

setup: setup-python-cdk

setup-python-cdk: .venv/install.touch

.PHONY: all setup setup-python-cdk

#
# the actual targets that should be targeted for running the various interesting activities
#

deploy-cdk-dev: setup-python-cdk typecheck
. .venv/bin/activate; npx --yes cdk deploy ica-credentials-dev/stack

deploy-cdk-dev-v2: setup-python-cdk typecheck
. .venv/bin/activate; npx --yes cdk deploy ica-credentials-dev-v2/stack

typecheck:
. .venv/bin/activate; mypy lambdas/notify_slack_lambda/app
. .venv/bin/activate; mypy lambdas/jwt_producer_lambda/app
. .venv/bin/activate; mypy cdk
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try this typechecker soon - https://github.com/python/mypy
It looks promising.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

100% - Python without mypy is like writing straight javascript


format:
. .venv/bin/activate; black .

.PHONY: deploy-cdk undeploy-cdk typecheck format

#
# clean up working folders
#

clean: clean-python-cdk

clean-python-cdk:
find . -type f -name '*.py[co]' -delete -o -type d -name __pycache__ -delete
rm -rf .venv/
rm -rf cdk.out/
rm -rf .mypy_cache/

.PHONY: clean clean-python-cdk

# actual rules that create files/folders and build things

# track the existence of the virtual with the activate script - everything is essentially dependent on this
# NOTE: the python version chosen here should match that chosen for lambdas in the CDK!
.venv/bin/activate:
uv venv -p 3.12

requirements.txt: requirements.in
. .venv/bin/activate; uv pip compile requirements.in -o requirements.txt

requirements-dev.txt: requirements-dev.in
. .venv/bin/activate; uv pip compile requirements-dev.in -o requirements-dev.txt

# track the actual time of pip install with a touch file - so that it will re-trigger on changes to requirements
.venv/install.touch: .venv/bin/activate requirements.txt requirements-dev.txt
. .venv/bin/activate; uv pip install -r requirements.txt -r requirements-dev.txt && touch .venv/install.touch
32 changes: 22 additions & 10 deletions cdk/apps/ica_credentials/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ICA Credentials

A stack for managing ICA credentials and the production of up to date JWTs.
A stack for managing ICA credentials and the production of up-to-date JWTs (as AWS secrets).

## Setup

Expand Down Expand Up @@ -48,21 +48,33 @@ The trick is:
- then apply the CDK changes

If your CDK stack gets stuck in "Failed Rollback":
- manually change each secret to Disabled
- manually change each secret rotation to Disabled
- go to the Cloud Formation and 'continue rollback'
- manually change each secret again to Disabled (the rollback will have re-enabled them)
- then apply the CDK changes


## Dev

### Create Python virtual environment and install the dependencies
A development system requires a working Python and Node.

```bash
python3.8 -m venv .venv
source .venv/bin/activate
# [Optional] Needed to upgrade dependencies and cleanup unused packages
pip install pip-tools==6.1.0
./scripts/install-deps.sh
```
`make` should be all that is required to do a setup and type check of the source code.

To actually deploy to dev, use

`make deploy-cdk-dev`

whilst in a shell with AWS access keys for dev. The deployed stack in dev will
perform rotations but only message to Slack infrequently. There are various
settings in the code if you want to test more frequent Slack messaging.

If you want to change the Python requirements, just edit the relevant `requirements.in`
file and then do a `make`. It will re-compile the actual `requirements.txt` (and maybe
also possibly update package versions).

NOTE: currently the lambdas do *not* require any Python libraries
(other than AWS and urllib which are built in) so are
built very simply by the CDK (they do not have their own `requirements.txt`). This
might need to change - at which point the CDK build will need to be more
sophisticated.

2 changes: 1 addition & 1 deletion cdk/apps/ica_credentials/cdk.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"app": "python app.py"
"app": "python cdk/app.py"
}
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
import os

from aws_cdk import core as cdk
from aws_cdk import App, Environment
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are all the v1 -> v2 migration


from deployment import IcaCredentialsDeployment

app = cdk.App()
app = App()

CDK_APP_NAME = "ica-credentials"
CDK_APP_PYTHON_VERSION = "3.8"

ICA_BASE_URL = "https://aps2.platform.illumina.com"
ICAV2_BASE_URL = "https://ica.illumina.com"
Expand All @@ -33,12 +32,22 @@
SLACK_WEBHOOK_SSM_NAME,
github_repos=[CWL_ICA_GITHUB_REPO],
github_role_name=f"{CDK_APP_NAME}-dev-umccr-pipelines-deployment-role",
env=cdk.Environment(
account=os.environ["CDK_DEFAULT_ACCOUNT"],
region=os.environ["CDK_DEFAULT_REGION"],
),
env=Environment(account="843407916570", region="ap-southeast-2"),
)

# V2 (single token)
IcaCredentialsDeployment(
app,
f"{CDK_APP_NAME}-dev-v2",
None, # Token does not require project context in v2
None, # Token does not require additional project list in v2
ICAV2_BASE_URL,
SLACK_HOST_SSM_NAME,
SLACK_WEBHOOK_SSM_NAME,
env=Environment(account="843407916570", region="ap-southeast-2"),
)


# Staging
IcaCredentialsDeployment(
app,
Expand All @@ -52,10 +61,7 @@
SLACK_WEBHOOK_SSM_NAME,
github_repos=[CWL_ICA_GITHUB_REPO],
github_role_name=f"{CDK_APP_NAME}-stg-umccr-pipelines-deployment-role",
env=cdk.Environment(
account="455634345446",
region="ap-southeast-2"
),
env=Environment(account="455634345446", region="ap-southeast-2"),
)

# Production
Expand All @@ -72,25 +78,7 @@
SLACK_WEBHOOK_SSM_NAME,
github_repos=[CWL_ICA_GITHUB_REPO],
github_role_name=f"{CDK_APP_NAME}-prod-umccr-pipelines-deployment-role",
env=cdk.Environment(
account="472057503814",
region="ap-southeast-2"
),
)

# V2 (single token)
IcaCredentialsDeployment(
app,
f"{CDK_APP_NAME}-dev-v2",
None, # Token does not require project context in v2
None, # Token does not require additional project list in v2
ICAV2_BASE_URL,
SLACK_HOST_SSM_NAME,
SLACK_WEBHOOK_SSM_NAME,
env=cdk.Environment(
account=os.environ["CDK_DEFAULT_ACCOUNT"],
region=os.environ["CDK_DEFAULT_REGION"],
),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took out the ability for these to be deployed account agnostic - because I don't know the use case for that?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One for Alexis

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to drop this

env=Environment(account="472057503814", region="ap-southeast-2"),
)

app.synth()
Original file line number Diff line number Diff line change
@@ -1,23 +1,24 @@
from typing import Any, List, Optional, Dict
from typing import List, Optional, Any

from aws_cdk import core as cdk
from constructs import Construct
from aws_cdk import App, Stack, Stage

from secrets.infrastructure import Secrets


class IcaCredentialsDeployment(cdk.Stage):
class IcaCredentialsDeployment(Stage):
def __init__(
self,
scope: cdk.Construct,
scope: Construct,
id_: str,
data_project: Optional[str],
workflow_projects: Optional[List[str]],
ica_base_url: str,
slack_host_ssm_name: str,
slack_webhook_ssm_name: str,
github_role_name: Optional[str] = None,
github_repos: Optional[List] = None,
**kwargs,
github_repos: Optional[List[str]] = None,
**kwargs: Any,
):
"""
Represents the deployment of our stack(s) to a particular environment with a particular set of settings.
Expand All @@ -34,7 +35,7 @@ def __init__(
"""
super().__init__(scope, id_, **kwargs)

stateful = cdk.Stack(self, "stack")
stateful = Stack(self, "stack")

# this name becomes the prefix of our secrets so we slip in the word ICA to make it
# obvious when someone sees them that they are associated with ICA
Expand All @@ -44,9 +45,9 @@ def __init__(
data_project,
workflow_projects,
ica_base_url,
"cron(0 4/12 * * ? *)",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using cron expressions to give more certainty to timings

slack_host_ssm_name,
slack_webhook_ssm_name,
github_role_name=github_role_name,
github_repos=github_repos,
cdk_env=kwargs.get("env")
)
Loading