Skip to content

Commit

Permalink
Merge branch 'develop' into feature/40-test-cloud-gov
Browse files Browse the repository at this point in the history
  • Loading branch information
lbeaufort committed Feb 2, 2022
2 parents 45f579b + c5d36f6 commit 044d6c3
Show file tree
Hide file tree
Showing 20 changed files with 206 additions and 515 deletions.
135 changes: 100 additions & 35 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,30 +10,47 @@ orbs:

# See: https://circleci.com/docs/2.0/configuration-reference/#jobs
jobs:
lint:
# These next lines defines a Docker executors: https://circleci.com/docs/2.0/executor-types/
# A list of available CircleCI Docker convenience images are available here: https://circleci.com/developer/images/image/cimg/python

dependency-check:
docker:
- image: cimg/python:3.7
- image: cimg/python:3.8

steps:
- checkout

- run:
name: Install python test requirements
command: pip install -r requirements-test.txt
working_directory: ~/project/
- python/install-packages:
pkg-manager: pip
pip-dependency-file: requirements-test.txt

- run:
name: Run lint
command: flake8 fecfiler
working_directory: ~/project/django-backend/

unit-test:
- run:
name: Run depency check
command: |
export today=$(date "+%Y-%m-%d")
# gather up the -i ignore IDs fro safety check
export ignores=$(
grep -vE "^\s*#" .safety.dependency.ignore | # print out any non-comment line
grep "[0-9]" | # filter out any line that doesn't have a number in it
awk -v "today=${today}" '{ if ($2 > today || $2 == "") print "-i", $1}' | # print any line after today
xargs echo # put all the output from previous command on one line
)
export command="safety check -r django-backend/requirements.txt --full-report $ignores"
echo "----------------------------------------------------"
echo "If you need to modify the ignore list for the safety"
echo "check, edit .safety.dependency.ignore file"
echo "----------------------------------------------------"
echo ""
echo "Running: $command"
eval $command
test:
# These next lines defines a Docker executors: https://circleci.com/docs/2.0/executor-types/
# A list of available CircleCI Docker convenience images are available here: https://circleci.com/developer/images/image/cimg/python
docker:
- image: cimg/python:3.7
- image: cimg/python:3.7-node
- image: cimg/postgres:12.8

steps:
Expand All @@ -49,6 +66,15 @@ jobs:
- checkout

- run:
name: Create unified requirements so CircleCI can cache them
command: |
cd ~/project/django-backend
ls -l
cat requirements.txt > requirements-all.txt
echo >> requirements-all.txt # blank in case new newline at end of requirements.txt
cat requirements-test.txt >> requirements-all.txt
- python/install-packages:
pkg-manager: pip
app-dir: ~/project/
Expand All @@ -68,33 +94,72 @@ jobs:
working_directory: ~/project/db

- run:
name: Create migrations
command: python manage.py makemigrations
name: Create/run migrations
command: |
python manage.py makemigrations
python manage.py migrate
working_directory: ~/project/django-backend/

# Only use SonarCloud linting or security checking for now.
# Leaving this as a comment in case SonarCloud isn't good enough and we need to re-enable
# - run:
# name: Run lint
# command: |
# flake8 --config django-backend/.flake8 --output-file flake8.out . || echo "Found lint errors: " && cat flake8.out
#
# - run:
# name: Bandit security checks
# command: |
# bandit -f json --output bandit.out -ii -ll -r . || echo "Bandit found issues" && cat bandit.out

- run:
name: run migrations
# This assumes pytest is installed via the install-package step above
command: python manage.py migrate
name: Run tests
# Use built-in Django test module
command: coverage run --source='.' manage.py test --keep
working_directory: ~/project/django-backend/

- run:
name: Create migrations
# This assumes pytest is installed via the install-package step above
command: python manage.py makemigrations
working_directory: ~/project/django-backend/
name: setup coverage dir
command: mkdir ~/project/coverage-reports

- run:
name: run migrations
# This assumes pytest is installed via the install-package step above
command: python manage.py migrate
working_directory: ~/project/django-backend/
name: Make XML coverage report
command: coverage xml -o ~/project/coverage-reports/coverage.xml
working_directory: ~/project/django-backend

# Sonar cloud setup and scanning
- run:
name: Run tests
# Use built-in Django test module
command: python manage.py test --keep
working_directory: ~/project/django-backend/
name: Create sonar-scanner cache directory if it doesn't exist
command: mkdir -p /tmp/cache/scanner
- restore_cache:
keys:
- v1-sonarcloud-scanner-4.6.2.2472
- run:
name: SonarCloud
command: |
set -e
VERSION=4.6.2.2472
if [ -z "$SONAR_TOKEN" ]; then
echo "You must set SONAR_TOKEN environemnt variable"
exit 1
fi
SCANNER_DIRECTORY=/tmp/cache/scanner
export SONAR_USER_HOME=$SCANNER_DIRECTORY/.sonar
OS="linux"
echo $SONAR_USER_HOME
if [[ ! -x "$SCANNER_DIRECTORY/sonar-scanner-$VERSION-$OS/bin/sonar-scanner" ]]; then
curl -Ol https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-$VERSION-$OS.zip
unzip -qq -o sonar-scanner-cli-$VERSION-$OS.zip -d $SCANNER_DIRECTORY
fi
chmod +x $SCANNER_DIRECTORY/sonar-scanner-$VERSION-$OS/bin/sonar-scanner
chmod +x $SCANNER_DIRECTORY/sonar-scanner-$VERSION-$OS/jre/bin/java
$SCANNER_DIRECTORY/sonar-scanner-$VERSION-$OS/bin/sonar-scanner
environment:
SONARQUBE_SCANNER_PARAMS: '{"sonar.host.url":"https://sonarcloud.io"}'
- save_cache:
key: v1-sonarcloud-scanner-4.6.2.2472
paths: /tmp/cache/scanner

deploy:

Expand Down Expand Up @@ -126,9 +191,9 @@ jobs:
workflows:
test: # This is the name of the workflow, feel free to change it to better match your workflow.
jobs:
- unit-test
- lint
- test
- dependency-check
- deploy:
requires:
- unit-test
- lint
- test
- dependency-check
14 changes: 14 additions & 0 deletions .safety.dependency.ignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Any vulnerability ID numbers listed in this file will be ignored when
# running the safety dependency check. Each line should have the ID number
# and a date. The ID will be ignored by the CI pipeline check unitl the date
# in YYYY-MM-DD format listed for that line.
# If no date is listed, the exception will never expire. (NOT RECOMMENDED)
#
# test
# Example:
# 40104 2022-01-15
#
44715 2022-03-01 # numpy - no fix yet
44717 2022-03-01 # numpy - dependencies not caught up yet
44716 2022-03-01 # numpy - dependencies not caught up yet
43975 2022-03-01 # urllib3 - botocore dependency needs to catch up
143 changes: 49 additions & 94 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,66 +13,62 @@ finance information. The project code is distributed across these repositories:

---

# Set up
## Set up

#### Prerequisites
### Prerequisites
Software necessary to run the application locally

[Docker](https://docs.docker.com/get-docker/)
[Docker Compose](https://docs.docker.com/compose/install/)
* [Docker](https://docs.docker.com/get-docker/)
* [Docker Compose](https://docs.docker.com/compose/install/)

## Docker basic usage.
when running docker-compose you will need to be in the root directory of the project. The reason for this is that docker-compose looks for docker-compose.yml to be in the same directory where it's run. You will also need at least 3GB of memory allocated for docker during the build.
### Docker basic usage.
When running docker-compose you will need to be in the root directory of the project. The reason for this is that docker-compose looks for docker-compose.yml to be in the same directory where it's run. You will also need at least 3GB of memory allocated for docker during the build.

## Run the front-end application
`docker-compose up`
# Shut down the containers
`docker-compose down`
# see all running containers
`docker ps`
# running commands in a running container
`docker compose exec <container name> <command>`
### Run the front-end application
`docker-compose up -d`

You should set the following environment variables in the shell where you are running 'docker-compose up'.
You should set the following environment variables in the shell where you are running 'docker-compose up -d'.
Proper values for the development variables are shown here as an example
```
export DATABASE_URL = "postgres://postgres:[email protected]/postgres"
export FECFILE_TEST_DB_NAME = "postgres"
```

# Shut down the containers
### Shut down the containers
`docker-compose down`
# see all running containers
### see all running containers
`docker ps`
# running commands in a running container
`docker compose exec <container name> <command>`
### running commands in a running container
`docker-compose exec <container name> <command>`


# Deployment (FEC team only)

### Create a changelog
If you're preparing a release to production, you should also create a changelog. The preferred way to do this is using the [changelog generator](https://github.com/skywinder/github-changelog-generator).

Once installed, run:

```
github_changelog_generator --since-tag <last public-relase> --t <your-gh-token>
```

When this finishes, commit the log to the release.

### Creating a new feature
### Create a feature branch
* Developer creates a feature branch and pushes to `origin`:

```
git flow feature start my-feature
git push origin feature/my-feature
git checkout develop
git pull
git checkout -b feature/my-feature develop
# Work happens here
git push --set-upstream origin feature/my-feature
```

* Reviewer merges feature branch into `develop` via GitHub
* Developer creates a GitHub PR when ready to merge to `develop` branch
* Reviewer reviews and merges feature branch into `develop` via GitHub
* [auto] `develop` is deployed to `dev`

### Creating a hotfix
### Create a release branch
* Developer creates a release branch and pushes to `origin`:

```
git checkout develop
git pull
git checkout -b release/sprint-# develop
git push --set-upstream origin release/sprint-#
```

### Create and deploy a hotfix
* Developer makes sure their local main and develop branches are up to date:

```
Expand All @@ -82,75 +78,36 @@ When this finishes, commit the log to the release.
git pull
```

* Developer creates a hotfix branch, commits changes, and **makes a PR to the `main` branch**:
* Developer creates a hotfix branch, commits changes, and **makes a PR to the `main` and `develop` branches**:

```
git flow hotfix start my-hotfix
git push origin hotfix/my-hotfix
git checkout -b hotfix/my-fix main
# Work happens here
git push --set-upstream origin hotfix/my-fix
```

* Reviewer merges hotfix branch into `develop` and `main` and pushes to `origin`:
* Reviewer merges hotfix/my-fix branch into `develop` and `main`
* [auto] `develop` is deployed to `dev`. Make sure the build passes before deploying to `main`.
* Developer deploys hotfix/my-fix branch to main using **Deploying a release to production** instructions below

```
git flow hotfix finish my-hotfix
git checkout develop
git push origin develop
```
### Deploying a release to production
* Developer creates a PR in GitHub to merge release/sprint-# branch into the `main` branch
* Reviewer approves PR and merges into `main`
* Check CircleCI for passing pipeline tests
* If tests pass, continue
* Delete release/sprint-# branch
* In GitHub, go to `Code -> tags -> releases -> Draft a new release`
* Publish a new release using tag sprint-#, be sure to Auto-generate release notes
* Deploy `sprint-#` tag to production

* `develop` is deployed to `dev`. Make sure the build passes before deploying to `main`.

```
git checkout main
git push origin main --follow-tags
```

* `main` is deployed to `prod`

### Creating a release
* Developer creates a release branch and pushes to `origin`:

```
git flow release start my-release
git push origin release/my-release
```

* [auto] `release/my-release` is deployed to `stage`
* Issue a pull request to main, tag reviewer(s)
* Review of staging
* Check if there are any SQL files changed. Depending on where the changes are, you may need to run migrations. Ask the person who made the change what, if anything, you need to run.
* Make sure your pull request has been approved
* Make sure local laptop copies of `main`, `develop`, and `release/[release name]` github branches are up-to-date by checking them out and using `git pull` for each branch.
* Rebuild candidate release branch, i.e., `release/public-YYYYMMDD`, in staging environment, and verify there are no errors and that build passes.
* Developer merges release branch into `main` (and backmerges into `develop`) and pushes to origin:

```
git config --global push.followTags true
git flow release finish my-release
```
You'll need to save several merge messages, and add a tag message which is named the name of the release (eg., public-beta-20170118). Check to see what `git branch` returns. If it shows you are on `main`, ignore the next step for checking out and pushing to `develop`.
```
git checkout develop
git push origin develop
```
Watch the develop build on Circle and make sure it passes. Now you are ready to push to prod (:tada:).

```
git checkout main
git log # make sure tag for release is present
git push origin main --follow-tags
```
Watch Circle to make sure it passes, then test the production site manually to make sure everything looks ok.

* `main` is deployed to `prod`
* `develop` is deployed to `dev`

## Additional developer notes
This section covers a few topics we think might help developers after setup.

### Git Secrets
Set up git secrets to protect oneself from committing sensitive information such as passwords to the repository.
- First install AWS git-secret utility in your PATH so it can be run at the command line: https://github.com/awslabs/git-secrets#installing-git-secrets
- Once you have git-secrets installed, run the fecfile-online/install-git-secrets-hook.sh shell script in the root directory of your cloned fecfile-online repo to install the pre-commit hooks.
- Once you have git-secrets installed, run the fecfile-web-api/install-git-secrets-hook.sh shell script in the root directory of your cloned fecfile-web-api repo to install the pre-commit hooks.
NOTE: The pre-commit hook is installed GLOBALLY by default so commits to all cloned repositories on your computer will be scanned for sensitive data. See the comments at the top of the script for local install options.
- See git-secrets README for more features: https://github.com/awslabs/git-secrets#readme

Expand All @@ -160,5 +117,3 @@ As a best practice policy, please commit any feature code changes made during th
### Google-style inline documentation
The project is using the Google Python Style Guide as the baseline to keep code style consistent across project repositories.
See here for comment style rules: https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings


Loading

0 comments on commit 044d6c3

Please sign in to comment.