From 716162d0209b473ef6ec9961e500aedca36272ef Mon Sep 17 00:00:00 2001 From: Reem Ibrahim Date: Thu, 17 Oct 2024 16:10:33 +0100 Subject: [PATCH] updated manual --- .DS_Store | Bin 0 -> 6148 bytes docs/README.md | 13 ++- docs/_sidebar.md | 2 - docs/deployments.md | 23 ++---- docs/infrastructure.md | 102 ++++------------------- docs/new-environments.md | 173 --------------------------------------- docs/onboarding.md | 15 ++-- docs/users.md | 2 +- 8 files changed, 38 insertions(+), 292 deletions(-) create mode 100644 .DS_Store delete mode 100644 docs/new-environments.md diff --git a/.DS_Store b/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..5918e88f0f5c3813e487293451f4024d2993088e GIT binary patch literal 6148 zcmeHKJ5Iwu5S3P>@I1gUWaw}2baQUfIipyVKF zc(XesUO9;<5JG51+I_q8*%?1ao-7fW&LnFSwTLLjU~H^oS|DuaQIm#fnn1&6jE3#* zpvcEHPgVR?2Ke1Yv`a&}pzGlMZ9ME%ZF!NV{ZY|}E}nioAKl%*9)$L6rG0!}`E6|R z(a4)oK?Ax9@;1iDRa-T0@A2g6_A9T%=EXekB72ZGa(SC{iM$z|(JAF_HK7r9VIA*p zH>hK&Z?!JF^D>pXR`}2B{EmOMZnIya6cuGa8Bhj(i~;m)mUz>n&dPu?pbX3y;O|3% z!Dt0bkM7ff$z1_}5!6nwZt((`2wK6?BPp36;RVnNmL8#j=pO+`gHFo8uQKon7i@Dz literal 0 HcmV?d00001 diff --git a/docs/README.md b/docs/README.md index 74174cb..3e114dc 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,17 +10,14 @@ Supplier-facing application: https://reportmi.crowncommercial.gov.uk CCS administration application: https://api.reportmi.crowncommercial.gov.uk/admin -The applications are also running in development, staging and preprod environments for testing at different stages. They can be accessed at the following URLs: +The applications are also running in development and staging environments for testing at different stages. They can be accessed at the following URLs: - development: - * https://www.development.rmi-paas.ai-cloud.uk/ - * https://api.development.rmi-paas.ai-cloud.uk/admin/ + * https://www.dev.reportmi.crowncommercial.gov.uk/ + * https://admin.dev.reportmi.crowncommercial.gov.uk/admin - staging - * https://www.staging.rmi-paas.ai-cloud.uk/ - * https://api.staging.rmi-paas.ai-cloud.uk/admin/ -- preprod - * https://www.preprod.rmi-paas.ai-cloud.uk/ - * https://api.preprod.rmi-paas.ai-cloud.uk/admin/ + * https://www.staging.reportmi.crowncommercial.gov.uk/ + * https://admin.staging.reportmi.crowncommercial.gov.uk/admin ## Principles https://www.gov.uk/service-manual diff --git a/docs/_sidebar.md b/docs/_sidebar.md index e86132e..20a76ad 100644 --- a/docs/_sidebar.md +++ b/docs/_sidebar.md @@ -4,10 +4,8 @@ * [Glossary](glossary.md) * [Users](users.md) * [Framework Definition Language](framework-definition-language.md) -* [Submissions](submissions.md) * [Validations](validations.md) * [Data Warehouse Export](data-warehouse-export.md) * [Decision Records](decision-records.md) -* [Setup New Environments](new-environments.md) * [Infrastructure](infrastructure.md) * [Deployments](deployments.md) diff --git a/docs/deployments.md b/docs/deployments.md index a6838b8..b5636f7 100644 --- a/docs/deployments.md +++ b/docs/deployments.md @@ -1,16 +1,8 @@ # Release process -The platform is currently hosted on the Government Platform as a Service (GPaaS). - -Cloud Foundary scripts exist within each application in the /CF directory. These scripts are responsible for infrastructure changes as a separate task. - ## Migrations -Migrations are manual. - -1. [get access to a rails console on the box](infrastructure.md) -1. run `bin/rails db:migrate` -1. run `cf restart ccs-rmi-api-staging` for these changes to take effect +Migrations are manual. See [infrastructure access](infrastructure.md) ## CHANGELOG @@ -32,7 +24,6 @@ Having tested in staging, reviewed and merged the relevant Feature/* branches ba ### 1. Create a Release/* branch - Create a branch from `develop` for the release called `Release/X` where X is the release number - - Test the release cut in the preprod environment. ### 2. Create a tag for the release: ```bash @@ -54,16 +45,20 @@ git push origin --tags are safe to ship and that CHANGELOG.md accurately reflects the changes included in the release. -### 5. Make a pull request to merge Release/* branch back into develop +### 5. Deploy the master branch in Jenkins + + - This is currently a separate step. + +### 6. Run any new migrations if required - - Get the pull request reviewed and approved. + - See guidance on manually running migrations -### 6. Production smoke test +### 7. Production smoke test Once the code has been deployed to production, carry out a quick smoke test to confirm that the changes have been successfully deployed. -### 7. Update Jira +### 8. Update Jira Update Jira to reflect the newly deployed tickets. diff --git a/docs/infrastructure.md b/docs/infrastructure.md index e9ce627..740a149 100644 --- a/docs/infrastructure.md +++ b/docs/infrastructure.md @@ -1,109 +1,35 @@ # Infrastructure access -The Data Submission Service is hosted on [GOV.UK's Platform as a Service](https://www.cloud.service.gov.uk/) which is based on [Cloud Foundry](https://www.cloudfoundry.org/). - -[Documentation for interacting with the PaaS can be found on GOV.UK](https://docs.cloud.service.gov.uk/). +RMI is hosted on AWS, general documentation can be found [here](https://docs.aws.amazon.com/) ## Prerequisites -1. GOV.UK PaaS account - the lead or senior developer can create this account for you and you'll recieve an email -2. [install command line tools](https://docs.cloud.service.gov.uk/get_started.html#get-an-account) for shell access +1. AWS account - TechOps can set you up with access to RMI. +2. [install command line tools](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) -## Get shell access on a box +## Access a running instance -Login to the right Cloud Foundary account +Login via the command line ```bash -$ cf login -a api.london.cloud.service.gov.uk -u +$ aws --profile ecs execute-command --cluster --task --container admin --interactive --command "/bin/sh" ``` +PROFILE is as set in your ~/.aws/config file +CLUSTER can be found in the console at Amazon Elastic Container Service > Clusters, then TASK can be found by copying the arn of the appropriate task. -Select the right environment using `cf target` +To check a database migration status ```bash -$ cf target -s staging +$ rails db:migrate:status ``` -SSH onto an application running within that environment +To run a database migration + ```bash -$ cf ssh ccs-rmi-api-staging +$ rails db:migrate:up VERSION= ``` -Start a Rails console +To start a Rails console ```bash -$ /tmp/lifecycle/shell $ bin/rails console ``` - -## Viewing environment variables - -The [GOV.UK PaaS documentation for viewing the current value for variables](https://docs.cloud.service.gov.uk/deploying_apps.html#environment-variables) can be used. - -For example you can view staging variables by: - -``` -cf login -a api.london.cloud.service.gov.uk -u -cf target -s staging -cf apps -cf env APP_NAME -``` - -## Change an environment variable - -We use [user provided services](https://docs.cloudfoundry.org/devguide/services/user-provided.html) to manage environment variables on the PaaS, this is different to the [default documentation](https://docs.cloud.service.gov.uk/deploying_apps.html#environment-variables). - -### Prerequisites - -#### Secrets - -Obtain the environment variable files. Ask a team mate for their location. - -#### Scripts - -The scripts that help us make these changes are found in the [frontend repository within the CF directory](https://github.com/Crown-Commercial-Service/DataSubmissionService/tree/develop/CF). You should have this repository and directory checked out to continue. - -#### JSON Parser - -``` -brew install jq -``` - -### Making a change - -1. Make a change to the environment variable file -2. Copy the contents into a new local file `DataSubmissionService/CF/.env.cf.staging` -3. Run the script to apply the changes `./create-user-services.sh -u -p -o ccs-report-management-info -s staging` - -### Verify the change - -``` -cf apps -cf env -``` - -### Deploy the change - -Using rolling deployment strategy (flag: --strategy rolling) with CF Cli v7, (instead of the previous V3 and the blue/green plugin), allows us to execute deployments without any downtime to the user natively. - -``` -cf apps -cf restart -``` - -### Watch the deployment progress - -You should see the process count for the intended application change as new processes are spun up and handed over to. This can take a few minutes. - -``` -watch cf apps -``` - -Once this is complete the containers should be running the new environment variables. - -## Restart an application - -Redeploy an application from apps with downtime - -``` -cf apps -cf restage -``` diff --git a/docs/new-environments.md b/docs/new-environments.md deleted file mode 100644 index f1e41e1..0000000 --- a/docs/new-environments.md +++ /dev/null @@ -1,173 +0,0 @@ -# Setting Up An Environment For RMI - -## Getting Started & Prerequisites - -The Report Management Information System is hosted on Cloud Foundry, using the gov GPaaS service layer. Inside of this infrastrucure, we have [Organisations](https://docs.cloud.service.gov.uk/orgs_spaces_users.html#organisations) which will hold the Cloud Foundry [Spaces](https://docs.cloud.service.gov.uk/orgs_spaces_users.html#spaces), or environments, that can be named and created as seen fit. For the RMI project, we have a Organisation called ['ccs-report-management-info'](https://admin.london.cloud.service.gov.uk/organisations/b2daa20a-d281-4874-bd10-0bbb494480bc). - -Optionally, you may also need a [Domain](https://docs.cloud.service.gov.uk/deploying_services/configure_cdn/#configure-your-custom-domain-in-cloud-foundry) that you wish to use for routing your apps, in the new environment, is required. This may be an exisiting one, already setup in Cloud Foundry, or a brand new one. New ones must be setup and confirmed with CCS Tech Ops. - -As result you will first need both a GOV UK PaaS Account and the Cloud Foundry cli v7 setup. See [Infrastructure](https://crown-commercial-service.github.io/ReportMI-service-manual/#/infrastructure?id=prerequisites) for more information. For the rolling deployment strategy and a successful push, cloud foundry cli v7 must be installed. This should also be the case in the travis yml file, where the deployment pipeline installs cloud foundry cli. - -Furthermore, a GPaaS user with admin priviliges is required when creating a new Space or Domain in Cloud Foundry. - -
-
- -## Create a Space & Domain - -A 'Space' in cloud foundry serves as our environments. As such to create a new development or deployment environment, you first need to login to the Gov PaaS Cloud Foundry cli, and then create a new space in your targeted Organisation, (with the correct priviliges level). For this project, we use the 'ccs-report-management-info' org: -```bash -cf login -a api.london.cloud.service.gov.uk -u -``` -```bash -cf create-space -o -``` -^ The optional quota flag is ignored and omitted here.
- -Once a Space has been created, this will be the starting point for our new environment. You can target into this new environment using the command: -```bash -cf target -s -``` -
- -To list already available Domains, use: -```bash -cf domains -``` -Then, if you need to add a new Domain, you can use: -```bash -cf create-domain -``` - -
- -* This whole step/section requires a GPaaS Admin, as creating a new Space is a restricted action.
-** New Domains will first need to be setup and confirmed with the CCS Tech Ops team.
- -
-
- -## Create & Setup Backing Services - -The initial steps you must carry out first are automatically done by a set of '.sh' [shell scripts](https://github.com/Crown-Commercial-Service/DataSubmissionService/tree/develop/CF), which are included in the [Front-End](https://github.com/Crown-Commercial-Service/DataSubmissionService) of RMI. - -These shell script are ran from inside terminal/cmd, within the CF/ folder. - -
- -1. Create & Initialise the Space, then configure Domain/CDN: -```bash -./create-cf-space.sh -u -p -o -s -``` -2. Once the previous shell script has fully completed, and all operations done, run the next script, which adds and configures all the remaining services: -```bash -./create-user-space.sh -u -p -o -s -``` - -
-
- -## Pushing The Apps - -Next is to push each of the required apps to Cloud Foundry, using the final '.sh' shell script provided in both the [Front-End](https://github.com/Crown-Commercial-Service/DataSubmissionService), and the [Back-End](https://github.com/Crown-Commercial-Service/DataSubmissionServiceAPI), and also the [Router](https://github.com/Crown-Commercial-Service/DataSubmissionServiceRouter) of RMI. - -Again, you must be running these shell scripts inside your terminal/cmd, from within the CF/ folder. - -
- -1. Starting in the the [Front-End](https://github.com/Crown-Commercial-Service/DataSubmissionService/blob/develop/CF/deploy-app.sh) CF folder, run: -```bash -./deploy-app.sh -u -p -o -s -``` -2. And now again, but from the [API Back-End](https://github.com/Crown-Commercial-Service/DataSubmissionServiceAPI/blob/develop/CF/deploy-app.sh) CF folder, run: -```bash -./deploy-app.sh -u -p -o -s -``` -3. Finally the same once more, but for the [Router](https://github.com/Crown-Commercial-Service/DataSubmissionServiceRouter/blob/master/CF/deploy-app.sh), from the CF folder, run: -```bash -./deploy-app.sh -u -p -o -s -``` - -
- -You will now have 5 applications successfully pushed and running, (seen by running 'cf apps' inside the new space), with a full set of backing services bound & supporting these apps, (seen by running 'cf services' inside the new space). - -
-
- -## Setup CDN Service & Routing Mapping -Now to setup any routing with a domain and associated CDN service, Use cf create-domain if you need to add a domain, which should be done first, (see above). With any domains you wish to use available, from your desired space, run: - -```bash -cf create-service cdn-route cdn-route ccs-rmi-cdn- -c '{"domain": ",,...", "headers": ["*"]}' -``` - -^ If you get an error, along the lines of a service already exists, then you can use the "cf delete-service \" command, to delete the service causing the problem, (it is an incomplete service).
- -After this you must bind any apps required to use this new CDN service, and also map these new routes: - -```bash -cf map-route --hostname -``` -```bash -cf bind-route-service --hostname -``` - -
- -* Typically, the service name used for SERVICE_NAME is something along the lines of "ccs-rmi-api-admin-route-\".
-** Note that for the hostname, it is either "API" or "WWW" depending on route and application involved. This does not include a full stop, (this is intentional). - -
-
- -## Add Network Policy - -Penultimately, we must create a network policy for our apps cliet and server to communicate with each other. Failure to do this step will result in a 500 Error, accompanied with a Connection Refused. By standard; CCS opens the default 8080 port, so this is the port we will use to setup the tcp network policy, by running: -```bash -cf add-network-policy --destination-app --protocol tcp --port 8080 -``` - -^ The public app is your client or front-end app, while the private app is your server or back-end app. - -
-
- -## Setup Databases - -Lastly, we need to connect to both the Front-End App and the Back-End Api applications running in your new space/environment, and setup the databases, through their schemas. To do that we first enable a couple of plugins/extensions that are required, (and not enabled by default). Run both these commands from your terminal/cmd, inside the space/environment: -```bash -cf update-service ccs-rmi-app- -c '{"enable_extensions":["pgcrypto","plpgsql"],"reboot":true}' -``` -```bash -cf update-service ccs-rmi-api- -c '{"enable_extensions":["pgcrypto","plpgsql"],"reboot":true}' -``` - -
- -Next, connect to both the App and Api applications. Connect by ssh, match the app's environment to yours, and finally run/load the database schema: -```bash -cf ssh -``` -```bash -/tmp/lifecycle/shell -``` -```bash -bin/rails db:schema:load -``` - -
-
-
- -And with that, you should now have a fully functioning environment, all good to go. The next steps would be visit both front-end and back-end urls, to ensure they are working, and then to onboard yourself as a user, via logging into the back-end api admin app. - -
-
- -## [OPTIONAL] Add Routes to Service Vendor Allow Lists - -This final section is not always necessary, and so is an optional step unless you get errors when signing in to either of the front or back end of the system. The errors to typically expect, while different depending on which service and link visited, will summise down to the url you are visiting from in order to login (via Auth0 or Google Client) is not allowed. As a result, it would be a case of adding any newly created domains + hostname (e.g.: api.staging.rmi-paas.ai-cloud.uk, or, www.staging.rmi-paas.ai-cloud.uk, etc), inside either the [Auth0 CCS Dashboard](https://manage.auth0.com/dashboard/eu/reportmi/), or the CCS Google G Suite lists (ticket to Cabinet Office IT required). Furthermore you may be required to also update the user provided service "ADMIN_EMAILS", which contains a list of users that are allowed to login to the admin site. Update this by adding your own email to the environment varibles list, as follows: -```bash -cf update-user-provided-service ADMIN_EMAILS -p '{"ADMIN_EMAILS": ",,..."}' -``` \ No newline at end of file diff --git a/docs/onboarding.md b/docs/onboarding.md index f2c2175..c3f7584 100644 --- a/docs/onboarding.md +++ b/docs/onboarding.md @@ -2,14 +2,17 @@ 1. Read the [project service manual](https://crown-commercial-service.github.io/ReportMI-service-manual/#/) 1. [Rollbar](https://rollbar.com/crowncommercial/all/items/) - CCS-TechOps can invite you -1. [Papertrail](https://papertrailapp.com/dashboard) - the lead developer or RMI service manager will be able to add you -1. [GOV.UK PaaS](https://www.cloud.service.gov.uk/sign-in) - the lead/senior developers or RMI service manager will be able to invite you as a user +1. [Auth0](https://manage.auth0.com/dashboard/eu/reportmi/) - TechOps or a dev team mate can add you +1. [AWS](https://aws.amazon.com/console/) - TechOps can provide access 1. Google rooms - any team mate can add you: - * RMI devs * Report MI service support -1. [Jira](https://crowncommercialservice.atlassian.net/jira/software/projects/RMI/boards/94/backlog) - the product manager will be able to add you -1. GitHub - the lead or senior developer can add you + * RMI/Workday Information + * Plus Ultra +1. [Jira](https://crowncommercialservice.atlassian.net/jira/people/712020%3A954e2dd2-5ae0-450e-b826-9af18ec35288/boards/222/backlog) - the product manager will be able to add you +1. GitHub - TechOps can provide access * https://github.com/Crown-Commercial-Service/ReportMI-service-manual * https://github.com/Crown-Commercial-Service/DataSubmissionService * https://github.com/Crown-Commercial-Service/DataSubmissionServiceAPI -1. Contact a team mate for access to service secrets +1. Jenkins - TechOps can provide access + * https://jenkins-eks.techopsdev.com/job/rmi/job/build-and-deploy-pipeline/job/build-and-deploy-pipeline-api/ + * https://jenkins-eks.techopsdev.com/job/rmi/job/build-and-deploy-pipeline/job/build-and-deploy-pipeline-frontend/ diff --git a/docs/users.md b/docs/users.md index 54c6491..dccaf52 100644 --- a/docs/users.md +++ b/docs/users.md @@ -9,7 +9,7 @@ Supplier users, once created, can be in one of two states: When active a supplier user can sign in to the service and submit management information on behalf of any of linked suppliers. #### Inactive -When inactive a supplier user can no longer sign in to the service and so cannot submit management information. A supplier user is never delete to maintain the record of which user submitted management information to the service. +When inactive a supplier user can no longer sign in to the service and so cannot submit management information. A supplier user is never deleted to maintain the record of which user submitted management information to the service. ## Crown Commercial Service users A CCS user can sign in to the service administration application. The user us authenticated via the Cabinet Office GSuite service. Once signed in, a CCS user has access to all areas of the service admin application