Skip to content

Commit

Permalink
update ETL documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
lstod committed May 30, 2024
1 parent 04e0c1a commit 5866dbb
Showing 1 changed file with 12 additions and 1 deletion.
13 changes: 12 additions & 1 deletion bin/migrate-oats-data/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,18 +49,29 @@ Pre-launch ETL command has been changed to:

`python migrate_1.py [action]`

Post-launch ETL command is `migrate_2.py` and imports everything post launch inclusive of SRWs
Post-launch ETL command is `migrate_2.py` and imports everything post launch inclusive of SRWs but not past SRW

ETLs post SRWs take over migrate.py command and is used as before

Commands for post-launch are stored in:

`menu/post_launch_commands`

Actions are contained within the migrate file, not all actions are available for each version of `migrate.py`. `import` and `clean` are shared for all

ETLs created post launch are to be located in the post-launch folder of their directory i.e.

`applications/post_launch` or `noi/post-launch`

`migrate_2.py` now contains the 2nd batch of ETL done

`migrate.py` contains the latest and final ETL migration which includes Planning Reviews and Inquiries.

### Future ETL
The rest of ETL / data fixes have been done with migrations, these can be found in `services/apps/alcs/src/providers/typeorm/migrations`. Going forward data fixes and other minor changes should be done with migrations.

If a new major ETL is required it is advisable to create a new updated migrate.py file and perform the required updates. Ensure obfuscation keeps up with the new changes. Depending on the scope the ETL can be run on openshift or run on a secure machine locally then restore the database from there.


## Prod data obfuscation

Expand Down

0 comments on commit 5866dbb

Please sign in to comment.