initial push

.gitattributes

@@ -0,0 +1,2 @@
+risctool_open-source_v0.1.tar filter=lfs diff=lfs merge=lfs -text
+risctool_docker.tar filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1 @@
+.env

LICENSE.txt

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2024, EPRI
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the [ORGANIZATION] nor the names of its contributors may be
+   used to endorse or promote products derived from this software without
+   specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -0,0 +1,100 @@
+# Climate RiSc-Tool
+
+Climate RiSc Tool provides a free, publicly accessible, open-source platform for calculating, understanding, and indentiying climate impacts in power systems. While power system tools struggle to integrate large climate datasets, RiSc provides a fast and approximate method for identifying time periods and events of potential high risk of energy shortfalls across many Global Climate Models (GCMs), Shared Socioeconomic Pathways (SSPs) and with granular time and geospatial resoltions. High-Impact Low Frequency events can be used to facilitate climate-informed integrated system planning and guide resilience and adaptation strategies when properly integrating them in capacity expansion, resource adequacy and production cost models. RiSc also facilitates the understanding of structural system vulnerabilities of plausible future capacity buildout scenarios. It allows to assess how climate change will impact the power system under different horizons and scenarios. 
+
+Climate RiSc was developed as part of the EPRI Resource Adequacy Iniatiave and Climate READi. EPRI plans to support continuing updates and enhancements.
+
+## Getting Started
+
+These instruction will help you to deploy RiSc on a remote/enterpise server and run a demo case. See the Software Manual for more detailed information including screenshots.
+
+### Installation
+
+1. #### Clone/download this repository onto your server.
+
+	Get Git installed on your server.
+
+	Navigate to a folder on your server where you want to place the tool. 
+
+	Enter the following command from a terminal/prompt to 'clone' the repo and download it to your computer:
+        ```
+        git clone  https://github.com/epri-dev/Climate-RiSc-Tool.git
+        ```
+	When cloning a repo with 'git clone', if you do not specify a new directory as the last argument (as shown above),
+	it will be named `Climate-RiSc-Tool`. Alternatively, you can specify this and name it as you please. 
+        
+	Note: You can also clone your repo on your local computer and transfer it to the server through a mount or SFTP.
+
+2. #### Pull and load the docker image
+
+	Get docker installed on your server. 
+
+	When risctool docker image is provided as .tar file in this repo. run the command: `docker load -i risctool_docker.tar`. 
+
+	If the docker image is provided through a public registry run docker `docker pull risctool:1.0`. If no tag is specified, Docker will pull the latest tag by default.
+
+ 	Finally verity that the image is loaded `docker images`.
+
+	Note: You might you need must have privileges (e.g., sudo in bash) to run docker commands including pulling and running docker images. 
+
+3. #### Run the application
+
+	Make sure the .env file is available in the application directory. The.env file should contain two variables: DJANGO_KEY and DB_PASSWORD
+
+	Docker compose file `deft.yml` contain all the required configuration settings to run the containers. However, to run the tool from your local laptop (otherwise comment this line using #) you will have to modify and specify the mount path in the docker compose backend volumes  `<your_mount_path>/input:/code/share` to your own path. 
+
+	Finally, execute `./up.sh` to run the containers and `./down.sh` to stop and remove them. Note: you might need priviledges to perform this (e.g., `sudo`) 
+
+
+
+### Running Your First Case: DEMO
+
+This version comes with a demo datset to test the application. The public input data is specific of the Texas power system. 
+
+1. #### Configuration settings
+	Navigate to the `demo` folder. There are two files  `config.yml` and  `sysconfig.yml`. Both are ready to be used when running the tool locally (on your server)
+
+	If you want to run the tool remotely (on your local laptop) in the `config.yml` you need to change `sync_type` from 'local' to 'remote_share'. In addition, in `sysconfig.yml` you need to change the `hostname`. 
+
+	Note: Username and password should not be changed. 
+
+2. #### Running the tool locally (on your server)
+
+	You need to access the docker container running this command `docker exec -it deft-backend-1 bash`. 
+
+	Make sure that the input data has been correctly transferred to the `cd backend/demo/input` folder inside the container. Run `docker cp /home/Climate-RiSc-Tool/demo/input/ deft-backend-1:/code/backend/demo/ `
+
+	Finally, run the case executing `python demo.py` in the demo folder within the container.
+
+	Solution will appear in the container `backend/demo/output` folder. You can export this running `docker cp deft-backend-1:/code/backend/demo/output /home/Climate-RiSc-Tool/demo/`
+
+3. #### Running the tool remotely (on your local computer)
+
+	After changing configuration settings accordingly as decribed in step 1 and configure the mount (heavy files are transferred through this channel).
+
+	Then, run the python demo.py script on your local machine:  `python demo.py` 
+
+	Note: remember to set-up the mount path as volume in the deft.yml file (docker-compose file)
+
+## Deployment
+
+The application is designed to run on enterprise environments where deployment is secure and not publicly exposed to the internet. A docker-proxy set-up to encript http requests through https is recommended. 
+
+## Versioning
+
+This is version 0.1 (1/1/2025)
+
+## Authors
+
+* **Amir Cicak**
+* **Juan Carlos Martin**
+* **Eamonn Lannoye**
+
+## Contributing
+Pull requests are welcome. For major changes, please contact the team to discuss what you would like to change.
+
+Please make sure to update tests as appropriate.
+
+## License
+
+This project is licensed under the BSD (3-clause) License - see [LICENSE.txt](./LICENSE.txt).

db/Dockerfile

@@ -0,0 +1 @@
+FROM postgres:14.11

db/clear_data.sql

@@ -0,0 +1,34 @@
+delete from files_users;
+delete from weather_data_variables;
+delete from power_systems;
+delete from event_tags;
+delete from weather_data;
+delete from stations;
+delete from station_locations;
+delete from risk_models;
+delete from locations;
+delete from raw_weather_data;
+delete from files;
+delete from projects;
+ALTER SEQUENCE files_users_id_seq RESTART WITH 1;
+UPDATE files_users  SET id=nextval('files_users_id_seq');
+ALTER SEQUENCE weather_data_variables_id_seq RESTART WITH 1;
+UPDATE weather_data_variables  SET id=nextval('weather_data_variables_id_seq');
+ALTER SEQUENCE power_systems_id_seq RESTART WITH 1;
+UPDATE power_systems SET id=nextval('power_systems_id_seq');
+ALTER SEQUENCE stations_id_seq RESTART WITH 1;
+UPDATE stations SET id=nextval('stations_id_seq');
+ALTER SEQUENCE station_locations_id_seq RESTART WITH 1;
+UPDATE station_locations SET id=nextval('station_locations_id_seq');
+ALTER SEQUENCE event_tags_id_seq RESTART WITH 1;
+UPDATE event_tags SET id=nextval('event_tags_id_seq');
+ALTER SEQUENCE risk_models_id_seq RESTART WITH 1;
+UPDATE risk_models SET id=nextval('risk_models_id_seq');
+ALTER SEQUENCE weather_data_id_seq RESTART WITH 1;
+UPDATE weather_data SET id=nextval('weather_data_id_seq');
+ALTER SEQUENCE locations_id_seq RESTART WITH 1;
+UPDATE locations  SET id=nextval('locations_id_seq');
+ALTER SEQUENCE raw_weather_data_id_seq RESTART WITH 1;
+UPDATE raw_weather_data SET id=nextval('raw_weather_data_id_seq');
+ALTER SEQUENCE files_id_seq RESTART WITH 1;
+UPDATE files  SET id=nextval('files_id_seq');

db_scripts/drop_db.sh

@@ -0,0 +1,3 @@
+./empty_db.sh
+docker exec -it deft_db_1 bash -c 'psql -U deft deft -c "delete from auth_user;"'
+docker exec -it deft_db_1 bash -c 'psql -U deft deft -c "drop table auth_group_permissions;drop table auth_user_groups;drop table auth_group;drop table auth_user_user_permissions;drop table auth_permission;drop table django_admin_log;drop table django_content_type;drop table django_migrations;drop table django_session;drop table event_tags;drop table files_users;drop table locations;drop table power_systems;drop table raw_weather_data;drop table resources;drop table risk_models;drop table projects;drop table auth_user;drop table technologies;drop table weather_data_variables;drop table variables;drop table weather_data;drop table files;drop table stations;drop table station_locations;drop table statuses;drop table weather_models;drop table weather_scenarios;drop table weather_years;"'

db_scripts/empty_db.sh

@@ -0,0 +1 @@
+docker exec -it deft_db_1 bash -c 'psql -U deft deft -c "delete from weather_data_variables;delete from weather_data;delete from event_tags;delete from weather_models;delete from weather_years;delete from weather_scenarios;delete from power_systems;delete from stations;delete from station_locations;delete from files_users;delete from files;delete from statuses;delete from risk_models;delete from technologies;delete from projects;delete from variables;delete from resources;delete from raw_weather_data;"'

db_scripts/export_db.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+
+dump_name=dump-"$(date +"%d-%m-%Y").sql"
+docker exec -it deft_db_1 bash -c "pg_dump -U deft deft > dumps/$dump_name"
+
+if [[ -d $1 ]]; then
+  cp "docker_dev/db/dumps/$dump_name" "$1"
+fi

db_scripts/import_clean_db.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+docker exec -it deft_db_1 bash -c "psql -U deft deft < /dumps/clean-2024.sql"

db_scripts/import_db.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+if [[ ! -f $1 ]]; then
+    echo 'Error: Please provide the full path to the dump you want to import'
+    exit 1
+fi
+
+cp "$1" docker_dev/db/dumps
+docker exec -it deft_db_1 bash -c "psql -U deft deft < dumps/$(basename "$1")"

deft.yml

@@ -0,0 +1,54 @@
+version: "3.8"
+
+networks:
+  deft-network:
+
+volumes:
+  db-data:
+
+services:
+  db:
+    image: postgres:14.11
+    tty: true
+    container_name: deft_db_1
+    volumes:
+      - db-data:/var/lib/postgresql/data
+      - ./db/dumps:/dumps
+    networks:
+      - deft-network
+    env_file:
+      - ./.env
+    ports:
+      - "54322:5432"
+    environment:
+      - POSTGRES_DATABASE=deft
+      - POSTGRES_USER= deft
+      - POSTGRES_PASSWORD=${DB_PASSWORD}
+      - TZ=Etc/Greenwich
+      - PGTZ=Etc/Greenwich
+    healthcheck:
+      test: ["CMD", "pg_isready", "-U", "deft", "-d", "deft"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+
+  backend:
+    image: risctool.azurecr.io/risctool:2.0     
+    tty: true
+    networks:
+      - deft-network
+    env_file:
+      - ./.env
+    volumes:
+      - ./settings:/code/backend/backend
+      - ./logs/backend.log:/code/backend/deft_error.log
+      - ./nc_files/:/code/nc_files
+      - ./risk_models/:/code/risk_models
+      - ./power_systems/:/code/power_systems
+      - ./power_system_csv_files/:/code/power_system_csv_files
+      - ./load_profiles/:/code/load_profiles
+      - ./risk_models_xlsx_files/:/code/risk_models_xlsx_files
+      - ./risk_result_reports/:/code/risk_result_reports
+      #- /mnt/deft_imports/input:/code/share
+    ports:
+      - "8000:8000"

demo/config.yml

@@ -0,0 +1,62 @@
+# reset_all - "yes" or "no"
+# if set to yes it will
+#   - clean the database
+#   - split weather data
+#   - import weather data
+#   - import stations and locations
+# if set to "no" none of the above operations will be executed
+# if set to "sys" individual flags are taken from sysconfig
+reset_all: "yes"
+
+# project configuration
+delete_project_first: "yes" # "yes" or "no", delete project before creating it
+
+# project region will be used as prefix for procent names
+project_region: "TEXAS"
+
+# scenarios and models
+scenarios:
+  - "demo"
+models:
+  - "ERA5"
+
+# weather data, stations, locations, weather data splitting
+start_year: 2021
+end_year: 2022
+weather_stations_in_screening: 26
+
+target_year: 2021 
+power_systems_file: "input/power_systems/generatorData_ACEP_REF_2021.csv"
+risk_model_file: "input/risk_models/risk_models_texas.csv"
+aggregate_capacity_file: "output/aggregated_capacity_2021.csv"
+event_day_interval: 7
+event_day_overlap: 7
+top_events_percentage: 5
+threshold_MW: 0
+
+# Settings related to importing weather data
+
+# yes - sync files, no - do not sync regardless of sync type, use if files already synced
+sync_files_enabled: "yes"
+
+# files syncing has two types
+# local - assumes demo is run on the same machine as the backend
+# and the files will be copied by the demo itself from source
+# local directory to the target local directory (app/backend root)
+# remote_share - assumes demo is run from remote machine and
+# server backend will perform copy operations from remote share
+# to the remote backend directory target
+sync_type: "local" # local, remote_share
+
+# these are source files and you should only change file names
+# and not paths to them as paths are static for both local and
+# remote_share sync type. For remote share the folder structure must
+# be the same as the demo/inputs folder structure
+src_predictive: "Texas_Data_2020_2021_Projected_2021.nc" # must be in folder nc_files/
+src_historical: "Texas_Data_2020_2021_Historical.nc" # must be in folder nc_files/
+weather_station_file: "station_location_texas.csv" # must be in folder locations/
+load_folder_name: "2021_TEXAS" # must be in folder load_profiles/
+
+# additional settings related to sync_type: "local" are available in sysconfig.yml file
+
+# System settings including username and password for the api are available in sysconfig.yml file

demo/demo.py

@@ -0,0 +1,52 @@
+import modules.auth as auth
+from modules.project import process_projects
+from modules.helpers import (
+    load_config,
+    update_config,
+    load_json,
+)
+from modules.db import empty_db
+from modules.transform import (
+    create_aggregate_capacity_file,
+    convert_power_systems,
+    convert_risk_models
+)
+from modules.logging import configure_logger
+from modules.transform_nc import split_nc_files
+from modules.weather_files_import import import_weather_files
+from modules.file_sync import sync_files
+
+def main():
+    logger = configure_logger()
+    config = load_config("config.yml")
+    sysconfig = load_config("sysconfig.yml")
+    config = update_config(config, sysconfig)
+    auth.setup_auth(config)
+
+    empty_db(config)
+
+    df_final, df_weather_station = create_aggregate_capacity_file(config)
+    power_systems_json = convert_power_systems(df_final, df_weather_station)
+    risk_models_json = convert_risk_models(config)
+
+    logger.info("Loading power systems and risk models")
+    power_systems_input = load_json(power_systems_json)
+    risk_models = load_json(risk_models_json)
+    logger.info("Power systems and risk models loaded.")
+
+    sync_files(config)
+
+    historical_names, projected_names = split_nc_files(config)
+
+    import_weather_files(config, historical_names, projected_names)
+
+    logger.info(f"Exported NC files: {historical_names}, {projected_names}")
+    weather_files_list = [list(pair) for pair in zip(historical_names, projected_names)]
+    logger.info(f"Weather files list: {weather_files_list}")
+    power_systems = [power_systems_input[:] for _ in range(len(weather_files_list))]
+
+    process_projects(config, weather_files_list, power_systems, risk_models)
+
+
+if __name__ == "__main__":
+    main()