Merge pull request #1 from idaholab/dev

Dev

.gitignore

@@ -11,7 +11,11 @@ __pycache__
 spine/projects/*
 spine/Spine-Database-API
 spine/Spine-Toolbox
-venv/*
+venv
 .vscode
 local.json
-client_secrets.json
+client_secrets.json
+
+*.log
+
+.vscode

README.md

@@ -1,4 +1,9 @@
 # ResDEEDS
+## Intro
+Access to reliable, resilient power systems is important in 21st century now more than ever. Terrestrial weather events exacerbated by climate change and extreme weather conditions are happening with greater frequency and intensity. Cyberattacks are seen at an increasing frequency against the power grid, and the attacks are becoming more sophisticated and targeted towards electric energy systems. The Idaho National Laboratory (INL), with funding from the Department of Energy (DOE) Wind Energy Technologies Office (WETO), has developed a [Resilinece Framework]([http://www.spine-model.org](https://resilience.inl.gov/wp-content/uploads/2021/07/21-50152_RF_EEDS_R4.pdf) for electric energy delivery systems (EEDS). The framework provides detailed steps for evaluating resiliency in the planning, operational, and future stages, and encompasses five core functions of resilience. It allows users to evaluate the resilience of distributed wind, taking into consideration the resilience of the wind systems themselves, as well as the effect they have on the resiliency of any systems they are connected to. This application follows the framework to allow stakeholders to evaluate their current position, create resiliency goals, compare different configuration and operation options, and decide which metrics are most appropriate for their system.
+
+## Overview
+This tool is implemented as a Flask (Python) web application. It is currently primarily intended to be run/hosted locally by the end user. The app uses the [Spine Toolbox and SpineOpt](http://www.spine-model.org/) modeling tools to calculate values for resilience metrics. The input to Spine is an Excel spreadsheet of user system data and hazard modeling data.
 
 ## Installation
 ### Linux
@@ -12,10 +17,6 @@
 
     `sudo install/install.sh`
 
-    TODO: should the install script create and use a venv?
-
-    TODO: install script for `yum`?
-
 ### Windows
 #### Prerequisites
 * Git (download for Windows [here](https://git-scm.com/download/win))
@@ -54,22 +55,35 @@
 
     `~/AppData/Local/Programs/Julia-1.9.0/bin/julia.exe -e 'using Pkg; Pkg.add(["XLSX", "DataFrames", "Distributions", "CSV", "Revise", "Cbc", "Clp"])'`
 
+1. Run Spine Toolbox to initialize configuration. You can close it after it opens.
+
+    `python -m toolbox &`
+
 1. Configure Spine Toolbox's Julia paths.
 
     `python venv/src/spinetoolbox/bin/configure_julia.py "C:/Users/$(whoami)/AppData/Local/Programs/Julia-1.9.0/bin/julia.exe" ""`
 
 1. Create `config/local.json` and override default config settings as needed. At a minimum, override `"app_secret_key"` with a random string.
 
 ## Running the app
-1. Navigate into the project folder on the command line: Find "resilience_calculator" folder -> Right-click on folder location in address bar and copy address -> Input 'cd ' and paste folder location address into CMD terminal or PowerShell
+### Windows
+1. Navigate into the project folder using GitBash or PowerShell (we don't recommend running the app in an IDE-integrated terminal).
+
+    `cd $PROJECT_ROOT`
+
+    where `$PROJECT_ROOT` is your `resilience_calculator` folder. On at least some version of Windows, you can also navigate to the folder in File Explorer, right-click on a blank space, and select `Open in Terminal`.
 
 1. Activate the Python virtual environment.
-    'venv\Scripts\activate'
+
+    `. venv\Scripts\activate`
 
 1. Run `python src/app.py`.
 
+### Linux
+1. Run `install/run.sh`.
+
 ## Accessing the app
-In a web browser, visit the URL printed on the console during startup (e.g. http://127.0.0.1:5000).
+In a web browser, visit `http://localhost:5000`, or the URL printed to the console during startup, if different.
 
 ## Configuration
 ### Overview
@@ -78,7 +92,8 @@ Configuration is done via JSON in `config/local.json` (create this file if it do
 ### Options
 * `app_secret_key` - identifies your Flask app. Should remain secret.
 * `debug_mode` - set to `true` to enable Flask debugging.
-* `use_okta` - set to `true` to enable Okta authentication. Configure your Okta instance in a `config/client_secrets.json`. If this is set to `false`, the app runs in an unauthenticated mode, where all users can see all projects.
+* `verbose_logging` - set to `true` to enable debug log messages.
+* `use_okta` - set to `true` to enable Okta authentication. Configure your Okta instance in a `config/client_secrets.json`. If this is set to `false`, the app runs in an unauthenticated mode, where all users can see all projects. For a single user running the app locally, we recommend setting this to `false`.
 * `okta` - partial Okta configuration. The remainder is found in a `config/client_secrets.json` file like [this one](https://github.com/okta/samples-python-flask/blob/master/okta-hosted-login/client_secrets.json.dist).
     * `orgUrl` - the Okta organization URL, e.g. https://example.okta.com.
     * `token` - your API token.
@@ -91,7 +106,7 @@ Configuration is done via JSON in `config/local.json` (create this file if it do
     * `drop_and_recreate` - if `true`, drops and recreates the database on Flask app startup. Intended for development/debugging only.
 
 ## MySQL Support
-To use MySQL, install the `mysqlclient` package with pip. TODO: document configuration.
+SQLite is used by default. MySQL is experimentally supported. To use MySQL, install the `mysqlclient` package with pip.
 
 ### Setting up MySQL on Windows
 If you are using the mysql dialect option for the database, you will need a MySQL or MariaDB server to connect to. These instructions are for if you want to run it locally on Windows.

config/config.json

@@ -1,6 +1,7 @@
 {
     "app_secret_key": "",
-    "debug_mode": true,
+    "debug_mode": false,
+    "verbose_logging": true,
     "use_okta": false,
     "okta": {
         "orgUrl": "",

src/app.py

@@ -1,16 +1,16 @@
 # Copyright 2023, Battelle Energy Alliance, LLC
 from types import SimpleNamespace
-import werkzeug
-from flask import Flask, render_template, g, send_file, url_for, request, redirect, session
-import asyncio
-from config import config
-from backend import DBSession
-from backend.project import *
 import logging
 import os
 import platform
+import asyncio
+import werkzeug
+from flask import Flask, render_template, g, send_file, url_for, request, redirect, session
 
-logging.basicConfig(filename="log.log", level=logging.DEBUG if config['debug_mode'] else logging.INFO)
+from config import config
+from backend import DBSession
+from backend.project import Project, HazardImpact, HazardLikelihood, GoalComparison
+from backend.spine.db import SpineDBSession
 
 if platform.system()=='Windows':
     asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
@@ -43,29 +43,29 @@ def allowed_file(filename):
 
 @app.before_request
 def before_request():
-    g.__setattr__('db_session', DBSession())
+    g.db_session = DBSession()
     if PROJECT_ID_KEY in session:
-        g.__setattr__('project', Project.get_by_id(g.db_session, session[PROJECT_ID_KEY]))
+        g.project = Project.get_by_id(g.db_session, session[PROJECT_ID_KEY])
     else:
-        g.__setattr__('project', None)
+        g.project = None
 
-    g.__setattr__('spine_db_session', SpineDBSession())
+    g.spine_db_session = SpineDBSession()
 
     if USE_OKTA and oidc.user_loggedin:
         async def _f():
             user, _, _ = await okta_client.get_user(oidc.user_getfield("sub"))
             # https://developer.okta.com/docs/reference/api/users/#example
-            g.__setattr__('user', user)
+            g.user = user
         asyncio.run(_f())
     elif USE_OKTA:
-        logging.info('User is not logged in!')
-        g.__setattr__('user', None)
+        logging.info('User is not logged in.')
+        g.user = None
     else:
-        logging.warning('Operating in unauthenticated mode.')
+        logging.info('Operating in unauthenticated mode.')
         user = SimpleNamespace()
         user.use_okta = False
         user.id = 'default'
-        g.__setattr__('user', user)
+        g.user = user
 
 @app.after_request
 def after_request(response):
@@ -80,18 +80,18 @@ def index():
         projects = Project.get_all_for_user(g.db_session, g.user.id)
     else:
         logging.debug("Anonymous user.")
-    logging.info(request.form)
+    logging.debug(request.form)
     if request.method == "POST":
         if len(projects) == 0 or request.form["projNameValAdd"] != "":
             if len(projects) > 0:
                 sys_name = request.form["projNameValAdd"]
             else:
                 sys_name = request.form["projNameVal"]
-            project = Project.build(g.db_session, g.spine_db_session, sys_name, g.user.id)
+            project = Project.build(g.db_session, sys_name, g.user.id)
             session[PROJECT_ID_KEY] = project.id
         else:
             try:
-                # grabs id of whatever project's "edit" was clicked
+                # Grabs id of whatever project's "edit" was clicked
                 session[PROJECT_ID_KEY] = request.form["edit"]
             except werkzeug.exceptions.BadRequestKeyError:
                 Project.get_by_id(g.db_session, request.form["delete"]).delete(g.db_session)
@@ -108,15 +108,16 @@ def qualities():
             file = request.files['system_spreadsheet']
             if allowed_file(file.filename):
                 try:
-                    result = g.project.import_system(g.db_session, g.spine_db_session, file, is_baseline=True)
+                    g.project.import_system(g.db_session, g.spine_db_session, file, is_baseline=True)
+                    result = 'System imported.'
                 except Exception as exception: #pylint: disable=W0718
                     logging.exception(exception)
                     return render_template("qualities.html", result=[], errors=["ERROR: unable to import system!"]), 500
 
             else:
-                logging.info(f'{file.filename} not allowed.')
+                logging.error('Tried to upload a file %s but this file type is not allowed.', file.filename)
         else:
-            logging.info('Did not find system_spreadsheet in posted files.')
+            logging.error('Did not find system_spreadsheet in posted files.')
     return render_template("qualities.html", result=result)
 
 @app.route('/download-template', methods=["GET"])
@@ -127,7 +128,6 @@ def download_template():
 @app.route('/initial-system', methods=["GET"])
 def initial_system():
     sys, rels = g.project.get_system(g.spine_db_session, baseline=True)
-    logging.debug(f'Sys: {sys}')
     return render_template("initial-system.html", system=sys, relationships=rels)
 
 @app.route('/hazards', methods=["GET", "POST"])
@@ -155,14 +155,13 @@ def goals():
         goal_comparisons = request.form.getlist('goalComparison')
         goal_target_values = request.form.getlist('goalTargetValue')
 
-        print(goal_names, goal_comparisons, goal_target_values)
-
         for n, c, tv in zip(goal_names, goal_comparisons, goal_target_values):
             hazard_name, goal_name = n.split('.')
             try:
                 g.project.update_goal(g.db_session, hazard_name, goal_name, c, float(tv))
-            except ValueError:
-                print(f'Could not convert {tv} to float for goal {n}.')
+            except ValueError as err:
+                logging.exception('Could not convert %s to float for goal %s.', str(tv), n)
+                logging.exception(err)
 
         return redirect("/spineopt")
     return render_template("goals.html", base_hazard=g.project.get_base_hazard(), hazards=sorted(g.project.get_hazards(), reverse=True, key=lambda x: x.get_risk_level().value), goal_comparisons=GoalComparison.get_all(), colors=hazard_risk_colors)
@@ -174,16 +173,16 @@ def spineopt():
     if request.method == "POST":
         # Update system objects
         if 'system_spreadsheet' in request.files and request.files['system_spreadsheet'].filename:
-            print('Importing spreadsheet, ignoring manual entries.')
+            logging.warning('Importing spreadsheet, ignoring manual entries.')
             file = request.files['system_spreadsheet']
             if allowed_file(file.filename):
                 g.project.import_system(g.db_session, g.spine_db_session, file, is_baseline=False)
                 sys, rels = g.project.get_system(g.spine_db_session)
                 logging.debug(sys)
             else:
-                print(f'{file.filename} not allowed.')
+                logging.error('Tried to upload a file %s but this file type is not allowed.', file.filename)
         else:
-            print('Doing GUI parameter updates.')
+            logging.info('Doing GUI parameter updates.')
             for k, v in request.form.items():
                 words = k.split('.')
                 if words[0] == 'obj':
@@ -201,14 +200,19 @@ def spineopt():
 def run_spineopt():
     spine_output = g.project.run_spineopt()
     session["spine_output"] = spine_output
-    print(type(session["spine_output"]))
+    logging.info(type(session["spine_output"]))
+    g.project.load_results(g.spine_db_session)
     return redirect("/results")
 
 @app.route('/results', methods=["GET"])
 def results():
     g.project.load_results(g.spine_db_session, baseline=False)
     g.project.load_results(g.spine_db_session, baseline=True)
-    return render_template("results.html", base_hazard=g.project.get_base_hazard(), hazards=sorted(g.project.get_hazards(), reverse=True, key=lambda x: x.get_risk_level().value), goal_comparisons=GoalComparison.get_all(), colors=hazard_risk_colors)
+    return render_template("results.html", 
+                           base_hazard=g.project.get_base_hazard(), 
+                           hazards=sorted(g.project.get_hazards(), reverse=True, key=lambda x: x.get_risk_level().value), 
+                           goal_comparisons=GoalComparison.get_all(), 
+                           colors=hazard_risk_colors)
 
 @app.route('/changes', methods=["GET"])
 def changes():