[GITFLOW]merging 'release-1.81.0' into 'master'

README.md

@@ -11,7 +11,6 @@
   <a href="https://github.com/hzi-braunschweig/SORMAS-Project/releases/latest"><img alt="Latest Release" src="https://img.shields.io/github/v/release/hzi-braunschweig/SORMAS-Project"/></a>
   <a href="https://github.com/hzi-braunschweig/SORMAS-Project/actions?query=workflow%3A%22Java+CI+with+Maven%22"><img alt="Development Build Status" src="https://github.com/hzi-braunschweig/SORMAS-Project/workflows/Java%20CI%20with%20Maven/badge.svg?branch=development"/></a>
   <a href="https://gitter.im/SORMAS-Project"><img alt="Gitter" src="https://badges.gitter.im/SORMAS-Project/dev-support.svg"/></a>
-  <a href="https://twitter.com/SORMASDev"><img alt="Twitter" src="https://img.shields.io/twitter/follow/SORMASDev?label=%40SORMASDev&style=social"/></a>
 </p>
 <br/>
 
@@ -22,12 +21,12 @@
 ## FAQ (Frequently Asked Questions)
 
 ### How Does it Work?
-You can give SORMAS a try on our demo server at <https://demoversion.sormas-oegd.de>!
+You can give SORMAS a try on our demo server at <https://demo.sormas.org>!
 
 ### How Can I Get Involved?
 Read through our [*Contributing Readme*](docs/CONTRIBUTING.md) and contact us at [email protected] or join our [developer chat on Gitter](https://gitter.im/SORMAS-Project) to learn how you can help to drive the development of SORMAS forward and to get development support from our core developers. SORMAS is a community-driven project, and we'd love to have you on board!
 If you want to contribute to the code, please strictly adhere to the [*Development Environment*](docs/DEVELOPMENT_ENVIRONMENT.md) guide to ensure that everything is set up correctly.
-Please also make sure that you've read the [*Development Contributing Guidelines*](docs/CONTRIBUTING.md#development-contributing-guidelines) before you start to develop, and either follow or regularly check our Twitter account <a href="https://twitter.com/SORMASDev" target="_blank">@SORMASDev</a> to stay up to date with our schedule, new releases, guideline changes and other announcements.
+Please also make sure that you've read the [*Development Contributing Guidelines*](docs/CONTRIBUTING.md#development-contributing-guidelines) before you start to develop.
 
 ### How Can I Report a Bug or Request a Feature?
 If you want to report a **security issue**, please read and follow our [*Security Policies*](docs/SECURITY.md). For bugs without security implications, change and feature requests, please [create a new issue](https://github.com/hzi-braunschweig/SORMAS-Project/issues/new/choose) and

docs/3RD_PARTY_ACK.md

@@ -1,6 +1,9 @@
 # 3rd Party License Acknowledgement
 
-Country flags in `sormas-ui/src/main/webapp/VAADIN/themes/sormas/img/flag-icons/` are licensed under MIT:
+Nearform **temporal tables** postgres function is licensed under MIT:
+sormas-backend/src/main/resources/sql/temporal_tables/LICENSE
+
+**Country flags** in `sormas-ui/src/main/webapp/VAADIN/themes/sormas/img/flag-icons/` are licensed under MIT:
 ```text
 Copyright (c) 2017 Go Squared Ltd. http://www.gosquared.com/
 

docs/SERVER_UPDATE.md

@@ -3,11 +3,6 @@
 SORMAS releases starting from 1.21.0 contain a script that automatically updates and deploys the server. If you are using an older version and therefore need to do a manual server update, please download the 1.21.0 release files and use the commands specified in the server-update.sh script.
 
 ## Preparations
-Note: At some versions it is mandatory to switch to a new Payara Server. If your version bump does apply to the listing below, please proceed with [Payara migration](SERVER_UPDATE.md#how-to-migrate-to-new-payara-server).
-* Switching from <=v1.66.4 to v1.67.0 or newer
-
-Note: You can skip this step if you've just set up your SORMAS server and have already downloaded the latest release.
-
 * Get the latest release files (deploy.zip) from <https://github.com/hzi-braunschweig/SORMAS-Project/releases/latest>
 * Unzip the archive and copy/upload its contents to **/root/deploy/sormas/$(date +%F)**
     ```bash
@@ -18,6 +13,46 @@ Note: You can skip this step if you've just set up your SORMAS server and have a
     mv deploy/ $(date +%F)
     rm sormas_${SORMAS_VERSION}.zip
     ```
+
+## Breaking Updates
+The following is a list of version that have breaking changes in the update. **You only have to consider this if you are coming from an earlier version.** For fresh installs, this is not relevant.
+
+### 1.67.0
+Coming from 1.66.4 or earlier, you have to update the payara server, as explained in the [Payara migration](SERVER_UPDATE.md#how-to-migrate-to-new-payara-server).
+
+### 1.73.0
+Deploying this release **will clear the userrolesconfig and userrole_userrights tables** and overwrite them with the default user role configurations of SORMAS. **If you added entries to these tables in order to customize the user roles on your system**, please run the following queries before deploying this release in order to prevent data loss:
+
+```SQL
+-- Retrieve all customized roles
+SELECT * FROM userrolesconfig;
+ 
+-- Overridden rights for roles
+SELECT c.userrole, ur.userright FROM userroles_userrights ur LEFT JOIN userrolesconfig c (ON c.id = ur.userrole_id);
+```
+
+After deploying the new version, the information retrieved from these queries can be used to alter the new user role configurations accordingly.
+
+### 1.81.0
+The [temporal tables extension is replaced](https://github.com/hzi-braunschweig/SORMAS-Project/issues/10260) during the deployment of the backend. As a preparation the following SQL **needs to be executed on the SORMAS database using the postgres user.**
+
+```SQL
+-- versioning function will be replaced during server backend startup
+ALTER FUNCTION public.versioning() OWNER TO sormas_user;
+```
+
+**After the successful deployment**, you can run the following SQL to get rid of the no longer used extension:
+
+```SQL
+-- needed because `ALTER FUNCTION ... NO DEPENDS ON EXTENSION` is not possible in Postgres 10 and below
+DELETE FROM pg_depend
+WHERE deptype = 'e'
+  AND refobjid = (SELECT oid FROM pg_extension WHERE extname = 'temporal_tables')
+  and objid = (SELECT oid FROM pg_proc WHERE proname = 'versioning');
+
+DROP EXTENSION IF EXISTS temporal_tables;
+```
+
 ## Automatic Server Update
 * Navigate to the  folder containing the unzipped deploy files:
   ``cd /root/deploy/sormas/$(date +%F)``

docs/TROUBLESHOOTING.md

@@ -1,7 +1,23 @@
+
 # Troubleshooting
 
 Please consult this collection of solutions to common problems if you have any issues before issuing a support request or asking developers for help. Also note that this resource has only been added recently and will be extended in the future. If you have encountered (and fixed) any issue that you think would be worth adding to this list, please don't hesitate to let us know!
 
+- [Android Application FAQ](#android-application-faq)
+- [Identify Performance Problems](#identify-performance-problems)
+  - [Log Slow Database Queries](#log-slow-database-queries)
+  - [Create Analysis of Slow Database Query (SORMAS-Docker)](#create-analysis-of-slow-database-query-sormas-docker)
+  - [Log Slow Java Code](#log-slow-java-code)
+- [Analyze Performance Problems](#analyze-performance-problems)
+  - [Analyze a Slow Query](#analyze-a-slow-query)
+  - [Analyze Java Code Performance Logs](#analyze-java-code-performance-logs)
+- [IDE Troubleshooting: Android Studio](#ide-troubleshooting-android-studio)
+- [IDE Troubleshooting: eclipse](#ide-troubleshooting-eclipse)
+  - [Deployment Problems](#deployment-problems)
+  - [News Feeds Polling](#news-feeds-polling)
+- [Redeployment problems](#redeployment-problems)
+- [Malware detection triggers](#malware-detection-triggers)
+
 ## Android Application FAQ
 
 **Q:** I don't see a logout option anywhere in the mobile app. How can I change my user?  
@@ -16,55 +32,30 @@ Please consult this collection of solutions to common problems if you have any i
 4. Open it and search for de.symeda.sormas to identify the process id. E.g. `de.symeda.sormas.app/de.symeda.sormas.app.login.LoginActivity$_11109#0` -> 11109 is the id
 5. Search for all occurences of the process id to filter the file down to lines that contain the actual log of sormas
 
-## Debugging Performance Problems
-
-Performance logging can be used to find out which part of the code or system might be responsible for long-running functions in the application. This helps the developers to identify the source of the problems quicker and find out whether there are several problems at once or performance problems that manifest in Java execution time instead of slow SQL queries.
-
-**Caution: Do not expose any private data!** Whenever you debug problems on an instance with productive data, please make sure that the logged information does not contain any personal data like real person names, birth dates, etc. to the public. Never provide such data anywhere on GitHub or any other online tool!
-
-### Switch on Performance Logging in SORMAS
-
-1. Open the logback file located in your domain (default path: `/opt/domains/sormas/config/logback.xml`) and change the log level of `PerformanceLoggingInterceptor` to `DEBUG` or `TRACE`. The config change will be recognized during runtime within 30s. After that you will see detailed log entries in the SORMAS log.
-
-2. Set the log level back to its default once the logging has been done since it can reduce the overall performance of SORMAS.
-
-### Analyze Performance Logs
+## Identify Performance Problems
 
-Performance logs can be analyzed in detail using the `PerformanceLogAnalysisGenerator`. To use this tool, set the `PerformanceLoggingInterceptor`'s log level
-to `TRACE` as described above and reproduce the scenario you want to investigate on the server instance.
+There are two main sources of bad performance in the application: 
 
-After this, process the debug log file (default path: `/opt/domains/sormas/logs/application.debug`) using the `PerformanceLogAnalysisGenerator`.
-The log file's path is specified as the program argument when calling `PerformanceLogAnalysisGenerator`'s `main` method. Processing the log file will
-generate three files (`<logfileName>.csv`, `<logfileName>.txt`, `<logfileName>.html`) to further investigate method runtimes.
+1. Slow database queries
+2. Slow Java code
 
-`<logfileName>.html` provides a navigable overview of methods along with runtime statistics (total, min, max and average time) and calls to sub methods.
+### Log Slow Database Queries
 
-Sometimes it is convenient to analyze a number of different scenarios in a row. To do so, produce snippets of the `application.debug` log using `tail` for each
-of the scenarios to be investigated:
+Possibly the most generally useful log setting for troubleshooting performance, especially on a production server.
 
-1. start `tail -f <logfileName> > <snippetDirectory>/<snippet.debug>`
-
-2. replay the steps to be analyzed
-
-3. stop `tail -f`
-
-The `PerformanceLogAnalysisGenerator` can now batch process all of the snippets by pointing to the directory instead of a log file.
-Calling `PerformanceLogAnalysisGenerator.main` with argument `<snippetDirectory>` generates the analysis files (`.csv`, `.txt`, `.html`)
-for each file `*.debug` in this directory. The generated files will be placed in `<snippetDirectory>`, too.
-
-### Log Slow SQL Queries in PostgreSQL
+**SORMAS-Docker** already logs slow SQL queries by default. You can view the log output on its host VM with `docker logs sormas-docker_postgres_1`.
 
 You can enable the logging of slow SQL queries in your PostgreSQL server in `postgresql.conf`:
 
 1. Change the value of `log_min_duration_statement` to a value that fits your need (e.g. 10000).
-
 2. Restart the PostgreSQL service or reload the config.
+3. Monitor the log file.
 
-### Run analysis of a SQL Query (SORMAS-Docker)
+### Create Analysis of Slow Database Query (SORMAS-Docker)
 
 You can provide an analysis of a slow running query to help the developers to see where the query is getting slow and how to fix it.
 
-1. **SORMAS-Docker** already logs slow SQL queries by default. You can view the log output on its host VM with `docker logs sormas-docker_postgres_1`.
+1. Extract the slow SQL statement from slow query log.
 
 2. Copy the SQL statement, replace all parameters (`$x`) with the values (see the following log statement) and place the SQL query on the system (outside Docker container on host):
 ```bash
@@ -89,7 +80,78 @@ psql -XqAt -d sormas -f explain.sql > analyze.json
 
 4. Copy the output to your home dir on the VM (not inside the Docker container) to be able to copy it from the VM to your local system: `mv analyze.json /home/user.name/`
 
-5. Create a visual report at <https://explain.dalibo.com/> in order to share the analysis.
+5. Create a visual report at <https://explain.dalibo.com/> in order to share the analysis. Make sure the raw query and timings are included.
+
+How to analyze this is explained below.
+
+### Log Slow Java Code
+
+If a specific view or action is facing bad performance it may be helpful to log the execution time of the responsible Java code.
+
+1. Open the logback file located in your domain (default path: `/opt/domains/sormas/config/logback.xml`) and change the log level of `PerformanceLoggingInterceptor` to `DEBUG` or `TRACE`. The config change will be recognized during runtime within 30s. After that you will see detailed log entries in the SORMAS log.
+2. Reproduce the scenario you want to investigate on the server instance.
+3. Set the log level back to its default once the logging has been done since it can reduce the overall performance of SORMAS.
+
+**Caution: Do not expose any private data!** Whenever you debug problems on an instance with productive data, please make sure that the logged information does not contain any personal data like real person names, birth dates, etc. to the public. Never provide such data anywhere on GitHub or any other online tool!
+
+How to analyze the log is explained below.
+
+## Analyze Performance Problems
+
+### Analyze a Slow Query
+
+To get your hands on a specific query executed by a view or action, you can do the following:
+
+1. Get an [analysis from a production system](#create-analysis-of-slow-database-query-sormas-docker)
+2. Create your own:
+   1. Identify or create a unit test that calls the related backend method
+   2. Set the 'hibernate.show_sql' config in the persistence.xml of the test resources to true
+   3. Run the unit test and extract the query from the log
+   4. Replace any parameters in the query
+   5. Execute the query on your local database or a test system pre-faced by 'EXPLAIN (ANALYZE, COSTS, VERBOSE, BUFFERS, FORMAT JSON)'
+   6. Create a visual report at <https://explain.dalibo.com/> in order to share the analysis. Make sure the raw query and timings are included.
+
+Important points when **interpreting the visual report**:
+
+* Look for the **red highlights** in the report, especially for nodes that have a long runtime
+* Look for nodes that are processing a **lot of rows** (100+ millions). This is often the case for joins
+* When you are dealing with a lof of data, most often you only need to output a subset 
+
+**How to improve the query** is most often a process of trial and error. Here are some things to consider:
+
+* Adding indices so postgres does not have to go through all data. You can use a btree index to include multiple columns and sorting
+* Getting rid of unnecessary joins. Example: Joining a region of a case to compare it to the region of the user, instead of directly doing this on the region_id field of the case
+* Using limit to reduce the output (e.g. the first 100). Make sure there is no sorting done close to the end of the query graph. This is often the case when 'DISTINCT' is used.
+  Example: https://github.com/hzi-braunschweig/SORMAS-Project/issues/9054#issuecomment-1420849461
+* Splitting the query into separate queries when 'DISTINCT' has to be used / using distinct on a sub-query
+* Using sub-queries to influence the query planner. 
+  Example: https://github.com/hzi-braunschweig/SORMAS-Project/issues/11465#issuecomment-1425789509
+
+**How to save time** when optimizing the query:
+
+* Make sure you have easy access to a database that allows you to reproduce the bad performance of the query and to **manipulate the query and re-run it directly on the database**.
+* Use the explain feature of pgAdmin to quickly output the query graph for debugging purposes
+* Make sure you have a unit test (see above) that allows you to create the SQL query from a criteria query without having to re-deploy your server
+
+### Analyze Java Code Performance Logs
+
+After [Logging Slow Java Code](#log-slow-java-code) the debug log file (default path: `/opt/domains/sormas/logs/application.debug`) can be analyzed in detail using the `PerformanceLogAnalysisGenerator`.
+
+The log file's path is specified as the program argument when calling `PerformanceLogAnalysisGenerator`'s `main` method. Processing the log file will
+generate three files (`<logfileName>.csv`, `<logfileName>.txt`, `<logfileName>.html`) to further investigate method runtimes.
+
+`<logfileName>.html` provides a navigable overview of methods along with runtime statistics (total, min, max and average time) and calls to sub methods.
+
+Sometimes it is convenient to analyze a number of different scenarios in a row. To do so, produce snippets of the `application.debug` log using `tail` for each
+of the scenarios to be investigated:
+
+1. start `tail -f <logfileName> > <snippetDirectory>/<snippet.debug>`
+2. replay the steps to be analyzed
+3. stop `tail -f`
+
+The `PerformanceLogAnalysisGenerator` can now batch process all of the snippets by pointing to the directory instead of a log file.
+Calling `PerformanceLogAnalysisGenerator.main` with argument `<snippetDirectory>` generates the analysis files (`.csv`, `.txt`, `.html`)
+for each file `*.debug` in this directory. The generated files will be placed in `<snippetDirectory>`, too.
 
 ## IDE Troubleshooting: Android Studio
 

sormas-api/pom.xml

@@ -2,7 +2,7 @@
 	<parent>
 		<groupId>de.symeda.sormas</groupId>
 		<artifactId>sormas-base</artifactId>
-		<version>1.80.0</version>
+		<version>1.81.0</version>
 		<relativePath>../sormas-base</relativePath>
 	</parent>
 	<modelVersion>4.0.0</modelVersion>

sormas-api/src/main/java/de/symeda/sormas/api/FacadeProvider.java

@@ -38,6 +38,7 @@
 import de.symeda.sormas.api.contact.ContactFacade;
 import de.symeda.sormas.api.customizableenum.CustomizableEnumFacade;
 import de.symeda.sormas.api.dashboard.DashboardFacade;
+import de.symeda.sormas.api.dashboard.sample.SampleDashboardFacade;
 import de.symeda.sormas.api.deletionconfiguration.DeletionConfigurationFacade;
 import de.symeda.sormas.api.disease.DiseaseConfigurationFacade;
 import de.symeda.sormas.api.docgeneneration.DocumentTemplateFacade;
@@ -307,6 +308,10 @@ public static DashboardFacade getDashboardFacade() {
 		return get().lookupEjbRemote(DashboardFacade.class);
 	}
 
+	public static SampleDashboardFacade getSampleDashboardFacade() {
+		return get().lookupEjbRemote(SampleDashboardFacade.class);
+	}
+
 	public static DiseaseConfigurationFacade getDiseaseConfigurationFacade() {
 		return get().lookupEjbRemote(DiseaseConfigurationFacade.class);
 	}

sormas-api/src/main/java/de/symeda/sormas/api/audit/Constants.java

@@ -15,6 +15,7 @@ public class Constants {
 				"count",
 				"get",
 				"is",
+				"are",
 				"has",
 				"does",
 				"validate",