Merge remote-tracking branch 'origin/development' into qu-auto/SORQA-482

docs/CONTRIBUTING.md

@@ -85,6 +85,9 @@ If you're interested in participating in the development of SORMAS, please follo
 If you have problems setting up your development environment or need assistance in choosing the first issue to work on, please get in touch with us by joining our [Gitter channel](https://gitter.im/SORMAS-Project) or contacting us at [email protected].
 Additionally, our [Wiki](https://github.com/hzi-braunschweig/SORMAS-Project/wiki) contains some specific development guides that cover common issues like adding new fields to an entity that we suggest to check out before you start implementing something related to those topics.
 
+* [Technical User Guides](https://github.com/hzi-braunschweig/SORMAS-Project/wiki#technical-user-guides)
+* [Development Guides](https://github.com/hzi-braunschweig/SORMAS-Project/wiki#development-guides)
+
 ### Development Contributing Guidelines
 
 In addition to the guidelines covered in the Development Environment Setup Instructions, please ensure to adhere to the following principles and procedures while developing code for SORMAS:

docs/SERVER_CUSTOMIZATION.md

@@ -48,7 +48,10 @@ Changing these entries overrides that default value. Unlike with features, disea
 [This Wiki page](https://github.com/hzi-braunschweig/SORMAS-Project/wiki/Disease-Configuration-Options) contains a list and explanation of all currently configurable disease properties.
 
 ## Deletion Configuration
-SORMAS can be set up to automatically delete entities after a specific time period. There are seven core entities for which automatic deletion can be enabled and configured: *Case, Contact, Event, Event Participant, Immunization, Travel Entry, and Campaign.* This configuration is currently only possible directly in the database via the `deleteconfiguration` table, which already contains rows for each of these entities. The table consists of the following columns:
+SORMAS can be set up to automatically delete entities after a specific time period.
+There are seven core entities for which automatic deletion can be enabled and configured: *Case, Contact, Event, Event Participant, Immunization, Travel Entry, and Campaign.*
+This configuration is currently only possible directly in the database via the `deleteconfiguration` table, which already contains rows for each of these entities.
+The table consists of the following columns:
 
 * **`entityType`:** The name of the entity that supports automatic deletion.
 * **`deletionReference`:** The reference date for the calculation of the date on which deletion takes place (see below).
@@ -57,7 +60,8 @@ SORMAS can be set up to automatically delete entities after a specific time peri
 Both `deletionReference` and `deletionPeriod` need to be filled in order for the automatic deletion to take place. Entities for which at least one of these fields is left empty will not be automatically deleted. Deletion is executed via a nightly cron job and might therefore not happen immediately when the deletion date has been reached.
 
 ### Deletion Reference
-The `deletionReference` field has four possible values which define the date that is used to calculate whether an entity needs to be deleted (i.e., when the date calculated by subtracting the deletion period from the current date is before the deletion reference date, the entity is deleted). A `MANUAL_DELETION` entry can exist in parallel to one of the other entries, and if both entries are configured, deletion is executed as soon as the threshold of one of these entries is met.
+The `deletionReference` field has four possible values which define the date that is used to calculate whether an entity needs to be deleted (i.e., when the date calculated by subtracting the deletion period from the current date is before the deletion reference date, the entity is deleted).
+A `MANUAL_DELETION` entry can exist in parallel to one of the other entries, and if both entries are configured, deletion is executed as soon as the threshold of one of these entries is met.
 
 * **`CREATION`**: The creation date of the entity will be used.
 * **`END`**: The latest change date of the entity itself and any of its depending entities will be used. E.g. for cases, this includes but is not limited to its epi data, symptoms, or hospitalization.
@@ -93,13 +97,13 @@ To cover the typical use cases, SORMAS defines a set of default user roles that
 
 Most user rights define an action related to an entity type, e.g. the user right *CASE_EDIT* allows users to edit case data.
 The following automatically generated documents list and describe **all user rights and the default user roles**:
-* https://github.com/hzi-braunschweig/SORMAS-Project/blob/development/sormas-api/src/main/resources/doc/SORMAS_User_Roles.xlsx
+* [https://github.com/hzi-braunschweig/SORMAS-Project/blob/development/sormas-api/src/main/resources/doc/SORMAS_User_Roles.xlsx](https://github.com/hzi-braunschweig/SORMAS-Project/blob/development/sormas-api/src/main/resources/doc/SORMAS_User_Roles.xlsx)
 
-User roles are fully configurable, allowing admins to create new user roles and edit existing ones, to customize the instance to the given needs and to make sure data protection requirements are fulfilled. 
+User roles are fully configurable, allowing admins to create new user roles and edit existing ones, to customize the instance to the given needs and to make sure data protection requirements are fulfilled.
 
 ### Related topics
 
 * Epic that introduced configurable user roles: #898
-* Data access based on user rights and user's jurisdiction: https://github.com/hzi-braunschweig/SORMAS-Project/blob/development/sormas-backend/doc/UserDataAccess.md
-* Using keycloak as authentication provider: https://github.com/hzi-braunschweig/SORMAS-Project/blob/development/docs/SERVER_SETUP.md#keycloak-server
-* The available jurisdiction levels (nation, region, district, health facility, etc.) are defined in Java code (https://github.com/hzi-braunschweig/SORMAS-Project/blob/development/sormas-api/src/main/java/de/symeda/sormas/api/user/JurisdictionLevel.java)
+* Data access based on user rights and user's jurisdiction: [https://github.com/hzi-braunschweig/SORMAS-Project/blob/development/sormas-backend/doc/UserDataAccess.md](https://github.com/hzi-braunschweig/SORMAS-Project/blob/development/sormas-backend/doc/UserDataAccess.md)
+* Using keycloak as authentication provider: [https://github.com/hzi-braunschweig/SORMAS-Project/blob/development/docs/SERVER_SETUP.md#keycloak-server](https://github.com/hzi-braunschweig/SORMAS-Project/blob/development/docs/SERVER_SETUP.md#keycloak-server)
+* The available jurisdiction levels (nation, region, district, health facility, etc.) are defined in Java code [https://github.com/hzi-braunschweig/SORMAS-Project/blob/development/sormas-api/src/main/java/de/symeda/sormas/api/user/JurisdictionLevel.java](https://github.com/hzi-braunschweig/SORMAS-Project/blob/development/sormas-api/src/main/java/de/symeda/sormas/api/user/JurisdictionLevel.java)

docs/TROUBLESHOOTING.md

@@ -33,11 +33,25 @@ Performance logging can be used to find out which part of the code or system mig
 Performance logs can be analyzed in detail using the `PerformanceLogAnalysisGenerator`. To use this tool, set the `PerformanceLoggingInterceptor`'s log level
 to `TRACE` as described above and reproduce the scenario you want to investigate on the server instance.
 
-After this, process the debug log file (default path: `/opt/domains/sormas/logs/application.debug`) using the `PerformanceLogAnalysisGenerator`. This will
+After this, process the debug log file (default path: `/opt/domains/sormas/logs/application.debug`) using the `PerformanceLogAnalysisGenerator`.
+The log file's path is specified as the program argument when calling `PerformanceLogAnalysisGenerator`'s `main` method. Processing the log file will
 generate three files (`<logfileName>.csv`, `<logfileName>.txt`, `<logfileName>.html`) to further investigate method runtimes.
 
 `<logfileName>.html` provides a navigable overview of methods along with runtime statistics (total, min, max and average time) and calls to sub methods.
 
+Sometimes it is convenient to analyze a number of different scenarios in a row. To do so, produce snippets of the `application.debug` log using `tail` for each
+of the scenarios to be investigated:
+
+1. start `tail -f <logfileName> > <snippetDirectory>/<snippet.debug>`
+
+2. replay the steps to be analyzed
+
+3. stop `tail -f`
+
+The `PerformanceLogAnalysisGenerator` can now batch process all of the snippets by pointing to the directory instead of a log file.
+Calling `PerformanceLogAnalysisGenerator.main` with argument `<snippetDirectory>` generates the analysis files (`.csv`, `.txt`, `.html`)
+for each file `*.debug` in this directory. The generated files will be placed in `<snippetDirectory>`, too.
+
 ### Log Slow SQL Queries in PostgreSQL
 
 You can enable the logging of slow SQL queries in your PostgreSQL server in `postgresql.conf`:

sormas-api/pom.xml

@@ -32,11 +32,6 @@
 			<artifactId>javaee-web-api</artifactId>
 		</dependency>
 
-		<dependency>
-			<groupId>joda-time</groupId>
-			<artifactId>joda-time</artifactId>
-		</dependency>
-
 		<dependency>
 			<groupId>com.opencsv</groupId>
 			<artifactId>opencsv</artifactId>
@@ -103,6 +98,12 @@
 			<scope>test</scope>
 		</dependency>
 
+		<dependency>
+			<groupId>ch.qos.logback</groupId>
+			<artifactId>logback-classic</artifactId>
+			<scope>test</scope>
+		</dependency>
+
 	</dependencies>
 
 </project>

sormas-api/src/main/java/de/symeda/sormas/api/EntityDto.java

@@ -25,6 +25,8 @@
 import com.fasterxml.jackson.annotation.JsonIgnore;
 import com.fasterxml.jackson.annotation.JsonInclude;
 
+import de.symeda.sormas.api.audit.AuditInclude;
+import de.symeda.sormas.api.audit.AuditedClass;
 import de.symeda.sormas.api.i18n.Validations;
 import de.symeda.sormas.api.utils.FieldConstraints;
 import de.symeda.sormas.api.utils.Outbreaks;
@@ -39,6 +41,7 @@
  *              will reduce data transferred to something between 20% and 50% -
  *              especially for fields that are not needed for all diseases
  */
+@AuditedClass
 @JsonInclude(JsonInclude.Include.NON_NULL)
 public abstract class EntityDto implements Serializable, Cloneable, HasUuid {
 
@@ -54,6 +57,7 @@ public abstract class EntityDto implements Serializable, Cloneable, HasUuid {
 	@Outbreaks
 	@Pattern(regexp = UUID_REGEX, message = Validations.uuidPatternNotMatching)
 	@Size(min = FieldConstraints.CHARACTER_LIMIT_UUID_MIN, max = FieldConstraints.CHARACTER_LIMIT_UUID_MAX, message = Validations.textSizeNotInRange)
+	@AuditInclude
 	private String uuid;
 
 	protected EntityDto() {

sormas-api/src/main/java/de/symeda/sormas/api/FacadeProvider.java

@@ -82,10 +82,10 @@
 import de.symeda.sormas.api.share.ExternalShareInfoFacade;
 import de.symeda.sormas.api.sormastosormas.SormasToSormasEncryptionFacade;
 import de.symeda.sormas.api.sormastosormas.SormasToSormasFacade;
-import de.symeda.sormas.api.sormastosormas.caze.SormasToSormasCaseFacade;
-import de.symeda.sormas.api.sormastosormas.contact.SormasToSormasContactFacade;
-import de.symeda.sormas.api.sormastosormas.event.SormasToSormasEventFacade;
-import de.symeda.sormas.api.sormastosormas.externalmessage.SormasToSormasExternalMessageFacade;
+import de.symeda.sormas.api.sormastosormas.entities.caze.SormasToSormasCaseFacade;
+import de.symeda.sormas.api.sormastosormas.entities.contact.SormasToSormasContactFacade;
+import de.symeda.sormas.api.sormastosormas.entities.event.SormasToSormasEventFacade;
+import de.symeda.sormas.api.sormastosormas.entities.externalmessage.SormasToSormasExternalMessageFacade;
 import de.symeda.sormas.api.sormastosormas.share.incoming.SormasToSormasShareRequestFacade;
 import de.symeda.sormas.api.sormastosormas.share.outgoing.ShareRequestInfoFacade;
 import de.symeda.sormas.api.sormastosormas.share.outgoing.SormasToSormasShareInfoFacade;

sormas-api/src/main/java/de/symeda/sormas/api/action/ActionHelper.java

@@ -19,7 +19,7 @@
 
 import java.util.Date;
 
-import org.joda.time.DateTime;
+import de.symeda.sormas.api.utils.UtilDate;
 
 public final class ActionHelper {
 
@@ -28,6 +28,6 @@ private ActionHelper() {
 	}
 
 	public static Date getDefaultDate() {
-		return new DateTime().toDate();
+		return UtilDate.now();
 	}
 }

sormas-api/src/main/java/de/symeda/sormas/api/audit/AuditInclude.java

@@ -0,0 +1,19 @@
+package de.symeda.sormas.api.audit;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * This annotation is used to mark fields (in DTOs or certain important classes referenced by DTO fields) which should
+ * be included in the audit log. It can be applied to fields and, if need be, to methods returning a specifically
+ * crafted audit representation. The audit logger will pick up the field if the containing class is annotated
+ * with {@link AuditedClass}.
+ */
+@Retention(RetentionPolicy.RUNTIME)
+@Target({
+	ElementType.FIELD,
+	ElementType.METHOD })
+public @interface AuditInclude {
+}

sormas-api/src/main/java/de/symeda/sormas/api/audit/AuditedClass.java

@@ -0,0 +1,16 @@
+package de.symeda.sormas.api.audit;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * This annotation is used to mark DTO as auditable. This annotation being present causes the log line to contain at
+ * least the class name. If more information is required, the fields to be included in the log line can be marked with
+ * {@link AuditInclude}.
+ */
+@Retention(RetentionPolicy.RUNTIME)
+@Target(ElementType.TYPE)
+public @interface AuditedClass {
+}

sormas-api/src/main/java/de/symeda/sormas/api/caze/CaseDataDto.java

@@ -508,6 +508,7 @@ public class CaseDataDto extends SormasToSormasShareableDto {
 	private YesNoUnknown postpartum;
 	private Trimester trimester;
 	private FollowUpStatus followUpStatus;
+	@SensitiveData
 	@Size(max = CHARACTER_LIMIT_BIG, message = Validations.textTooLong)
 	private String followUpComment;
 	private Date followUpUntil;
@@ -574,6 +575,7 @@ public class CaseDataDto extends SormasToSormasShareableDto {
 	private boolean notACaseReasonOther;
 
 	@HideForCountriesExcept
+	@SensitiveData
 	@Size(max = FieldConstraints.CHARACTER_LIMIT_TEXT, message = Validations.textTooLong)
 	private String notACaseReasonDetails;
 	private Date followUpStatusChangeDate;
@@ -592,6 +594,7 @@ public class CaseDataDto extends SormasToSormasShareableDto {
 	private Map<String, String> externalData;
 	private boolean deleted;
 	private DeletionReason deletionReason;
+	@SensitiveData
 	@Size(max = FieldConstraints.CHARACTER_LIMIT_TEXT, message = Validations.textTooLong)
 	private String otherDeletionReason;
 

sormas-api/src/main/java/de/symeda/sormas/api/caze/CaseFacade.java

@@ -56,13 +56,6 @@
 @Remote
 public interface CaseFacade extends CoreFacade<CaseDataDto, CaseIndexDto, CaseReferenceDto, CaseCriteria> {
 
-	List<CaseDataDto> getAllActiveCasesAfter(Date date);
-
-	/**
-	 * Additional change dates filters for: sample, pathogenTests, patient and location.
-	 */
-	List<CaseDataDto> getAllActiveCasesAfter(Date date, boolean includeExtendedChangeDateFilters);
-
 	long count(CaseCriteria caseCriteria, boolean ignoreUserFilter);
 
 	List<CaseSelectionDto> getCaseSelectionList(CaseCriteria caseCriteria);
@@ -106,8 +99,6 @@ List<CaseExportDto> getExportList(
 
 	List<String> getAllActiveUuids();
 
-	List<CaseDataDto> getAllActiveCasesAfter(Date date, Integer batchSize, String lastSynchronizedUuid);
-
 	String getUuidByUuidEpidNumberOrExternalId(String searchTerm, CaseCriteria caseCriteria);
 
 	List<MapCaseDto> getCasesForMap(

sormas-api/src/main/java/de/symeda/sormas/api/contact/ContactJurisdictionFlagsDto.java

@@ -20,19 +20,23 @@
 
 public class ContactJurisdictionFlagsDto implements Serializable {
 
-    private Boolean isInJurisdiction;
-    private Boolean isCaseInJurisdiction;
+	private Boolean isInJurisdiction;
+	private Boolean isCaseInJurisdiction;
 
-    public ContactJurisdictionFlagsDto(Boolean isInJurisdiction, Boolean isCaseInJurisdiction) {
-        this.isInJurisdiction = isInJurisdiction;
-        this.isCaseInJurisdiction = isCaseInJurisdiction;
-    }
+	public ContactJurisdictionFlagsDto(Object[] flags) {
+		this((boolean) flags[1], (boolean) flags[2]);
+	}
 
-    public Boolean getInJurisdiction() {
-        return isInJurisdiction;
-    }
+	public ContactJurisdictionFlagsDto(Boolean isInJurisdiction, Boolean isCaseInJurisdiction) {
+		this.isInJurisdiction = isInJurisdiction;
+		this.isCaseInJurisdiction = isCaseInJurisdiction;
+	}
 
-    public Boolean getCaseInJurisdiction() {
-        return isCaseInJurisdiction;
-    }
+	public Boolean getInJurisdiction() {
+		return isInJurisdiction;
+	}
+
+	public Boolean getCaseInJurisdiction() {
+		return isCaseInJurisdiction;
+	}
 }

sormas-backend/src/main/java/de/symeda/sormas/backend/dashboard/PathogenTestResultDto.java → sormas-api/src/main/java/de/symeda/sormas/api/dashboard/PathogenTestResultDto.java

@@ -1,11 +1,11 @@
-package de.symeda.sormas.backend.dashboard;
+package de.symeda.sormas.api.dashboard;
 
 import java.io.Serializable;
 import java.util.Date;
 
 import de.symeda.sormas.api.sample.PathogenTestResultType;
 
-class PathogenTestResultDto implements Serializable {
+public class PathogenTestResultDto implements Serializable {
 
 	private static final long serialVersionUID = 1L;