-
Notifications
You must be signed in to change notification settings - Fork 142
Auditing & Data History
Levente Gal edited this page Mar 7, 2024
·
2 revisions
To comply with data protection regulation, we need to make sure that SORMAS provides an audit log trail which can be easily ingested by dedicated log processing systems and allows investigation by officials.
Use cases:
- User opens case in UI -> call to backend method CaseFacade.getCaseDataByUuid needs to be logged
- User edits/deletes case in UI -> call to backend method CaseFacade.save/deleteCase needs to be logged
- User does export -> call to CaseFacade.getExportList needs to be logged
- User opens case directory -> call to CaseFacade.getIndexList needs to be logged
The audit trail gets populated by automatically logging every invocation of a facade/EJB method. By this, we can trace every interaction with the system (i.e., via Vaadin UI or REST). We output the collected logs to a user configurable log sink such that the logs can be easily ingested for further processing.
The most important module that is covered is the SORMAS backend.
Related epic: https://github.com/hzi-braunschweig/SORMAS-Project/issues/7904
Audit logging can be set up by setting the audit.logger.config property in the sormas.properties file to a path that points to a logback configuration file.
There is an example file in SORMAS source code in the sormas-base/setup folder. This writes audit logs to a file but Logback can be easily configured to write to other destinations like a database or a log processing system.
For example adding a Loki4jAppender appender to the logback configuration file will send the logs to a Loki instance.
e.g. The following configuration will send the logs to a Loki instance running on localhost:
<configuration>
<appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
<http>
<url>http://localhost:3100/loki/api/v1/push</url>
</http>
<format>
<label>
<pattern>app=my-app,host=localhost,level=%level</pattern>
</label>
<message>
<pattern>l=%level h=localhost c=%logger{20} t=%thread | %msg %ex</pattern>
</message>
<sortByTime>true</sortByTime>
</format>
</appender>
<root level="DEBUG">
<appender-ref ref="LOKI"/>
</root>
</configuration>NOTE: in order to make the Loki appender work, you need to add the latest
loki-logback-appender.jaras a dependency in the libs folder of the SORMAS domain in your payara
For the general purpose of audit logging SORMAS logs access to the SORMAS backend with details on what exactly has been accessed, without including sensitive data (e.g. names).
For the logging output format, SORMAS is using the FHIR R4 AuditEvent resource, which is based on the IHE ATNA audit profile. FHIR is a well-known and established standard for exchange of medical related data.
A FHIR AuditEvent resource has a specified JSON representation which is compact and easy to digest by log processing systems like Loki or ELK.
These are the most important fields logged in the AuditEvent class:
| Content | Description | Field in AuditEvent resource |
|---|---|---|
| Access type | Create/Read/Update/Delete/Execute | action |
| timestamp | Timestamp of the event | recorded |
| actor | Functional instance causing the event (e.g., user). Users should be identified with a human readable name besides using the UUID | agent |
| Executing instance | Identifier of the system(component) generating the audit event | source.site |
| Activity/Event | Description of the activity with unique reference(uuid) w.r.t. the accesses data | entitiy |
Additionally, there is the field type, which will be populated according to the following table.
| Code | System | Name | Description |
|---|---|---|---|
| 110100 | http://dicom.nema.org/resources/ontology/DCM | Application Activity | Start, Stop |
| 110106 | http://dicom.nema.org/resources/ontology/DCM | Export | Database level export |
| 110112 | http://dicom.nema.org/resources/ontology/DCM | Query | A request for multiple entities |
| 110110 | http://dicom.nema.org/resources/ontology/DCM | Patient Record | Use for read/write/delete operations on patient related entities (case, contact, symptoms, samples, ...) |
| object | http://terminology.hl7.org/CodeSystem/audit-event-type | An Operation on other Objects | Use for read/write/delete operations on all other entities, most importantly users, infrastructure, configuration and tasks |
SORMAS logs the following events:
- Application lifecycle events (start, stop)
- Data reads
- Creation and change of data
- Deletion of data
- REST api calls
- Failed login attempts
External message adapters can also log events using the AuditLoggerFacade
- Under all circumstances the principle of data minimization is followed
- Only the UUID of entities is logged, not the full entity
- The timestamps are in ISO format.
In general the log contains only pseudonymized personal data, the only exception being the name of the active user.
-
Failed login
{ "resourceType": "AuditEvent", "type": { "system": "https://hl7.org/fhir/R4/valueset-audit-event-type.html", "code": "110114", "display": "User Authentication" }, "subtype": [ { "system": "https://hl7.org/fhir/R4/valueset-audit-event-sub-type.html", "code": "110122", "display": "Login" } ], "action": "E", "recorded": "2024-03-07T12:38:17.803+02:00", "outcome": "4", "outcomeDesc": "Authentication failed", "agent": [ { "name": "Username cannot be determined without ID token. Check Keycloak logs for details." } ], "source": { "site": "sormas.lu - UI MultiAuthenticationMechanism" }, "entity": [ { "what": { "reference": "sormas-ui/Callback" } } ] } -
Load case directory
{ "resourceType": "AuditEvent", "action": "R", "period": { "start": "2024-03-07T12:38:34+02:00", "end": "2024-03-07T12:38:34+02:00" }, "recorded": "2024-03-07T12:38:34.912+02:00", "outcomeDesc": "[CaseIndexDto(uuid=SYQMFJ-N2YXQ7-PMHXUE-SAQU2FIE),CaseIndexDto(uuid=RAG4WE-2BUPEN-3OXQW5-TB7OKE4Y),CaseIndexDto(uuid=WILIOE-FFX3L7-C46FKD-PCN6CJQU),CaseIndexDto(uuid=SCYB5H-MLZJJB-PFAVRD-6IQQ2FMU),CaseIndexDto(uuid=S7DQVF-4XCHEO-YA57KV-HHAHKF5Q),CaseIndexDto(uuid=XFLEBJ-A5FAY7-DGVLX7-XZ4VKP6U),CaseIndexDto(uuid=TUWWHZ-E5LQPN-GCVYDQ-VYTVKMAY),CaseIndexDto(uuid=RH7FHT-BMD6BV-T7V7VY-LIC4CHCM),CaseIndexDto(uuid=Q2YYLE-TXYILI-X5Q7FE-BJHL2IG4),CaseIndexDto(uuid=SF2Y72-YYG335-XI3ZFX-VTMBSMT4),CaseIndexDto(uuid=VORYUS-BB3TN5-EWQ6CB-TLM7KI4E),CaseIndexDto(uuid=UI47KF-UCZMA4-DHEHXU-MLMKKE5A),CaseIndexDto(uuid=TCO4DA-2GI64Y-D7IK2G-KBKACNXM),CaseIndexDto(uuid=T62OHY-7ID2Z2-LTVWSR-VCZVKMQE),CaseIndexDto(uuid=VCGLLU-REDG56-BVEULC-WZKASOOU),CaseIndexDto(uuid=T4ZZBB-2UHGS2-2TLMTH-26FOSBPQ),CaseIndexDto(uuid=SAGWGH-JS2LQA-WOV3F5-VAQICK5Y),CaseIndexDto(uuid=VZ4SMM-OBWY72-7DVLC4-6FZL2LFY),CaseIndexDto(uuid=V3ND6S-66VQIV-S642FO-5F42KIQ4),CaseIndexDto(uuid=VQFIG5-QDTIK7-UC2UAX-KD52CAHI),CaseIndexDto(uuid=QYL2XE-KU6BKX-CO7JXV-35PN2F4E),CaseIndexDto(uuid=VT3ADI-TYG6PJ-YYOHCT-UVWUKMPQ),CaseIndexDto(uuid=RVXCET-Y35DWT-3DYA34-4MXX2HLI),CaseIndexDto(uuid=RKU56K-HGXFNO-AFXOCX-YWH5KJZQ),CaseIndexDto(uuid=XABT2R-MVHQA4-PN3BNS-DWKKCCY4),CaseIndexDto(uuid=TDPU3J-ZSKZBD-BLRHEV-7PC2KELY),CaseIndexDto(uuid=X2TFPZ-BULBSB-7NVZIP-4K3D2ICA),CaseIndexDto(uuid=WKWEG3-EGYL2J-GKUHDC-F22CSBRU),CaseIndexDto(uuid=TOAQVR-IU3CKF-LXYJKL-IT6VKKOI),CaseIndexDto(uuid=QR2PLT-JVRN5H-BIMWJO-H3MJKAN4),CaseIndexDto(uuid=QS3UDV-QFVBMM-LGDB2F-LNCLSIRI),CaseIndexDto(uuid=WLJVI5-GNOLVN-D335JX-Q3HBKE7I),CaseIndexDto(uuid=UJLPOL-LSJ7ZS-MUWCCO-3FGQCICI),CaseIndexDto(uuid=XMXHHD-NE526P-EMUVCU-NGPDCFQ4),CaseIndexDto(uuid=UNPGCP-7HRLLF-JCRCZ6-HMAG2E6E),CaseIndexDto(uuid=Q44DQL-ECYSWP-XHRB5G-FNHXKEOA),CaseIndexDto(uuid=XZN26C-NZJXPS-3BLLPD-GDSD2GIQ),CaseIndexDto(uuid=RPZS4S-MIWQY6-7HZ6A5-SG7T2NRY),CaseIndexDto(uuid=RGJQ3S-2OBPHZ-AJY7CM-C6M3CNWU),CaseIndexDto(uuid=RXZ3FB-XNNB2M-TJZX2I-4C3EKAXI),CaseIndexDto(uuid=STPE5C-JV62ST-P7NTB4-5JMZCOQE),CaseIndexDto(uuid=WTP4W2-3DP76T-N4E5EV-YZESSKXM),CaseIndexDto(uuid=QUNZWX-AAKOXA-PEHSCB-DU6HKHSI),CaseIndexDto(uuid=WJOFHO-2X5NIA-VKPJGT-FON52IMU),CaseIndexDto(uuid=QGHGY4-B3RUOC-TX27YD-JUSQ2DPI),CaseIndexDto(uuid=RT4XRJ-VM74IK-5TYURK-B5NGKO2Y),CaseIndexDto(uuid=V6XOJS-SX4ESD-XQQSCS-JJXOCBQA),CaseIndexDto(uuid=RPVLIR-R7YK3E-QAXKU3-QVRFKLZA),CaseIndexDto(uuid=XSQTVG-4IXW7A-LAHYTW-2D5CKMDY),CaseIndexDto(uuid=W7UK6H-KZSPPD-PGHWIO-JI4MCCGU),CaseIndexDto(uuid=UMVX63-P6R4I3-SI7IK5-ZNEMSLC4),CaseIndexDto(uuid=VZKFEJ-Q5V2XU-5SQPRM-ZZACSDMQ),CaseIndexDto(uuid=RGH5HS-A3TSNU-5LZXIR-OOCICME4),CaseIndexDto(uuid=SYXQBI-UOXGW7-KPNDZU-ZCJNSDYU),CaseIndexDto(uuid=TJL5UI-QFCEOX-4NA4WA-UTVRCPJQ),CaseIndexDto(uuid=RDEGZP-OEFQZK-E3ZC5C-MYG2SHWY),CaseIndexDto(uuid=Q4ICNF-Y5R56L-NPSAXN-O52WCOUU),CaseIndexDto(uuid=USASG5-3HG5QV-GUZBS6-VKLOKF3A),CaseIndexDto(uuid=SRYNG6-7BL73Q-LAC6TK-V2H62NKI),CaseIndexDto(uuid=V6WVV4-MFF6G7-CBTDMQ-VNX7KG34),CaseIndexDto(uuid=UEQ4TR-QXWOXK-IERBBB-LE5SSNEA),CaseIndexDto(uuid=U2W5EI-34BFKM-FMCGON-4COT2F3M),CaseIndexDto(uuid=UFKBX6-WLAQRE-DOJNWN-2AFWSEYA),CaseIndexDto(uuid=XAOL6R-S7BFCM-6BDOLB-SZOXKGIY),CaseIndexDto(uuid=SFPALU-Z4QORK-ZYPOEK-2QK32DUE),CaseIndexDto(uuid=RBMPYH-VBOOBC-BSEJKD-7KL7CMQQ),CaseIndexDto(uuid=V26N2S-5XKDTB-BMTJBS-HS7S2LAU)]", "agent": [ { "type": { "coding": [ { "system": "https://www.hl7.org/fhir/valueset-participation-role-type.html", "code": "humanuser", "display": "human user" } ] }, "who": { "identifier": { "value": "UOSUJW-BRSDJL-ZGSXEW-3XCUCLHI" } }, "name": "NatUser" } ], "source": { "site": "sormas.lu", "type": [ { "system": "http://terminology.hl7.org/CodeSystem/security-source-type", "code": "4", "display": "Application Server" } ] }, "entity": [ { "what": { "reference": "public java.util.List de.symeda.sormas.backend.caze.CaseFacadeEjb.getIndexList(de.symeda.sormas.api.utils.criteria.BaseCriteria,java.lang.Integer,java.lang.Integer,java.util.List)" }, "detail": [ { "type": "param", "valueString": "CaseCriteria(birthdateDD=null,birthdateMM=null,birthdateYYYY=null,caseClassification=null,caseLike=null,caseOrigin=null,caseUuidsForMerge=null,community=null,creationDateFrom=null,creationDateTo=null,dateFilterOption=By Date,dateTypeCalss=class de.symeda.sormas.api.caze.NewCaseDateType,disease=null,diseaseVariant=null,district=null,eventLike=null,facilityType=null,facilityTypeGroup=null,followUpStatus=null,followUpUntilFrom=null,followUpUntilTo=null,followUpVisitsFrom=null,followUpVisitsInterval=null,followUpVisitsTo=null,healthFacility=null,includeCasesFromOtherJurisdictions=false,investigationStatus=null,jurisdictionType=null,mustBePortHealthCaseWithoutFacility=null,mustHaveCaseManagementData=null,mustHaveNoGeoCoordinates=null,newCaseDateFrom=null,newCaseDateTo=null,newCaseDateType=null,onlyCasesWithDontShareWithExternalSurvTool=null,onlyCasesWithEvents=false,onlyCasesWithReinfection=null,onlyContactsFromOtherInstances=null,onlyEntitiesChangedSinceLastSharedWithExternalSurvTool=null,onlyEntitiesNotSharedWithExternalSurvTool=null,onlyEntitiesSharedWithExternalSurvTool=null,onlyQuarantineHelpNeeded=null,onlyShowCasesWithFulfilledReferenceDefinition=null,outcome=null,person=null,personLike=null,pointOfEntry=null,presentCondition=null,quarantineTo=null,quarantineType=null,region=null,reinfectionStatus=null,relevanceStatus=Active,reportDateTo=null,reportingUserLike=null,reportingUserRole=null,sourceCaseInfoLike=null,surveillanceOfficer=null,symptomJournalStatus=null,vaccinationStatus=null,withExtendedQuarantine=null,withOwnership=true,withReducedQuarantine=null,withoutResponsibleOfficer=null)" }, { "type": "param", "valueString": "0" }, { "type": "param", "valueString": "100" }, { "type": "param", "valueString": "[]" } ] } ] } -
Load case data
{ "resourceType": "AuditEvent", "action": "R", "period": { "start": "2024-03-07T12:38:39+02:00", "end": "2024-03-07T12:38:39+02:00" }, "recorded": "2024-03-07T12:38:39.341+02:00", "outcomeDesc": "CaseDataDto(uuid=SYQMFJ-N2YXQ7-PMHXUE-SAQU2FIE)", "agent": [ { "type": { "coding": [ { "system": "https://www.hl7.org/fhir/valueset-participation-role-type.html", "code": "humanuser", "display": "human user" } ] }, "who": { "identifier": { "value": "UOSUJW-BRSDJL-ZGSXEW-3XCUCLHI" } }, "name": "NatUser" } ], "source": { "site": "sormas.lu", "type": [ { "system": "http://terminology.hl7.org/CodeSystem/security-source-type", "code": "4", "display": "Application Server" } ] }, "entity": [ { "what": { "reference": "public de.symeda.sormas.api.caze.CaseDataDto de.symeda.sormas.backend.caze.CaseFacadeEjb.getCaseDataByUuid(java.lang.String)" }, "detail": [ { "type": "param", "valueString": "SYQMFJ-N2YXQ7-PMHXUE-SAQU2FIE" } ] } ] } -
Update a case
{ "resourceType": "AuditEvent", "action": "U", "period": { "start": "2024-03-07T12:39:32+02:00", "end": "2024-03-07T12:39:33+02:00" }, "recorded": "2024-03-07T12:39:33.434+02:00", "outcomeDesc": "CaseDataDto(uuid=SYQMFJ-N2YXQ7-PMHXUE-SAQU2FIE)", "agent": [ { "type": { "coding": [ { "system": "https://www.hl7.org/fhir/valueset-participation-role-type.html", "code": "humanuser", "display": "human user" } ] }, "who": { "identifier": { "value": "UOSUJW-BRSDJL-ZGSXEW-3XCUCLHI" } }, "name": "NatUser" } ], "source": { "site": "sormas.lu", "type": [ { "system": "http://terminology.hl7.org/CodeSystem/security-source-type", "code": "4", "display": "Application Server" } ] }, "entity": [ { "what": { "reference": "public de.symeda.sormas.api.EntityDto de.symeda.sormas.backend.caze.CaseFacadeEjb.save(de.symeda.sormas.api.EntityDto)" }, "detail": [ { "type": "param", "valueString": "CaseDataDto(uuid=SYQMFJ-N2YXQ7-PMHXUE-SAQU2FIE)" } ] } ] } -
Archive a case
{ "resourceType": "AuditEvent", "action": "U", "period": { "start": "2024-03-07T12:39:39+02:00", "end": "2024-03-07T12:39:39+02:00" }, "recorded": "2024-03-07T12:39:39.393+02:00", "outcomeDesc": "ProcessedEntity", "agent": [ { "type": { "coding": [ { "system": "https://www.hl7.org/fhir/valueset-participation-role-type.html", "code": "humanuser", "display": "human user" } ] }, "who": { "identifier": { "value": "UOSUJW-BRSDJL-ZGSXEW-3XCUCLHI" } }, "name": "NatUser" } ], "source": { "site": "sormas.lu", "type": [ { "system": "http://terminology.hl7.org/CodeSystem/security-source-type", "code": "4", "display": "Application Server" } ] }, "entity": [ { "what": { "reference": "public de.symeda.sormas.api.common.progress.ProcessedEntity de.symeda.sormas.backend.caze.CaseFacadeEjb.archive(java.lang.String,java.util.Date,boolean)" }, "detail": [ { "type": "param", "valueString": "SYQMFJ-N2YXQ7-PMHXUE-SAQU2FIE" }, { "type": "param", "valueString": "Thu Mar 07 00:00:00 EET 2024" }, { "type": "param", "valueString": "false" } ] } ] } -
Delete a case
{ "resourceType": "AuditEvent", "action": "D", "period": { "start": "2024-03-07T12:39:49+02:00", "end": "2024-03-07T12:39:49+02:00" }, "recorded": "2024-03-07T12:39:49.307+02:00", "outcomeDesc": "null", "agent": [ { "type": { "coding": [ { "system": "https://www.hl7.org/fhir/valueset-participation-role-type.html", "code": "humanuser", "display": "human user" } ] }, "who": { "identifier": { "value": "UOSUJW-BRSDJL-ZGSXEW-3XCUCLHI" } }, "name": "NatUser" } ], "source": { "site": "sormas.lu", "type": [ { "system": "http://terminology.hl7.org/CodeSystem/security-source-type", "code": "4", "display": "Application Server" } ] }, "entity": [ { "what": { "reference": "public void de.symeda.sormas.backend.caze.CaseFacadeEjb.delete(java.lang.String,de.symeda.sormas.api.common.DeletionDetails) throws de.symeda.sormas.api.externalsurveillancetool.ExternalSurveillanceToolRuntimeException,de.symeda.sormas.api.sormastosormas.SormasToSormasRuntimeException" }, "detail": [ { "type": "param", "valueString": "SYQMFJ-N2YXQ7-PMHXUE-SAQU2FIE" }, { "type": "param", "valueString": "DeletionDetails(deletionReason=Deletion request by affected person according to GDPR)" } ] } ] }
Use case: A user observes incorrect data in a few cases. To understand how exactly this came to be it should be possible to extract the change history of the cases, including what exactly changed, at what point in time and by which user the change was made.
Goals:
- Provide the information when and by whom a change was made
- Provide what was changed / what the data looked like before and after the change
SORMAS uses temporal tables to provide a history of all data changes. These automatically create a copy of the previous status in a history table each time a database entry is changed and provide it with a validity period. This also makes it possible to query the status of the data at any time in the past with simple SQL queries.