Merge pull request #135 from EC-DIGIT-CSIRC/resilience-fix

chg: improved resilience when parsing/analysing data
EC-DIGIT-CSIRC · Feb 5, 2025 · 6e19540 · 6e19540
2 parents 7429044 + b816e30
commit 6e19540
Show file tree

Hide file tree

Showing 4 changed files with 61 additions and 20 deletions.
diff --git a/README.md b/README.md
@@ -2,7 +2,6 @@
 
 ![sysdiagnose-512x512](https://github.com/EC-DIGIT-CSIRC/sysdiagnose/assets/750019/2742ca75-758e-4393-a2d1-5c94d09b0eb3)
 
-
 # Installation
 
 Note that you will need Python 3.11 or higher.
@@ -18,13 +17,13 @@ Create a virtual environment and install dependencies:
 
 On linux systems you may wish to install the [unifiedlogs](#unifiedlogs) parser. See below for instructions how to do this.
 
-
 # Quickstart
 
-## Case management:
+## Case management
 
 Creating a new case, with the optional `-c` parameter if you want to specify the case number yourself. (such as an uuid)
-```
+
+```bash
 $ sysdiag init test-data/iOS12/sysdiagnose_2019.02.13_15-50-14+0100_iPhone_OS_iPhone_16C101.tar.gz
 
 Sysdiagnose file has been processed
@@ -33,7 +32,7 @@ Case ID: 1
 
 Listing existing cases can be done easily:
 
-```
+```bash
 $ sysdiag cases
 Case ID              acquisition date           Serial number    Unique device ID                          iOS Version    Tags
 -------------------  -------------------------  ---------------  ----------------------------------------  -------------  ------
@@ -42,18 +41,19 @@ public               2023-05-24T13:29:15-07:00  F4GT2K24HG7K     e22f7f830e5dcc1
 
 The `cases` folder is the current folder by default.
 You can change this using the environment variable `SYSDIAGNOSE_CASES_PATH`, for example.
-```
+
+```bash
 $ export SYSDIAGNOSE_CASES_PATH='/path/to/folder'
 $ sysdiag list cases
 ```
 
+## Parsing data and converting it to a usable format
 
-## Parsing data and converting it to a usable format:
 Data of sysdiagnose is not always usable directly, use parsers to convert them to a nice json file.
 
 Run parsers:
 
-```
+```bash
 $ sysdiag -c 1 parse ps
 Execution success, output saved in: cases/1/parsed_data/ps.json
 
@@ -65,7 +65,7 @@ To run on all cases do not specify a case number or use `-c all`.
 
 List available parsers :
 
-```
+```bash
 $ sysdiag list parsers
 Parser Name            Parser Description
 ---------------------  ---------------------------------------------------------------------
@@ -103,9 +103,10 @@ wifisecurity           Parsing WiFi Security logs
 ```
 
 ## Analysers to process parsed data
+
 List analysers:
 
-```
+```bash
 $ sysdiag list analysers
 Analyser Name         Analyser Description
 --------------------  -------------------------------------------------------------------------------
@@ -121,12 +122,14 @@ yarascan              Scan the case folder using YARA rules ('./yara' or SYSDIAG
 ```
 
 Run analyser (make sure you run `parse all` before)
-```
+
+```bash
 $ sysdiag -c 1 analyse timesketch
 Execution success, output saved in: cases/1/parsed_data/timesketch.jsonl
 ```
 
 # Using the output
+
 Most of the parsers and analysers save their results in `jsonl` or `json` format. A few analysers use `txt` and more.
 Exported data is stored in the `<cases>/<case_id>/parsed_data` folder. You can configure your ingestion tool to automatically monitor and all that data.
 
@@ -137,6 +140,7 @@ The JSONL files are event based and (most often) structured with a a `timestamp`
 You might want to visualise timelines which you can extract via sysdiagnose in [Timesketch](https://timesketch.org/guides/admin/install/). Do note that timesketch expects timestamps in microseconds, that's why we made the `timesketch` analyser.
 
 Note that for a reasonable sysdiagnose log output, we recommend the following base requirements:
+
 - Ubuntu 20.04 or higher
 - 128GB of RAM
 - 4-8 virtual CPUs
@@ -148,6 +152,7 @@ Note that for a reasonable sysdiagnose log output, we recommend the following ba
 Using YARA rules are an easy and flexible way of spotting 'evil', the Yarascan analyser will help you out with that. It looks for YARA rules within __.yar__ files saved in the `./yara` folder or in the one designated by the environment varirable `SYSDIAGNOSE_YARA_RULES_PATH`.
 
 # UnifiedLogs
+
 This unifiedlogs parser tool is natively provided on a MacOS system. Fortunately some entities developed a linux compatible parser.
 
 By default sysdiagnose will use the Apple unifiedlogs `log` binary.
@@ -156,22 +161,48 @@ On linux it expects the Mandiant developed UnifiedLogs tool to be present in the
 ## Building macos-UnifiedLogs for linux
 
 First, ensure `cargo` is installed so you can build rust projects.
-```
+
+```bash
 sudo apt install cargo
 ```
+
 Now you can download and compile the code:
+
 ```bash
 git clone https://github.com/mandiant/macos-UnifiedLogs
 cd macos-UnifiedLogs/examples/unifiedlog_iterator/
 cargo build --release
 sudo cp ../target/release/unifiedlog_iterator /usr/local/bin/
 ```
+
 See `unifiedlog_iterator --help` for more instructions to use the tool, or use it directly through sysdiagnose.
 
+# Troubleshooting
+
+By default, whenever you [parse](#parsing-data-and-converting-it-to-a-usable-format) or [analyse](#analysers-to-process-parsed-data) data, the framework generates a log file in JSONL format in the subfolder `logs` located in the case folder.
+
+__Note:__ You can get that very same information in the output by adding the flag `-l DEBUG` to the parse/analyse command. But this is not very manageable when using all parsers/analysers.
+
+You will be able to identify the execution of a parser/analyser by, at least, two entries generated by the main module (__module: main__) with an extra field named parser/analyser that contains the name of the parser/analyser.
+
+In between those two entries, you may see any other produced by the parser/analyser, where the module, this time, will match the parser/analyser name.
+
+__Note:__ It is of utmost important that the parser/analyser provides logging to help troubleshooting potential issues. Please take a look to the [demo_parser](src/sysdiagnose/parsers/demo_parser.py) and the [demo_analyser](src/sysdiagnose/analysers/demo_analyser.py) files for inspiration.
+
+Below you can find an example of traces within the log file.
+
+```json
+{"levelname": "INFO", "module": "main", "message": "Parser 'demo_parser' started", "taskName": null, "parser": "demo_parser", "datetime": "2025-01-28T12:16:59.153931+01:00", "timestamp": 1738063019.153931}
+{"levelname": "INFO", "module": "demo_parser", "message": "Processing file ./cases/1/data/sysdiagnose_2024.09.18_11-33-57+0200_iPhone-OS_iPhone_21G93/demo_input_file.txt, new entry added", "taskName": null, "log_file": "./cases/1/data/sysdiagnose_2024.09.18_11-33-57+0200_iPhone-OS_iPhone_21G93/demo_input_file.txt", "entry": "{}", "datetime": "2025-01-28T12:16:59.201914+01:00", "timestamp": 1738063019.201914}
+{"levelname": "WARNING", "module": "demo_parser", "message": "Empty entry.", "taskName": null, "datetime": "2025-01-28T12:16:59.202071+01:00", "timestamp": 1738063019.202071}
+{"levelname": "INFO", "module": "main", "message": "Parser 'demo_parser' finished successfully", "taskName": null, "parser": "demo_parser", "result": 0, "datetime": "2025-01-28T12:16:59.202495+01:00", "timestamp": 1738063019.2024949}
+
+```
 
 # Supported iOS versions
 
-Tested On:
+Tested on:
+
 - python 3.11
 - iOS13 (to be confirmed)
 - iOS14 (to be confirmed)
@@ -180,7 +211,6 @@ Tested On:
 - iOS17
 - iOS18
 
-
 # Contributors
 
 - Dario BORREGUERO RINCON (European Commission - EC DIGIT Cybersecurity Operation Centre)
@@ -191,6 +221,7 @@ Tested On:
 - Benoît ROUSSILLE (European Parliament)
 - For the Apollo library: https://github.com/mac4n6/APOLLO
 
-# License
+# Licence
+
 This project is released under the European Public Licence
 https://commission.europa.eu/content/european-union-public-licence_en
diff --git a/src/sysdiagnose/analysers/demo_analyser.py b/src/sysdiagnose/analysers/demo_analyser.py
@@ -22,7 +22,8 @@ def execute(self):
         """
         try:
             print("DO SOMETHING HERE")
-            logger.info("log something here", extra={'analyser': __name__})
+            logger.info("log something here", extra={'field1': 'field1_info_details'})
+            logger.debug("log something for debugging purposes", extra={'field1': 'field1_debug_details'})
             if True:
                 logger.warning("This will log a warning")
                 # logger.error("This will log an error")

diff --git a/src/sysdiagnose/main.py b/src/sysdiagnose/main.py
@@ -142,7 +142,11 @@ def main():
                     result_str = "successfully" if result == 0 else "with errors"
                     logger.info(f"Parser '{parser}' finished {result_str}", extra={'parser': parser, 'result': result})
                 except NotImplementedError:
-                    logger.warning(f"Parser '{parser}' is not implemented yet, skipping", extra={'parser': parser})
+                    logger.warning(f"Parser '{parser}' is not implemented yet, skipping",
+                                   extra={'parser': parser, 'result': 'skipped'})
+                except Exception:
+                    logger.exception(f"Parser '{parser}' finished unexpectedly. Might be a sign of evil or a bug!",
+                                     extra={'parser': parser, 'result': 'error'})
 
     elif args.mode == 'analyse':
         # Handle analyse mode
@@ -177,7 +181,11 @@ def main():
                     result_str = "successfully" if result == 0 else "with errors"
                     logger.info(f"Analyser '{analyser}' finished {result_str}", extra={'analyser': analyser, 'result': result})
                 except NotImplementedError:
-                    logger.warning(f"Analyser '{analyser}' is not implemented yet, skipping", extra={'analyser': analyser})
+                    logger.warning(f"Analyser '{analyser}' is not implemented yet, skipping",
+                                   extra={'analyser': analyser, 'result': 'skipped'})
+                except Exception:
+                    logger.exception(f"Analyser '{analyser}' finished unexpectedley. Might be a sign of evil or a bug!",
+                                     extra={'analyser': analyser, 'result': 'error'})
 
     else:
         parser.print_help()

diff --git a/src/sysdiagnose/parsers/demo_parser.py b/src/sysdiagnose/parsers/demo_parser.py
@@ -31,11 +31,12 @@ def execute(self) -> list | dict:
                 # entry['datetime'] = timestamp.isoformat(timespec='microseconds')
                 # entry['timestamp'] = timestamp.timestamp()
                 result.append(entry)
-                logger.info(f"Processing file {log_file}, new entry added", extra={'log_file': log_file, 'entry': entry})
+                logger.info(f"Processing file {log_file}, new entry added", extra={'log_file': log_file})
+                logger.debug(f"Entry details {str(entry)}", extra={'entry': str(entry)})
                 if not entry:
                     logger.warning("Empty entry.")
                     # logger.error("Empty entry.")
-            except Exception as e:
+            except Exception:
                 logger.exception("This will log an error with the exception information")
                 # logger.warning("This will log a warning with the exception information", exc_info=True)
         return result