Home

ArkThor(Threat Categorization Based on Malware’s C2 Communication) is live

The ArkThor Tool is now live, offering users full functionality to explore its features and capabilities.

http://arkthor.westeurope.cloudapp.azure.com

Table of Contents

ArkThor

In today's digital world, network security is of utmost importance. Cyber-attacks are becoming more sophisticated and complex, making it increasingly difficult to detect and prevent them. Command-and-Control (C2) communication is a common technique used by attackers to control infected hosts and steal sensitive information. Therefore, it is crucial to identify and categorize network threats accurately to prevent and mitigate cyber-attacks. However, traditional methods of threat categorization are often insufficient in identifying and classifying these communications. This capstone project aims to develop a threat categorization tool based on C2 communication in archived/live stream Pcap files that can help organizations more effectively detect and respond to cyber threats. The resulting tool, ArkThor, represents safety and strength and is a cutting-edge threat categorization engine designed to empower organizations to stay ahead of emerging threats in the cybersecurity landscape.

ArkThor is comprised of three distinct layers, including the Presentation Layer, Middle Layer, and Core Layer, based on the analysis of C2 communication in Pcap files. The tool includes a user-friendly interface, various APIs for flexibility and scalability, and a SQLite database for data storage. The heart of ArkThor is the CORE Engine, which is responsible for analyzing the Pcap file for threat categorization. The CORE Engine includes three independent modules, including Packet Processing, Rule Parser, and Rule Authoring modules.

Additionally, the use of RabbitMQ in ArkThor allows for reliable communication between different components, making it a powerful tool for organizations of all sizes. The containerization of all components makes ArkThor easily deployable, with a microservice architecture that allows it to be used as a plug-and-analyze solution in any organization. The future work includes the integration of a Machine Learning model to train the core module, further enhancing the threat categorization engine's accuracy and effectiveness. Overall, ArkThor is a valuable tool for organizations looking to improve their threat categorization capabilities and enhance their overall cybersecurity infrastructure. It has been developed with the motto of “Organizations don’t buy products; they buy solutions to their problems”.

Key Words: Cybersecurity, Threat categorization, Command-and-Control (C2) communication, Pcap files, Network security, Cyber-attacks, Threat detection, Threat mitigation, ArkThor, CORE Engine, Packet Processing, Rule Parser, Rule Authoring, Machine Learning, Microservice architecture, RabbitMQ, User interface, APIs, SQLite database, Containerization, Scapy, Python, rule-engine.

Uncovering the Birth of ArkThor

The ArkThor product had a unique origin story, diverging from the conventional garage or basement narratives often associated with product development. Instead, its inception took place during the CSCD (Advanced Certification Program in Cyber Security and Cyber Defense) program, a capstone project organized by IIT-Kanpur in collaboration with Talent Sprint between 2022 and 2023.

At the project's outset, I (Mohammed, Jawed) and Sriram P generated some initial ideas. However, it was through the invaluable guidance and feedback of our exceptional mentors, namely Prof. Anand Handa and Nitish Kumar, that we were able to continuously refine and augment our product. Additionally, Prof. Sandeep Shukla's timely suggestions proved instrumental in shaping the direction of ArkThor, as we incorporated additional features and functionalities, gradually transforming our nascent concepts into a comprehensive and fully-fledged product.

The name "ArkThor" was meticulously chosen to capture the essence of the product. "Ark" symbolizes safety and protection, while "Thor" evokes associations with strength and power. By combining these elements, we sought to convey the robust security measures and formidable capabilities that the ArkThor product embodies.

Although we initially focused on simplicity, as the product evolved, we embraced the opportunity to introduce new features and enhancements to ArkThor. We remain open to receiving suggestions and feedback, and encourage you to reach out to us via email at [email protected]. Your input is highly valued as we strive to continually improve the ArkThor experience.

- Mohammed Jawed (MD)

ArkThor Demystified

The development of the ArkThor product was driven by the recognition of the vital importance of understanding Command and Control (C2) communications employed by attackers in the realm of cybersecurity. C2 communication serves as a prevalent technique utilized by malicious actors to exert control over compromised hosts and pilfer sensitive information. Accurate identification and categorization of network threats arising from C2 communication play a pivotal role in the prevention and mitigation of cyber attacks.

This project specifically delves into the networking aspect of C2 communicating malwares, delving into the intricate details of network packets and undertaking the task of classifying threats based on the distinct communication patterns employed by various malware families. By thoroughly analyzing and parsing network packets, the ArkThor product endeavors to provide a comprehensive understanding of the threat landscape, facilitating proactive measures against cyber attacks.

Logo

Executive Summary

The objective of the project is to classify network risks by examining the interaction between Command-and-Control (C2) and the compromised host within a company using either real-time pcap or archived pcap files. Analyzing the hazards identified within a corporate setting will enable the information security team to evaluate the influence of the threat on the organization's overall security stance.

The Core engine of this Project consists of three independent modules - Packet Processing, Rule Parser, and Rule Authoring - that work together to provide threat categorization by analyzing the C2 communication patterns and categorize the network threat into various categories such as BOKBOT, IcedID, Graftor, STRRAT, Cobalt Strike etc.

It includes a user-friendly interface built on ASP.NET Core and Bootstrap, making it easy for organizations to visualize and analyze their security data. The product also includes various APIs and a SQLite database, allowing for flexibility and scalability.

This Project is available for free on Github and DockerHub. Download and use it to grow and enhance your security infrastructure, and be sure to provide feedback to help the product evolve and improve over time. With ArkThor, you can take your organization's cybersecurity to the next level.

The project's outcome will provide valuable insights into the network threat landscape and help organizations take proactive measures to prevent and mitigate cyber-attacks.

In addition, the product is containerized, including the UI, APIs, core engine, and RabbitMQ. This allows for easy deployment and use in any organization.

Features and Contributions of ArkThor

ArkThor is an innovative threat categorization tool that offers a range of advanced features designed to improve threat detection capabilities and enhance overall cybersecurity infrastructure. ArkThor is available for free on Github and DockerHub.

Here are some of the key features and contributions of ArkThor:

Threat Categorization Based on C2 Communication: ArkThor is designed to categorize threats based on Command-and-Control (C2) communication in archived or live stream Pcap files. This allows the system to categorize the network threat into various categories, including BOKBOT, IcedID, Graftor, STRRAT, Cobalt Strike, and more. The core engine of ArkThor consists of three independent modules - Packet Processing, Rule Parser, and Rule Authoring - that work together to provide a comprehensive threat categorization engine. This approach provides organizations with a more accurate and effective way to detect and respond to cyber threats.
Three Distinct Layers: ArkThor is built with three distinct layers that enable organizations to identify and mitigate threats before they cause damage. The presentation layer provides end-users with a comprehensive set of information, enabling them to gain valuable insights into the threat landscape and take proactive measures to prevent and mitigate cyber-attacks. The middle layer consists of APIs, a database, and a message broker that work together to connect the outside world to the core layer. Finally, the core layer includes three independent modules, namely Packet Processing, Rule Parser, and Rule Authoring, that analyze the Pcap file based on C2 communication.
User-Friendly Interface: ArkThor includes a user-friendly interface that provides a simple and intuitive way for users to interact with the system. The interface offers various views and visualizations that enable users to quickly identify threats and take appropriate action.
Flexible and Scalable: ArkThor includes various APIs that provide flexibility and scalability to organizations of all sizes. The system can be easily integrated into existing infrastructures, allowing organizations to customize the tool to meet their specific needs.
Containerization and Microservice Architecture: ArkThor is built with containerization and microservice architecture that enables it to be easily deployed and used as a plug-and-analyze solution in any organization. The containerization also allows the system to be easily updated and maintained, ensuring that it remains up-to-date and effective.
Machine Learning Integration: In the future, ArkThor aims to integrate a Machine Learning model to train the core module, further enhancing the threat categorization engine's accuracy and effectiveness. This feature will enable the system to learn from past incidents and adapt to new threats in real-time.

The contributions of ArkThor to the field of cybersecurity are significant. The tool provides organizations with an advanced system for improving their threat detection capabilities and overall security posture. By categorizing threats based on C2 communication, ArkThor enables organizations to better understand the nature of the threat and take appropriate action. Additionally, the system's flexibility and scalability make it a valuable tool for organizations of all sizes, from small businesses to large enterprises. Overall, ArkThor is a powerful solution to the challenges of threat categorization in today's digital world.

Methodology

The ArkThor product is built using the following methodology: (1) ArkThor-Layer

Core Engine Development: The first step in the development process was to create the Core Engine. This involved designing and building three independent modules – Packet Processing, Rule Parser, and Rule Authoring. The Packet Processing module uses Scapy, an opensource library, to process packets. The Rule Parser module loads the ArkThor format rules and matches them with the output of the Packet Processing module. The Rule Authoring module contains rule components that can convert open-source rules or human-authored rules.
UI Development: The user interface (UI) of the product is built using Asp DotNet Core, Javascript and Bootstrap. The UI includes a dashboard, various pages for displaying analysis results, and a measurement page that shows valuable KPIs in the form of pie charts, bar graphs, and knobs.
API Development: To provide flexibility and scalability, the product includes various APIs. These APIs allow users to interact with the product programmatically and access the data in the product's SQLite database.
Database Management: The product uses a SQLite database to store analysis records, configuration settings, and other data. The database is managed using SQLite commands and queries.
Message Queue Integration: The product uses RabbitMQ, a message queue system, to handle communication between different modules and components. This allows for efficient and reliable communication between different parts of the product.
Containerization: The product has been containerized using Docker, making it easy to deploy and use in any organization.
Testing and Deployment: The product is tested thoroughly to ensure that it meets the requirements and works as expected. It is then deployed to the production environment using Docker, which allows for easy deployment and management of the product.
Machine Learning: In the future, we will collect a dataset of labeled network traffic that includes various types of threats by using public datasets and creating our own by collecting traffic from the C3i network and labeling it based on the threat type. After obtaining the labeled dataset, we will train a machine learning model using one of the available algorithms, such as decision trees, random forests, or neural networks. The trained model will be integrated into ArkThor's core engine, allowing it to categorize threats using both rule-based and machine learning-based methods.

When a new network packet is captured, ArkThor's packet processing module will extract relevant features from the packet and send it to the rule parser. The rule parser will then apply the predefined rules and the machine learning model to categorize the threat. The categorization result will be passed to the APIs and then to the user interface, where it can be displayed and analyzed.

Installation

Requirements

Host: Linux with Docker engine and Docker Compose Plugin
Host Resources: Analysis of PCAP files is resources intensive process but to start with
- RAM: 12 GB (minimum)
- Disk Space: 15GB (minimum)
Installing Docker Engine on UBUNTU
- https://docs.docker.com/engine/install/ubuntu/
Installing Docker Compose Plugin
- https://docs.docker.com/compose/install/linux/#install-using-the-repository

Running ArkThor Product(without SSL/TLS)

Host- Linux

Following section is also application to Windows host with Docker Desktop (make sure docker is running as Linux container )

With Out reverse Proxy

1. Get Docker Compose File to local host machine (With Default Configuration and Settings)
curl -O https://raw.githubusercontent.com/JawedCIA/ArkThor/master/Arkthor/Docker/Linux/DockerCompose.yml
2. Run DockerCompose.yml file to bring ArkThor Product in live
docker-compose -f DockerCompose.yml up -d

The above command will create multiple containers, including ArkthorUI, ArkThorAPI, and three containers of ArkThorCore engine. You can refer to the screenshot below to see the containers in action. Screenshot from 2023-06-25 18-48-13

Containers:
- ArkthorUI (1 container): This container is for the user interface of the ArkThor application.
- ArkThorAPI (1 container): This container is for the API component of the ArkThor application.
- ArkThorCore (3 containers): These containers represent instances of the ArkThorCore engine.
- rabitmq (1 container): This container is for the communication between ArkThor API and ArkThor core engine.
Network:
- nwarkthor: This is the name of the network being created.
- Driver type "bridge": This indicates that the network is of type "bridge," which allows containers within the network to communicate with each other.

3. Access ArkThor

ArkThor UI
In the browser type http://localhost:24297 and hit enter

When you access the ArkThor Home page below, you'll find that ArkThor is up and running, fully prepared to perform comprehensive analysis of PCAP files. To initiate an analysis, simply navigate to the dashboard and effortlessly drag and drop any archived PCAP file into the designated upload section. Our advanced analysis engine will promptly process the file, providing you with valuable insights and detailed analysis of the captured network
ArkThor API
http://localhost:33900
RabbitMQ
http://locahost:15672

Modify ArkThor Core Engine Configuration

Open ArkThor Home WebPage (http://localhost:24297) -> ->On Left Navigation Pane Under MISCELLANEOUS, Select "View/Edit Core Config" -> Update the configuration setting as appropriate -> Click on Save button

Manually restart ArkThor Core Engine after the changes
docker-compose -f DockerCompose.yml restart arkthorcore

Running ArkThor Product(with SSL/TLS)

Following section outlines the steps to run the ArkThor product with SSL (Secure Sockets Layer) configuration and Nginx as a reverse proxy. A self-signed certificate is used for SSL, and the necessary files and configurations are provided in the ArkThor repository. The section also guides the process of updating the host entry for a custom URL and accessing the ArkThor product securely.

To ensure secure communication between clients and the ArkThor system, SSL encryption is implemented using a self-signed certificate. Additionally, Nginx is utilized as a reverse proxy to efficiently manage incoming requests.

Prerequisites:

Docker is installed and configured on the target system.
The ArkThor repository is accessible.

Setting up SSL and Nginx Reverse Proxy:

Downloading the Required Files:

Retrieve the necessary files from the ArkThor repository by executing the following command,

curl -LO https://github.com/JawedCIA/ArkThor/raw/master/Arkthor/Docker/Linux/arkthorDockerCompose.zip

Unzip the downloaded zip file,
unzip arkthorDockerCompose.zip -d .

The downloaded ZIP file includes:

DockerCompose-nginx-https.yml: Docker Compose configuration file for deploying the ArkThor system with Nginx and SSL.
Cert folder: Contains the self-signed certificates required for SSL encryption.
nginx.conf: Nginx configuration file for TLS (Transport Layer Security).

Running Docker Compose:

Deploy the ArkThor containers using Docker Compose by running the following command:
docker-compose -f DockerCompose-nginx-https.yml up -d

This command will bring up the required containers and establish the necessary network connections.

[Optional]Configuring Host Entries

To access the ArkThor product using custom URLs(as we are using self signed cert), update the host entries on the system. Open the "/etc/hosts" file with administrative privileges and add the following entries:
127.0.0.1 arkthor.local
127.0.0.1 api.arkthor.local
127.0.0.1 rabbitmq.arkthor.local

These entries map the custom URLs to the loopback address on the local machine.

Accessing the ArkThor Product:

Once the setup is complete, the ArkThor product can be accessed securely using the following URL:
https://arkthor.local

Please note that as the self-signed certificate is being used, web browsers may display a warning. Accept the warning to proceed and continue using the ArkThor product securely.

Deep Dive into DockerCompose.yml file

The above snippet defines a service named arkthorui that runs the ArkThor user interface container.
It pulls the latest image from the arkthor/arkthor-ui repository.
The container will restart automatically if it fails.
It depends on the arkthorapi service and maps port 80 inside the container to port 24297 on the host machine and
It connects to the nwarkthor network.

The arkthorapi service runs the ArkThor API container.
It uses the latest image from the arkthor/arkthor-api repository.
The container restarts automatically if it fails.
It depends on the rabbitmq service.
Port 80 inside the container is mapped to port 33900 on the host machine.
It also mounts two volumes, AnalysesFiles and Database, to specific paths inside the container
- AnalysesFiles: Uploaded PCAP files will get recreated under this folder so that CORE Engine can access this mapped volume for Analysis.
- Database: This mount volume include ArkThor SQLite (ArkThor.db) to persist data
It connects to the nwarkthor network.

The arkthorcore service runs the ArkThor Core engine container.
It uses the latest image from the arkthor/arkthor-core repository.
The container restarts automatically if it fails.
It is deployed with three replicas.
It depends on both the arkthorapi and rabbitmq services.
It mounts ~~three~~ One volume, AnalysesFiles, ~~ArkThorRule~~ , and ~~ArkthorCoreConfig~~, to specific paths inside the container.
- AnalysesFiles: Uploaded PCAP files will get recreated under this folder so that CORE Engine can access this mapped volume for Analysis
  * ArkThorRule: This mounted volume will contents all Analysis Threat Fox Rule files and it will shared by all instances of arkthorcore
  containers.
  * ArkthorCoreConfig: This mount volume include configuration file(config.json) for arkthor core engine, Users have the flexibility to customize configuration values according to their requirements in order to exert control over the core engine. To ensure that the modifications made in the configuration file are reflected, it is necessary to update the configuration of the arkthorcore service accordingly.
  Use below command to restart ArkthorCore service after modification/update of config.json.
It connects to the nwarkthor network.

The rabbitmq service runs the RabbitMQ container with version 3.12 and includes the management plugin.
The container restarts automatically if it fails and is assigned the container name "rabbitmq."
Port 15672 inside the container is mapped to port 15672 on the host machine.
It connects to the nwarkthor network.

The above section defines the volumes and network for the services. Four volumes are defined: AnalysesFiles, Database, ArkThorRule, and ArkthorCoreConfig. These volumes are used to persist data or share files between the host machine and containers.
The nwarkthor network is created with the name "nwarkthor" to allow communication between the containers.

Logging

To view the log run following command as needed

All logs
docker-compose -f DockerCompose.yml logs
ArkThor Core Engine logs
docker-compose -f DockerCompose.yml logs arkthorcore
ArkThor API Engine logs
docker-compose -f DockerCompose.yml logs arkthorapi
ArkThor Core UI logs
docker-compose -f DockerCompose.yml logs arkthorui
RabbitMQ logs
docker-compose -f DockerCompose.yml logs rabbitmq

Docker Images Registry

The ArkThor tool engines can be freely accessed and utilized as a product through Docker Hub. They are available at the provided URL, enabling users to leverage the capabilities of ArkThor for their needs.

https://hub.docker.com/r/arkthor/arkthor-ui
https://hub.docker.com/r/arkthor/arkthor-api
https://hub.docker.com/r/arkthor/arkthor-core

Deep Dive into ArkThor Core Engine Configuration file (config.json)

"watcher": This section configures the file watcher functionality.
- "watch-folder": Specifies the folder where ArkThor will monitor for new files to analyze. In this case, it is set to "UploadedFiles".
- "watch-delay": Defines the delay (in seconds) between each check for new files.
"debugmode": When set to "true", enables the debug mode for ArkThor, allowing for more detailed logging and debugging information.
"deleteprocessed": If set to "true", ArkThor will automatically delete processed files after analysis.
"arkthor": This section contains configuration options related to ArkThor's integration with other components.
- "usearkthorapi": When set to "true", ArkThor utilizes the ArkThor API for specific functionalities.
- "userabbitmq": If set to "true", ArkThor employs RabbitMQ for message queueing and communication.
- "rabbitmqhost": Specifies the hostname of the RabbitMQ server, in this case, "rabbitmq".
- "apibaseurl": Defines the base URL of the ArkThor API, set to "http://arkthorapi/".
"update_ip2asn": When set to "true", ArkThor will update the IP-to-ASN database.
"filename_ip2asn": Specifies the name of the IP-to-ASN database file, set to "ipasn.sqlite3".
"multithreaded_rules_processing": If set to "true", ArkThor will utilize multiple threads for processing rules, improving performance.
"run_rules_on_processed_pcap": When set to "true", ArkThor applies rules to already processed PCAP files.
"threatfox_rule_update_from_Date" and
"threatfox_rule_update_to_Date": Define the start and end dates for updating rules from ThreatFox, a threat intelligence platform.

These configuration settings allow customization and control over various aspects of ArkThor's behavior and integration with external components, ensuring optimal performance and functionality for the tool.

Refresh and Update Core Engine Module

To Refresh/Update IP2ASN module or/and Fetch Latest ThreatFox arkThor rule please follow below steps. Open ArkThor Home Page->On Left Navigation Pane Under MISCELLANEOUS, Select Refresh Click on corresponding Action button for Component to trigger refresh, as shown in below screenshot Refresh_Component

ArkThor Architect Workflow Diagram

UI - The topmost layer of the ArkThor product
- UI Layer provides end-users with a comprehensive set of information, enabling them to gain valuable insights into the threat categorization results.
- The user interface layer directly interacts with users, transmitting uploaded information to the layer immediately below (APIs) for further processing.
- The user interface layer also retrieves information from the APIs layer and presents it to end-users in both visual and textual formats.

The ArkThor user interface comprises several noteworthy features and functionalities, including:

Dashboard
Analysis Information
Analysis Records
Statistics for Measurement
Real-time status tracking through the ArkThor Board
A subscription service allowing users to receive periodic email notifications with executive summaries of threat categorization, cybersecurity, cyber defense, and new updates available in the ArkThor product.
Allow user to Refresh/Update/Edit Core Engine Module as needed.

API- The Middle Layer

In ArkThor, there are several APIs that have been developed using ASP.NET Core and C#. These APIs allow for the receipt of Pcap files, which are then stored in a database. Additionally, a physical copy of the Pcap file is created and stored on a share location. Finally, the SHA256 of the Pcap file is stored on RabbitMQ for CORE engine.

The APIs are available in several different types, including POST, GET, and PUT. These APIs are designed to work with ArkThor and include functionality such as uploading JSON results, uploading Pcap files, uploading support files, updating status and threat type, retrieving measurements, creating file records, and more.

By leveraging these APIs, users can easily integrate ArkThor into their existing workflows and gain access to its powerful analysis capabilities.

APIs – Message broker

RabbitMQ is a popular open-source message broker that can be used to manage message queues, route messages between applications, and distribute work across multiple systems. It provides a reliable and scalable messaging system that can be integrated with various programming languages and frameworks.

In ArkThor, RabbitMQ is used as a message broker in this project to alert CORE about uploaded pcap file for analysis. However, in future we can use this to manage the communication between different module /components of the ArkThor. For example, the Pcap files can be processed by a component that reads the files and extracts relevant features, and then sends the feature data to another component that applies machine learning algorithms(Future work) for threat categorization. RabbitMQ can be used to manage the message queues between these components, ensuring that messages are delivered reliably and efficiently.

RabbitMQ can also be used to distribute work across multiple systems. For example, the machine learning algorithms(Future work) may require significant computational resources to process large volumes of data. RabbitMQ can be used to distribute the workload across multiple systems, ensuring that the processing is efficient and scalable.

CORE- The Heart of ArkThor product

ArkThor - End to End Workflow

1. User uploads Pcap file to the UI
- The user selects a Pcap file from their local device and uploads it through the UI.
2. UI sends the Pcap file to the API
- Once the user has uploaded the file, the UI sends it to the API for storage and analysis.
- The API provides a RESTful endpoint that accepts Pcap files as input.
3. API saves the Pcap file in SQLLitedatabase and on local drive
- Upon receiving the Pcap file, the API saves it in a shared directory for later use by CORE Engine. This directory can be specified in the configuration file of the API otherwise by default it will save at same location on API under “UploadedFiles” folder.
- The API also stores the file in a SQLLitedatabase table to keep track of all the files that have been analyzed. This table can contain information such as the file name, file size, upload date,UploadedBy, sha256 hash value, and the analysis status, C2 communication Countries, Threat type, Severity and much more data related to uploaded file.
- After saving the Pcap file, the API generates the sha256 hash value of the file and sends it to RabbitMQ as a message. This message serves as a notification to the core model that a new Pcap file is available for analysis.
4. Core (Watcher Module) reads the sha256 from RabbitMQ
- The Watcher module is a separate application or process that runs in the background, continuously listening to the RabbitMQ message queue for new tasks.
5. Core (Watcher Module) picks the Pcap file for analysis
- Upon receiving a new message containing the sha256 hash value, the Watcher module retrieves the corresponding Pcap file from the shared directory by matching the hash values and pass on to SCAPY for threat categorization.
6. Core Engine analyzes the Pcap file for threat categorization
- All necessary checks for validating the pcap are done first, after validation, the pcap is loaded with scapy.
- Scapy runs first in stream mode to extract all the stream HTTP artifacts
- Scapy then runs in packet capture mode to extract UDP artifacts
- Once extracted, all component results are stored in their respective json
- The rule engine is now invoked by the watcher
- Rule engine then parses all the available rules and runs them over the generated json artifacts
- Results are aggregated and is given to aggregator which returns back with the valid threat family formulated by the rules.
- The analysis results are formatted as a JSON object that contains the Pcap file SHA256, the threat type, MITRE att&ck technique, and any other relevant information such as the severity score, the analyzed time, and so on.
7. Core Engine submits the JSON object to the API as well as Status
- Once the analysis is complete, the core engine sends the JSON object containing the analysis results to the API through a RESTful endpoint.
8. The API saves the JSON object in a SQLLite database table for later retrieval based on SHA256 of uploaded file.
9. UI fetches the analysis results from the API and displays the results to the user.
- The UI retrieves the analysis results from the API through a RESTful endpoint by passing SHA256 of file.
- The UI parses the JSON object and displays the analysis results to the user in a user-friendly format, such as a table or a chart.
- The user can interact with the UI to view the analysis results of different Pcap files, filter the results based on different criteria, or export the results to a PDF file and view measurements.

ArkThor - Deep Dive

UI

1. Dashboard

The ArkThor product have a dashboard as the primary interface, providing users with an overview of their measurements, such as the number of files queued, the total number of files submitted for threat categorization, the number of files analyzed by the ArkThor core engine, and the number of distinct threats categorized by the engine.

Additionally, users can search through the internal database for threats and upload files for threat categorization using select file as well as the drag-and-drop feature.

The dashboard will also display the latest top 10 analysis records, each with unique properties, and users can navigate to view more detailed file analysis information.

The file upload process will involve an internal check against criteria such as file extension, size limit, and file signature, followed by passing the file properties to the ArkThor APIs for storage in an internal SQLlite Database and a copy of the file will be created in the drop location and SH256 of the uploaded file will be pushed to RabbitMQ Queue. The core engine will then pick up the file for threat categorization using custom rules as well as machine learning.

Additionally, users can switch the table displaying the latest top 10 files analysis track records to real-time mode for auto-updating every 5 minutes using ON/OFF switch.

Other functionalities of the ArkThor product can be accessed through the left navigation.

2. Left Navigation Pane

The Left Navigation Pane in the ArkThor Tool offers users access to various features provided by the tool. This pane is consistent throughout the ArkThor Tool webpage, allowing users to easily navigate between different features. It provides flexibility to the end user, enabling them to move seamlessly from one feature to another.

Furthermore, users have the option to collapse the left navigation pane for their convenience. This feature allows users to maximize their workspace or focus on specific areas of the tool without the distraction of the navigation pane.

3. Analysis Information

ArkThor users can access a comprehensive analysis report of the analyzed file from various pages, including the Dashboard, Records, or ArkThor Live Tracking Board.

To make it easier to understand the information included in this report, the webpage is divided into three different sections.

The top section displays critical information about the analyzed file, such as its final status, threat category type, and threat severity.

The right section includes informative details such as the SHA-256 of the analyzed file, All Threat Type Identified in Analyzed file by ArkThor Core Engine, the user who submitted the file for analysis, the file upload time, the analysis report completion time by the Core Engine, the RAW analyzed file (which can be downloaded by the user), the final JSON analyzed result (available for user to download locally by selecting), and the ability to select similar threat category files from the internal ArkThor database. Additionally, users can view precise information on the MITRE ATT&CK techniques used by the attacker in C2 communication.

The middle section is divided into three subsections, each containing critical information relevant to the analyzed file. The first subsection, "C2 Communication Flow," displays all the communication dots on a flow diagram from the attacker to the target. The second subsection, "Affected Countries," includes a list of country names affected by the output threat category, with respective flags for better presentation. The third subsection displays the names of the countries and their corresponding flags involved in C2 communication.

Furthermore, users can download all this information as a PDF file by selecting the download icon located in the top right corner of the page.

4. Analysis Record

The ArkThor product's Analysis Records feature offers a powerful functionality for users to access their analyzed records.

It can be easily accessed from the left navigation pane of the ArkThor Home page. This feature allows users to search for records based on upload date range, making it simple to locate the records they require. The results are displayed in a convenient tabular format, showcasing essential properties of the records, which helps users quickly identify the relevant records they need. By selecting a record, users can view more detailed information about that specific record. This feature offers a comprehensive view of the analyzed records and streamlines the record retrieval process for maximum efficiency.

5. Statistics - Measurement

In the ArkThor product, we have developed a Statistics or Measurement page that showcases valuable Key Performance Indicators (KPIs) through the use of Pie Charts, Bar Graphs, and Knobs.

Graph

These visual aids help provide insightful and meaningful data to organizations based on the analysis records available in the ArkThor database. By presenting data in a clear and concise manner, users can quickly interpret and identify patterns and trends that can help them make informed decisions. The data available on the Statistics or Measurement page in the ArkThor product is sourced from the ArkThor APIs, but organizations are not limited to only using these visualizations.

If preferred, organizations can utilize other available tools such as Kibana, Grafana, or other data visualization platforms to analyze the data available in the ArkThor database. The goal of ArkThor is to provide a comprehensive and flexible cybersecurity solution that can integrate with existing systems and tools, and the inclusion of APIs and the ability to export data is designed to facilitate this flexibility.

Therefore, organizations can choose the best approach for their unique needs and use the data as they see fit.

6. Track Live Status (ArkThor Board)

The ArkThor product also features a dedicated page called the "ArkThor Board," which provides users with valuable insights into the analysis of files using the ArkThor system. This feature was developed with the idea that organizations may want to display all relevant information on a big screen for easy viewing and monitoring. The ArkThor Board is designed to show analysis information in real-time, with automatic refresh every 180 seconds (configurable).

To ensure the best visual appearance and usability, the ArkThor Board includes various indicators and visualizations that make it easy for users to quickly interpret and understand the data being presented, whether on a big screen or a mobile device.

The inclusion of these features enhances the overall value and usefulness of the ArkThor product, allowing organizations to stay on top of their cybersecurity posture with ease and efficiency.

Additionally, The ArkThor Board also includes information on the current state of different ArkThor Engine/modules. This information helps users to understand whether the system is online or offline and whether any specific modules require attention.

6.1 ArkThor Board Card Notation

7. Subscribe

The Subscribe feature in ArkThor product allows users to receive periodic email notifications that contain an executive summary of threat categorization, cyber security, and cyber defense. This feature provides users with valuable information about the current state of cyber threats and helps them stay up-to-date with the latest developments in the field.

By subscribing to this feature, users can ensure that they receive timely and relevant information about the latest threats and vulnerabilities, as well as updates to the ArkThor product itself. The email notifications contain a concise summary of the most important information, making it easy for users to quickly scan and digest the information without having to spend a lot of time reading lengthy reports.

Users can easily manage their subscription preferences, including the frequency and content of the email notifications they receive. They can choose to receive notifications daily, weekly, or monthly, depending on their preferences and the level of information they require.

The Subscribe feature is a valuable tool for anyone who wants to stay informed about the latest developments in cyber security and cyber defense. Whether you are an IT professional, a security analyst, or a business owner, this feature provides you with the information you need to protect your systems and stay ahead of emerging threats.

8. Core Control- Refresh

It is accessible under MISCELLANEOUS -> Core Control -> Refresh

Users can utilize the ArkThor feature to update the IP2ASN Database of the Core Engine with the most recent IP address information available from the open-source database located at https://iptoasn.com/.
Additionally, users can employ this feature to retrieve the latest IOC database from threatfox MISP. By simply clicking the refresh button, ArkThor will obtain the most up-to-date IOC details from threatfox and convert them into arkthorule, which will be utilized during file analysis.

9. Core Control- View/Edit Core Config

We have introduced the "config.json" file in the CORE engine to enable users to have control over its configuration. This file allows users to customize the behavior of the CORE engine, such as running it as a standalone tool or converting threatfox OIC to ArkThor rules within a specified timeframe, among other functionalities.

You can find detailed explanations about these features in the "Deep Dive into ArkThor Core Engine Configuration File" section. To access this feature, navigate to the left navigation pane, go to MISCELLANEOUS, select Core Control, and then choose View/Edit Core Config. After making the necessary edits or updates, click the SAVE button to apply the changes to the CORE engine.

Note: If you have modified the values of the "threatfox_rule_update_from_date" or/and "threatfox_rule_update_from_date" keys, please ensure to select the "refresh ThreatFox Rule" component in the previous section to download or update the rules.

For any other changes in the key values, it is recommended to restart the Core Engine service using Docker Compose.

APIs

APIs Swagger is accessible in browser: http://localhost:33900/swagger/index.html