-
-
Notifications
You must be signed in to change notification settings - Fork 144
Database
OWASP Python Honeypot Project currently uses ElasticSearch to store the data in the server where the OWASP Honeypot is running. That means the server where OWASP Honeypot is running should have ElasticSearch installed.
Running the honeypot modules would result in the creation of two databases-
-
ohp_events: for storing event data -
ohp_file_archive: for storing network captured files
The following collections would be created in the database ohp_events:
There is Honeypot events queue which is being maintained for inserting all the honeypot events in the bulk insert as
each bulk insert is faster than instantiating insert for each of the records. The format of the data inserted
in honeypot_events index is:
{
"_index": "honeypot_events",
"_type": "_doc",
"_id": "2ZJZ5XkBXRc7-vE_2DAk",
"_version": 1,
"_seq_no": 20,
"_primary_term": 2,
"_source": {
"ip_dest": "34.107.221.82",
"port_dest": 80,
"ip_src": "192.168.0.107",
"port_src": 60212,
"protocol": "TCP",
"module_name": "http/basic_auth_weak_password",
"machine_name": "stockholm_server_1",
"date": "2021-06-07 12:37:13",
"country_ip_src": "-",
"country_ip_dest": "US"
}
}All the network events data is separated from the honeypot events as they are not harmful to the server running. Network
events can also be used for analysis and hence they are stored in a separate table. The format of data inserted in
the network_events index is:
{
"_index": "network_events",
"_type": "_doc",
"_id": "7JGq3HkBXRc7-vE_J_mk",
"_version": 1,
"_seq_no": 54,
"_primary_term": 2,
"_source": {
"ip_dest": "13.107.42.14",
"port_dest": 443,
"ip_src": "192.168.0.104",
"port_src": 53751,
"protocol": "TCP",
"machine_name": "stockholm_server_1",
"date": "2021-06-05 20:23:26",
"country_ip_src": "-",
"country_ip_dest": "CZ"
}
}
There is a special type of event which stores credentials that are obtained from the modules like ssh/strong_password,
ftp/strong_password, http/basic_auth_strong_password and smtp/strong_password. The format of data inserted in
the credential_events collection is:
{
"_index": "credential_events",
"_type": "_doc",
"_id": "bCec-3kBivfr1s5MRArb",
"_version": 1,
"_seq_no": 0,
"_primary_term": 4,
"_source": {
"ip_src": "88.99.11.22",
"module_name": "http/basic_auth_weak_password",
"date": "2021-06-11 20:36:33",
"username": "admin",
"password": "password",
"machine_name": "stockholm_server_1",
"country_ip_src": "DE"
}
}
These are different type of events which is keeping track of the file path, modified by the hacker on the system as it
is very easy to get into the system for weak password modules. Hence the file change events are integrated into modules
like ssh/weak_password and ftp/weak_password. The format of data inserted in file_change_events index is:
{
"_index": "file_change_events",
"_type": "_doc",
"_id": "5f1-803c26c76f3c11bd",
"_version": 1,
"_seq_no": 10,
"_primary_term": 41,
"_source": {
"file_path": "/root/OWASP-Honeypot/tmp/ohp_ssh_weak_container/.bash_history",
"module_name": "ssh/weak_password",
"date": "2020-07-23 00:46:27",
"status": "modified",
"machine_name": "stockholm_server_1",
"is_directory": false
}
}
These are the events used to store data collected from modules like smtp and ics. The format of data inserted in
the data_events index is:
{
"_index": "data_events",
"_type": "_doc",
"_id": "hCeo-3kBivfr1s5MVgrV",
"_version": 1,
"_seq_no": 0,
"_primary_term": 4,
"_source": {
"ip_src": "55.66.77.88",
"module_name": "ics/veeder_root_guardian_ast",
"date": "2021-06-11 20:49:44",
"data": {
"message": "test data"
},
"machine_name": "stockholm_server_1",
"country_ip_src": "US"
}
}
The file archive database is used to store the network captured files. The format of data inserted in ohp_file_archive index is:
{
"_index": "ohp_file_archive",
"_type": "_doc",
"_id": "I8I-NXoB9paUMAsLo2D3",
"_version": 1,
"_seq_no": 0,
"_primary_term": 6,
"_source": {
"content": "Cg0Nx98dNX6Aa1ag8bSEp6/zlBARJ8Nbx98dNX6Aa1ag8bSEp6/zlBARJ8NbXeAXeA ",
"date": "2021-06-23 01:10:26",
"filename": "captured-traffic-1624390826.pcap",
"machine_name": "stockholm_server_1",
"md5": "97d7228b35e217505a3cd358cefc2d63",
"splitTimeout": 3600
}
}In the code, we have defined data-types using classes for storing data specific to each of the databases specified above. The objects of these classes are directly converted to a dictionary and inserted into the database.
These datatypes can be found in the database/datatypes.py file.
class HoneypotEvent:
"""
Object to store Honeypot Event Parameters.
Attributes:
ip_dest: Destination IP address (machine)
port_dest: Destination port (machine)
ip_src: Source IP address
port_src: Source port
date: Date and time of the event
module_name: Module name ran on the port
machine_name: Real machine name
country_ip_src: Country of source IP Address
country_ip_dest: Country of destination IP Address
"""
def __init__(self, ip_dest, port_dest, ip_src,
port_src, protocol, module_name, machine_name):
self.ip_dest = ip_dest
self.port_dest = port_dest
self.ip_src = ip_src
self.port_src = port_src
self.protocol = protocol
self.module_name = module_name
self.machine_name = machine_name
self.date = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
self.country_ip_src = None
self.country_ip_dest = Noneclass NetworkEvent:
"""
Object to store Network Event Parameters
Attributes:
ip_dest: Destination IP address (machine)
port_dest: Destination port (machine)
ip_src: Source IP address
port_src: Source port
date: Date and time of the event
protocol: Protocol type of the packet
machine_name: Real machine name
country_ip_src: Country of source IP Address
country_ip_dest: Country of destination IP Address
"""
def __init__(
self, ip_dest, port_dest,
ip_src, port_src, protocol, machine_name):
self.ip_dest = ip_dest
self.port_dest = port_dest
self.ip_src = ip_src
self.port_src = port_src
self.protocol = protocol
self.machine_name = machine_name
self.date = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
self.country_ip_src = None
self.country_ip_dest = Noneclass CredentialEvent:
"""
Object to store Credential Event Parameters
Attributes:
ip_src: Client ip used for connecting to the module
module_name: Which module was accessed
date: Date and time of the event
username: Username tried for connecting to modules
password: Password tried for connecting to modules
machine_name: Real machine name
country_ip_src: Country corresponding to the IP Address
"""
def __init__(self, ip_src, module_name, date, username, password):
self.ip_src = ip_src
self.module_name = module_name
self.date = date
self.username = username
self.password = password
self.machine_name = None
self.country_ip_src = Noneclass EventData:
"""
Object to store Honeypot Event Data collected from
modules such as ICS Module.
Attributes:
ip: Client IP used for putting the data
date: Date and time of the event
module_name: Module client accessed by the client
data: Data which is obtained from the client
country_ip_src: Country corresponding to the IP Address
"""
def __init__(self, ip, module_name, date, data):
self.ip_src = ip
self.module_name = module_name
self.date = date
self.data = data
self.machine_name = None
self.country_ip_src = Noneclass FileEventsData:
"""
Object to store file changes events data collected from
modules such as ftp/ssh weak_password module.
Attributes:
file_path : the path of the file which is changed
status: status of the file would be added/modified/deleted
module_name : on which module client accessed
date : datetime of the event
is_directory: is directory?
"""
def __init__(self, file_path, status, module_name, date, is_directory):
self.file_path = file_path
self.module_name = module_name
self.date = date
self.status = status
self.is_directory = is_directory
self.machine_name = None
self.file_content = Noneclass FileArchive:
"""
Object to store details about captured network traffic files
to be stored in the File Archive
Attributes:
file_path: the path of the PCAP file
date: generation date and time of the file
split_timeout: timeout value to be used to split PCAP files
"""
def __init__(self, file_path, date, split_timeout):
self.file_path = file_path
self.date = date
self.split_timeout = split_timeout
self.md5 = None
self.file_content = NoneThe database/connector.py file provides the connector functions required to insert the data into the databases.
Inserting to honeypot_event and network events databases take place in 2 steps to reduce I/O operations to the database
and improve performance -
- Adding a record to queue.
- Pushing all records in the queue to the database.
Add the record to the honeypot event queue.
-
Function Name:
insert_to_honeypot_events_queue -
Arguments:
- honeypot_event: Object of HoneypotEvent class with event parameters
- honeypot_events_queue: Multiprocessing queue which stores the list of honeypot_events in dict format.
-
Example:
honeypot_event = HoneypotEvent( ip_dest="11.22.33.44", port_dest=80, ip_src="12.23.34.45", port_src=1010, protocol='TCP', module_name="http/basic_auth_weak_password", machine_name="stockholm_server_1" ) honeypot_events_queue = Queue() # Insert events to queue insert_to_honeypot_events_queue(honeypot_event, honeypot_events_queue)
Add the record to the network event queue.
-
Function Name:
insert_to_network_events_queue -
Arguments:
- network_event: Object of NetworkEvent Class with event parameters
- network_events_queue: Multiprocessing queue which stores the list of network_events in dict format.
-
Example:
network_event = NetworkEvent( ip_dest="13.14.15.16", port_dest=8090, ip_src="22.33.44.55", port_src=1100, protocol='UDP', machine_name="stockholm_server_1" ) network_events_queue = Queue() insert_to_network_events_queue(network_event, network_events_queue)
Insert all the records in both honeypot_events_queue and network_events_queue to the respective collections.
The push_events_to_database_from_thread function runs in a separate thread, calling this function inside a while loop
to ensure regular data push to the collection.
-
Function Name:
push_events_queues_to_database -
Arguments:
- honeypot_events_queue: Multiprocessing queue which stores the list of honeypot_events in dict format.
- network_events_queue: Multiprocessing queue which stores the list of network_events in dict format.
-
Example:
push_events_queues_to_database(honeypot_events_queue, network_events_queue)
Credential events are directly added to the collection.
-
Function Name:
insert_to_credential_events_collection -
Arguments:
- credential_event: Object of CredentialEvent Class with event parameters
-
Example:
credential_event = CredentialEvent( ip_src="88.99.11.22", username="admin", password="password", module_name="http/basic_auth_weak_password", date=datetime.now().strftime("%Y-%m-%d %H:%M:%S") ) insert_to_credential_events_collection(credential_event)
Inserts file change events which are obtained from ftp/ssh weak_password module. It reads the file contents from the file path in the file_change_event_data and adds it to the data to be stored. File change events are directly added to the collection.
-
Function Name:
insert_to_file_change_events_collection -
Arguments:
- file_change_event_data: Object of FileEventsData Class with event parameters
-
Example:
file_event = FileEventsData( file_path=byte_to_str(event.src_path), status=byte_to_str(event.event_type), module_name=self.module_name, date=now(), is_directory=event.is_directory ) insert_to_file_change_events_collection(file_event)
Insert data collected from module processors of modules such as ICS module and SMTP.
-
Function Name:
insert_to_events_data_collection -
Arguments:
- event_data: Object of EventData Class with ip, module_name, machine_name, date, data
-
Example:
event_data = EventData( ip="55.66.77.88", module_name="ics/veeder_root_guardian_ast", date=datetime.now().strftime("%Y-%m-%d %H:%M:%S"), data={"message": "test data"} ) insert_to_events_data_collection(event_data)
This inserts PCAP files to the file_archive collection using GridFS which splits the file into chunks and stores it.
-
Function Name:
insert_pcap_files_to_collection -
Arguments:
- file_archive: Object of FileArchive Class with file_path, date and split_timeout
-
Example:
file_archive = FileArchive( output_file_path, datetime.now().strftime("%Y-%m-%d %H:%M:%S"), 3600 # this is in seconds ) insert_pcap_files_to_collection(file_archive)