Malcolm v2.0.3

This release includes a minor update to the LDAP authentication feature:

• Now, for encrypted connections (whether using StartTLS or LDAPS), Malcolm will require and verify certificates when one or more trusted CA certificate files are placed in the nginx/ca-trust/ directory. Otherwise, any certificate presented by the domain server will be accepted (which was the default behavior for prior versions).

idaholab/Malcolm@v2.0.2...v2.0.3

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on Github, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v2.0.2

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on Github, but may be downloaded from https://malcolm.fyi/download/.

This is a patch release to fix a couple of bugs:

• regression: LDAP authentication no longer handles group membership correctly (#126)
• bro-xor-exe-plugin doesn't do anything (#122, backported from the development 2.1.x branch)

idaholab/Malcolm@v2.0.1...v2.0.2

Malcolm v2.0.1

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on Github, but may be downloaded from https://malcolm.fyi/download/.

This is security patch release which is identical to v2.0.0 except that Zeek has been bumped from v3.0.3 to v3.0.5 to address a security vulnerability (see Malcolm issue #123, and Zeek commit zeek/zeek@bb3250c).

idaholab/Malcolm@v2.0.0...v2.0.1

Malcolm v2.0.0

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on Github, but may be downloaded from https://malcolm.fyi/download/.

This is a major release containing the following new features:

idaholab/Malcolm@v1.8.1...v2.0.0

Network map baseline comparison (issue #2)
Improving on Moloch’s connections view, you can now compare current logical network topology against a
previous time frame, using any of Malcolm’s 900+ fields as references for the graph’s source and destination nodes.
Network changes are easily visualized with icons for new ( ✨ ) and removed (🚫 ) nodes. The graph of connections can be switched on the fly between all nodes, actual nodes (i.e., nodes in the specified query time frame), baseline nodes (i.e., nodes in the specified and baseline query time frames), new nodes only and baseline nodes only.
This feature makes it easy to answer questions like:
• “Are there any hosts in my network this week that didn’t exist last month?”
• “Are hosts in my OT network making any new DNS queries compared to last quarter?”
• “Does my network contain any hardware from new vendors not accounted for the last time inventory was
taken?”
This connections report can be accessed visually in the web browser (see screenshot) or programatically via REST API.

Security overview dashboards
Two new “security overview” Kibana dashboards have been created to bring potential network security issues to
the forefront for IT and OT networks:
• Security Overview (issue #108)
◦ Zeek notices by category
◦ AV signatures triggered by files carved from network traffic
◦ Clear-text transmission of passwords
◦ Outdated/insecure application protocols (e.g., TLSv1.0, SMBv1)
◦ Inbound external traffic by country (i.e., traffic where the source is a publicly routable IP and the
destination is an internal/private IP)
◦ Outbound internal traffic by country (i.e., traffic where the source is an internal/private IP and the
destination is a publicly routable IP)
◦ Summary of file types observed in file downloads/transfers
◦ External remote access over time (i.e., use of “remote access” protocols such as SSH, RDP, VNC, etc.
where either end of the connection is a publicly routable IP address)
◦ DNS queries by randomness (for identifying domain generation algorithms (DGA) used by some
malware)
• ICS/IoT Security Overview (issue #109)
◦ Log count by ICS/IoT protocol
◦ Traffic over time by ICS/IoT protocol
◦ ICS/IoT external traffic (i.e., any use of ICS/IoT protocols where either end of the connection is a publicly
routable IP address)
◦ ICS/IoT action summary
◦ Non-ICS/IoT protocols observed (for identifying IT protocols in OT networks)
◦ Source and destination IP summaries for ICS/IoT traffic
◦ File types by transport

Character frequency/entropy analysis (issue #107)
Malcolm can now optionally employ character frequency analysis to detect domain generation algorithm (DGA) hostnames often used by malware. Currently Malcolm employs this technique on DNS queries and SSL certificate servers. This makes it easier to find suspect domains (e.g., fqoxibdvbycnsappxc.nu) vs. common ones (e.g., example.org).

User interface for defining host and subnet name assignment (see issue #118)

Track user access to Malcolm web interfaces
All access to Malcolm’s web interfaces (e.g., Moloch, Kibana, PCAP upload, etc.) requires authentication by a valid account. These accesses to Malcolm’s own interfaces can now be logged and viewed in Kibana dashboards built for that purpose.

ISO (live USB and installed) improvements
Both Malcolm and Hedgehog Linux can be installed using a standard ISO file image on systems supporting UEFI
boot. Hedgehog Linux can also be run in live USB mode, effectively turning any commodity hardware into an ad-
hoc network sensor. Improvements have been made to the base OS, including:
• improved hardening for both Malcolm and Hedgehog Linux
• installations should now detect virtual environments (VMWare and VirtualBox) and install the correct guest mode drivers for changing video resolution on the fly, shared folders, etc.
• many more minor fixes and improvements

Component version updates
Updated the following components to their latest stable released versions for security updates, bug fixes,
performance improvements and new features
• Elastic stack (Elasticsearch, Kibana, Logstash and Beats) 7.6.2
• Moloch 2.2.3
• Zeek 3.0.3

Miscellaneous fixes and improvements
• Fixed cross-platform compatibility of control scripts (#103)
• Fixed offline region maps (#112 and #84)
• Fixed intermittent failure when uploading very large PCAP files (#101)
• Fixed /upload URL incorrect redirect without trailing slash (#104)
• Fixed MANAGE_PCAP_FILES not working (#114)
• and more

Malcolm v2.0.0-pre1

This is a major release containing the following new features:

idaholab/Malcolm@v1.8.1...v2.0.0-pre1

Network map baseline comparison (issue #2)
Improving on Moloch’s connections view, you can now compare current logical network topology against a
previous time frame, using any of Malcolm’s 900+ fields as references for the graph’s source and destination nodes.
Network changes are easily visualized with icons for new ( ✨ ) and removed (🚫 ) nodes. The graph of connections can be switched on the fly between all nodes, actual nodes (i.e., nodes in the specified query time frame), baseline nodes (i.e., nodes in the specified and baseline query time frames), new nodes only and baseline nodes only.
This feature makes it easy to answer questions like:
• “Are there any hosts in my network this week that didn’t exist last month?”
• “Are hosts in my OT network making any new DNS queries compared to last quarter?”
• “Does my network contain any hardware from new vendors not accounted for the last time inventory was
taken?”
This connections report can be accessed visually in the web browser (see screenshot) or programatically via REST API.

Security overview dashboards
Two new “security overview” Kibana dashboards have been created to bring potential network security issues to
the forefront for IT and OT networks:
• Security Overview (issue #108)
◦ Zeek notices by category
◦ AV signatures triggered by files carved from network traffic
◦ Clear-text transmission of passwords
◦ Outdated/insecure application protocols (e.g., TLSv1.0, SMBv1)
◦ Inbound external traffic by country (i.e., traffic where the source is a publicly routable IP and the
destination is an internal/private IP)
◦ Outbound internal traffic by country (i.e., traffic where the source is an internal/private IP and the
destination is a publicly routable IP)
◦ Summary of file types observed in file downloads/transfers
◦ External remote access over time (i.e., use of “remote access” protocols such as SSH, RDP, VNC, etc.
where either end of the connection is a publicly routable IP address)
◦ DNS queries by randomness (for identifying domain generation algorithms (DGA) used by some
malware)
• ICS/IoT Security Overview (issue #109)
◦ Log count by ICS/IoT protocol
◦ Traffic over time by ICS/IoT protocol
◦ ICS/IoT external traffic (i.e., any use of ICS/IoT protocols where either end of the connection is a publicly
routable IP address)
◦ ICS/IoT action summary
◦ Non-ICS/IoT protocols observed (for identifying IT protocols in OT networks)
◦ Source and destination IP summaries for ICS/IoT traffic
◦ File types by transport

Character frequency/entropy analysis (issue #107)
Malcolm can now optionally employ character frequency analysis to detect domain generation algorithm (DGA) hostnames often used by malware. Currently Malcolm employs this technique on DNS queries and SSL certificate servers. This makes it easier to find suspect domains (e.g., fqoxibdvbycnsappxc.nu) vs. common ones (e.g., example.org).

User interface for defining host and subnet name assignment (see issue #118)

Track user access to Malcolm web interfaces
All access to Malcolm’s web interfaces (e.g., Moloch, Kibana, PCAP upload, etc.) requires authentication by a valid account. These accesses to Malcolm’s own interfaces can now be logged and viewed in Kibana dashboards built for that purpose.

ISO (live USB and installed) improvements
Both Malcolm and Hedgehog Linux can be installed using a standard ISO file image on systems supporting UEFI
boot. Hedgehog Linux can also be run in live USB mode, effectively turning any commodity hardware into an ad-
hoc network sensor. Improvements have been made to the base OS, including:
• improved hardening for both Malcolm and Hedgehog Linux
• installations should now detect virtual environments (VMWare and VirtualBox) and install the correct guest mode drivers for changing video resolution on the fly, shared folders, etc.
• many more minor fixes and improvements

Component version updates
Updated the following components to their latest stable released versions for security updates, bug fixes,
performance improvements and new features
• Elastic stack (Elasticsearch, Kibana, Logstash and Beats) 7.6.1
• Moloch 2.2.3
• Zeek 3.0.3

Miscellaneous fixes and improvements
• Fixed cross-platform compatibility of control scripts (#103)
• Fixed offline region maps (#112 and #84)
• Fixed intermittent failure when uploading very large PCAP files (#101)
• Fixed /upload URL incorrect redirect without trailing slash (#104)
• Fixed MANAGE_PCAP_FILES not working (#114)
• and more

Malcolm v1.8.1

Malcolm v1.8.1

idaholab/Malcolm@v1.8.0...v1.8.1

• Update to Elastic stack 7.5.1 (and fixed hopefully all the compatibility issues that arose)
• Moloch version 2.1.2
• fix issues with initial build and download of maxmind geoip database files
• documentation updates and fixes
• some improvements to help with higher bitrate capture (increasing ring buffer sizes)
• improvements to ISO for Malcolm (aggregator) and Hedgehog (sensor)

Malcolm v1.8.0

Malcolm v1.8.0

idaholab/Malcolm@v1.7.2...v1.8.0

• build scripts for network sensor OS installable and live ISO, Hedgehog Linux
• authentication against an LDAP server (tested against Microsoft Active Directory Domain Services in Windows Server 2016 and OpenLDAP, each with StartTLS, LDAPS, and unencrypted connections) (issue #77)
• minor improvements to file carving and Malcolm/Hedgehog ISO configuration
• reduced noise of auditd messages sent from Hedgehog ISO installation
• bump Moloch to 2.1.1
• bump Zeek to 3.0.1

Malcolm v1.7.2

Malcolm v1.7.2

idaholab/Malcolm@v1.7.1a...v1.7.2

• Fixes issue #86
• adds some sample configuration for sensor/forwarder usage

Malcolm v1.7.1a

Malcolm v1.7.1a

idaholab/Malcolm@v1.7.0...v1.7.1a

• redesign PCAP processing pipeline (pull request #81, issue #80) so that there is one service that watches the /data/pcap/processed directory and publishes to a ØMQ topic), then other services can subscribe to that topic and do what they want with the PCAP information they receive. This will make it much easier to add future PCAP processors, and also increases parallel-ness of the code

• move common Logstash enrichments to a separate pipeline (pull request #81, issue #78). I've made the pipelines used for processing Logstash events more modular, and I've also made it more extensible by having the startup script dynamically detect and configure new pipelines on the fly. this will make it easier to add new parsers in the future (need to document how to do that in the readme though)

• set opencontainers-compatible labels on docker containers

• fix issue #82, OUI vendor names used by Logstash don't match those used by Moloch

• split moloch container into pcap-monitor, zeek, and moloch containers

• documentation fixex

• dockerfile cleanup

• bump Moloch to 2.1.0 (see changelog and security).

• enable readTruncatedPackets for moloch's config.ini to handle more pcaps

Malcolm v1.7.0

Malcolm v1.7.0

idaholab/Malcolm@v1.6.0...v1.7.0

Malcolm v1.7.0 is a big release, with the following goodness:

• Zeek 3.0
• New parsers/analyzers, complete list:
  • Amazon.com, Inc.'s ICS protocol analyzers
  • Corelight's bro-xor-exe plugin
  • Corelight's community ID flow hashing plugin
  • J-Gras' Bro::AF_Packet plugin
  • Lexi Brent's EternalSafety plugin
  • MITRE Cyber Analytics Repository's Bro/Zeek ATT&CK-Based Analytics (BZAR) script
  • Salesforce's gQUIC analyzer
  • Salesforce's HASSH SSH fingerprinting plugin
  • Salesforce's JA3 TLS fingerprinting plugin
  • SoftwareConsultingEmporium's Bro::LDAP analyzer

• Logstash: use the cidr plugin to assign internal_source, external_source, internal_destination, external_destination tags based on srcIp and dstIp Zeek logs
• ISO installer tweaks
• hardening compliance tweaks
• Dashboards for all new protocols
• Documentation updates
• user account management (htadmin) improvements
• bump Elastic to 6.8.4-oss
• added human-readable names to types created with Moloch WISE
• use ZeroMQ-based approach for file scanning queue