Malcolm v5.0.4

Malcolm v5.0.4 is a patch release with improvements to Zeek detection of CVE-2021-44228 ("Log4Shell" Log4J vulnerability).

v5.0.3...v5.0.4

• build with latest corelight/cve-2021-44228 release

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.0.3

Malcolm v5.0.3 is a patch release with a few minor bug fixes and improvements to Zeek detection of CVE-2021-44228 ("Log4Shell" Log4J vulnerability).

v5.0.2...v5.0.3

• build with latest zeek/spicy-ldap release (dpd-based detection rather than just port-based)
• build with latest corelight/cve-2021-44228 release
• fix idaholab#69 (zeek resists shutdown on sensor during halt/reboot)
• bump OpenSearch to v1.2.2 which has log4j 2.16
• added convenience script for working with GitHub workflow-built images

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.0.2

Malcolm v5.0.2 is a patch release adding HTTP header-based Zeek detection of CVE-2021-44228 ("Log4Shell" Log4J vulnerability).

v5.0.1...v5.0.2

• Added Corelight's Zeek detection script for CVE-2021-44228 ("Log4Shell" Log4J vulnerability)
• move zeek.http.tags field up to top-level tags
• Version bumps
  • Arkime to v3.2.1
  • Alpine (for dashboards-helper, name-map-ui and nginx-proxy Docker containers) to v3.15.0
  • NGINX (for nginx-proxy Docker container) to v1.20.2

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.0.1

Malcolm v5.0.1 is a patch release with minor bug- and security-related fixes.

v5.0.0...v5.0.1

• Security vulnerabilities addressed:

  • mitigations for CVE-2021-44228 (log4shell) idaholab#68

• Bugs addressed:

  • Very large pcaps don't get proccesed idaholab#44
  • pcap files with colon (:) in the name don't process correctly idaholab#2
  • turning off AUTO_TAG feature disables tagging altogether idaholab#12
  • recent debinterfaces release broke configure-interfaces.py idaholab#48
  • opensearch indexes in yellow state idaholab#67
  • arkime capture gives mlockall_init() warning on startup idaholab#66

• Other

  • bumped Arkime from v3.1.1 to v3.2.0
  • bumped OpenSearch to v1.2.1
  • switched from elasticsearch to opensearch python client libraries
  • write contributor's guide for source code contributions/modifications idaholab#25
  • handle new fields in ethernet/IP logs (cisagov/icsnpp-enip@c4ae505)
  • use more recognizable dashboards logo for OpenSearch dashboards launcher in Malcolm ISO
  • include patches used to build Arkime Dockerfile when building Arkime for hedgehog as well
  • build Zeek spicy analyzers from their various repos rather than the zeek/spicy-analyzer meta-repo

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.0.0

Malcolm v5.0.0 is a major release which addresses idaholab#54, transition from ElasticSearch to OpenSearch

v4.0.1...v5.0.0

Malcolm has switched to the OpenSearch project as the basis of its search and analytics capabilities, mainly for two reasons:

1. Elastic.co's decision to no longer release Elasticsearch and Kibana under an open source license
2. Capabilities available under OpenSearch (and previously under Open Distro for Elasticsearch) that are only available with paid "premium" Elastic.co subscriptions (machine learning anomaly detection, alerting, reporting, etc.)

As the Malcolm project uses semantic versioning when choosing version numbers, this backwards-compatibility breaking change is the reason for bumping the major version number from 4 to 5. It is not recommended to attempt an upgrade from a previous release; a fresh install is required.

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Historical context for the events and reasoning behind this change:

• Elastic announces license change

  • Amazon NOT OK
  • Doubling Down on Open
  • Doubling Down on Open, Part II
  • Elastic License v2
  • FAQ on 2021 License Change
    • Does this mean that Elasticsearch and Kibana are no longer Open Source? Yes. Neither the Elastic License nor SSPL have been approved by the OSI, so to prevent confusion, we no longer refer to Elasticsearch or Kibana as open source.
  • old "open source" tier ("Apache 2.0: Now and always" 🙄) goes away
  • The SSPL is not an open source license

• OpenSearch fork:

  • Stepping up for a truly open source Elasticsearch
  • Introducing OpenSearch
  • Truly Doubling Down on Open Source @ logz.io
  • FAQ

• Third-party blogs, etc.

  • Elasticsearch does not belong to Elastic
  • Elasticearch and Kibana are now business risks
  • Is Elasticsearch no longer open source software?
  • The Implications of Elasticsearch and Kibana License Change
  • Let's talk about the Elastic license change
  • Elastic is going closed-source. Where does that leave MSSPs?

Malcolm v4.0.1

Malcolm v4.0.1 is a point release with the following updates:

v4.0.0...v4.0.1

• Incorporate support for OSPF package analyzer and add relevant visualizations
• Fix for building Zeek Spicy analyzer plugins as they are being split out into separate repositories rather than just the Zeek spicy-analyzers repo

This may be the final release of Malcolm prior to the completion of the transition from Elasticsearch to OpenSearch.

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v4.0.0

Malcolm v4.0.0 consists of a major restructuring of the underlying data schema used to represent Zeek logs (and, going forward, logs from other data sources) in the Elasticsearch data store. As the Malcolm project uses semantic versioning when choosing version numbers, this backwards-compatibility breaking change is the reason for bumping the major version number from 3 to 4 despite no significant new functionality being introduced.

The details of the drivers behind this change can be found at idaholab#64 and idaholab#16. This change, though somewhat painful, will make it easier to integrate more data sources into Malcolm in the future and potentially makes Malcolm's network session data more compatible with other tools that use the Elastic Common Schema.

v3.4.0...v4.0.0

BREAKING CHANGES:

• as many field names have changed, custom saved dashboards and/or bookmarks to Kibana or Arkime visualizations may need to be adjusted accordingly
• old network session data (stored in the sessions2-* indices in Elasticsearch) will not be visible (as the indices are now named arkime-sessions3-*)

A fresh install of Malcolm is recommended with this release. Upgrading from previous versions of Malcolm to v4.0.0+ is not suggested.

Changes:

• added GitHub workflow files which contain instructions for GitHub to build the docker images and sensor and Malcolm installer ISOs.
• moved many fields that were named zeek-specific to generic ECS-specified (or at least "ECS-inspired") field names, updating related parsing code and dashboard definitions
• changed Zeek-specific field naming schema (e.g., zeek_foo.bar becomes zeek.foo.bar)
• added Corelight's Microsoft Excel privilege escalation detection (CVE-2021-42292) plugin
• integrated updates to the LDAP parser which improve the detail given from observed LDAP searches
• improved and genericized the code for mapping MAC addresses to vendor OUIs, replacing the use of logstash-filter-ieee_oui
• updated some Dockerfiles to use Debian 11 "bullseye" instead of Debian 10 "buster"

Malcolm v3.4.0

Malcolm v3.4.0 is a feature release focused on bringing its major underlying components up-to-date with the latest released versions, increasing stability, improving performance and adding new features.

v3.3.1...v3.4.0

• Component version updates
  • Arkime v3.3.1 (from v2.7.1)
  • Zeek v4.1.1 (from v4.0.4)
    • Zeek v4.1 Feature Release
  • Spicy v1.3.0 (from v1.2.1)
  • Yara v4.1.3 (from v4.1.2)
  • Capa v3.0.3 (from v3.0.2)
  • Debian v10 to v11 (for ISO images)
• Added GitHub actions for building the Malcolm Docker images on GitHub and pushing them to GHCR
• Moved common Logstash Ruby code to file-based scripting
• Use standard stunnel package in NGINX proxy container rather than building from source
• Switched from CLANG to GCC build toolchain for Zeek and Spicy plugins
• Replaced LXDE desktop environment with XFCE (for ISO images)
• Renamed various fields to align with Arkime's gradual adoption of the Elastic Common Schema
• Added parser support and dashboard for the STUN (Session Traversal Utilities for NAT) protocol
• Further improved capabilities for tagging ICS traffic
  • Logs from known ICS protocols how have ics added to the tags field
  • Logs identified by "ICS best guess" lookups now have ics_best_guess added to the tags field
  • "ICS best guess" lookups have been augmented with a MAC address lookup table of ICS hardware vendors
  • ICS-related overview dashboards have been updated accordingly

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v3.3.1

v3.3.1 is a minor Malcolm release with the following updates:

v3.3.0...v3.3.1

• Incorporate Corelight's "OMIGOD" (CVE-2021-38647) plugin
• Bump capa to v3.0.2 which now includes ELF scanning capabilities
• Bump zeek to v4.0.4
• minor fix on race condition creating default anomaly detectors
• minor tweak to build.sh script for building docker images

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v3.3.0

List of changes in Malcolm v3.3.0:

v3.2.1...v3.3.0

Version 3.3.0 is a feature release of Malcolm.

• New features

  • Automatically create some broadly useful anomaly detectors when initializing Kibana
    • connection size
    • file transfer MIME type
    • action and result (by application protocol)
  • Configurable event severity scoring (idaholab#19) and new Severity dashboard

• Other changes

  • vagrant-based ISO build can now work with either VirtualBox or libvirt providers
  • change wording of terms such as "master"/"slave" to "client"/"server" as instructed by DHS directive

• Version updates

  • Update base image for Debian-based Docker images from 10 (buster) to 11 (bullseye)
  • Update Yara to 4.1.2
  • Update Capa to 2.0.0
  • Update Spicy to 1.2.1
  • Update remainder of python 2 code to python 3

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.