v2.0.0

StreamAlert Release v2.0.0

New Features

Alert Merging

This release brings new support for merging similar alerts. The rate at which alerts can be triggered is occasionally unpredictable and be a potential pain for analysts down stream. Combing through a massive amount of alerts becomes tiresome, and could easily result in missing the needle in the haystack. We have introduced an alert merging feature that allows rules to dictate how numerous alerts generated by the rule in a given timeframe are merged together into a single, unified alert. This feature has proven to greatly reduce alert overload. See the rules page of the documentation for how to configure alert merging. See also: #612, #640, #642, #666

Rule Baking/Staging

Release 2.0.0 introduces a new feature we’ve appropriately deemed Rule Baking. Often, new rules have not been battle tested and need to be tuned over time to prevent alert fatigue. This feature introduces a staging period for new rules, along with a daily email digest. The digest includes statistics on alerts that have fired for staged rules, allowing analysts to get metrics on new rules before they go into production. As an added bonus, any rule, new or old, can be toggled into and out of staging on-demand without any code changes.

SNS & SQS Alerting Outputs

This release introduces support for sending alerts to AWS SNS (Simple Notification Service) and SQS (Simple Queue Service). SNS is ideal for sending email or text messages when alerts occur, while SQS better enables downstream and reliable alert processing by other tools.

Slack App

StreamAlert now supports reading files from S3 that contain lookup information to be used within rules. These JSON files can be gzip compressed, and will be refreshed every few minutes to ensure the AWS Lambda container always has an up-to-date intelligence set.

Salesforce App

Salesforce is a commonly used CRM service, and in the latest version of their API (at the time of this writing), it supports 42 types of event logs, and auditing/alerting on these events is no simple task. The new StreamAlert App will collect Console, Login, LoginAs, Report and ReportAs event logs and send to StreamAlert for rule processing and alerting.

Carbon Black Hash Banning

Thanks to @fusionrace for adding support for automated banning of potential malicious files via the new Carbon Black alert output. This feature enables rules to specify md5 hashes which should be banned, and the output handles the rest.

Lookup Tables via S3

StreamAlert now supports reading files from S3 that contain lookup information to be used within rules. These JSON files can be gzip compressed, and will be refreshed every few minutes to ensure the AWS Lambda container always has an up-to-date intelligence set.

Log Parsing Features

Occasionally, logs can be a JSON object containing an embedded csv log of interest. This is the case with AWS RDS logs that are captured via CloudWatch Logs log group and forwarded on to another destination with a CloudWatch Logs Subscription. This version of StreamAlert introduces the ability to extract embedded csv from a JSON object and parse it accordingly. It also adds support for encoded JSON within another JSON object. Various other added benefits were introduced as well; see also: #744, #745

To view a list of all of the new features, see here. Many of these features are sparsely documented at the moment, but look out for many updates to documentation in the coming weeks!

Improvements

Terraform Consolidation

A huge thanks to @austinbyers for refactoring the redundant Terraform Lambda code into a centralized module. The consolidation has enabled us to build new Lambda functions faster and reduced the amount of tests we need to write. The new module is easily adaptable and most of StreamAlert’s Lambda functions have already been ported to leverage it. See also: #638

Alert Processor Consolidation

As part of the move toward alert merging, ‘clustered’ alert processors became unnecessary and overly complex. This change removes multiple alert processors (one for each cluster) in favor of a single processor for all alerts. The architecture of StreamAlert has become more intuitive and simplified thanks to this change.

Rule Test Templates

Up until now, writing tests for rules in StreamAlert requires a copy of a 'real' log that will classify properly in the rules engine. This can be a little arduous if your rule only tests one or two conditions. This release introduce new functionality, where a user can supply a partial test event, and the testing framework will ‘intelligently’ fill the rest of the fields in with default data.

Firehose Optimizations

Batches of records being sent to AWS Firehose, for historical data retention, will now only have records that failed to send retried instead of the entire batch. Additionally, the data will only be serialized to JSON once at the start of the sending, instead of on every attempt. The client has also been tuned with connect and read timeouts to reduce latency when sending batches. See also: #649, #736

Threat Intel Improvements

This release introduces the ability to exclude specific indicators of compromise from the Threat Intel lookups. For instance, IP addresses that are known-goods could be omitted when querying the threat intel DynamoDB table. This can speed up rule processing and reduce cost. The ThreatIntel downloader has also been updated to support filtering out bad or unactionable indicators at the source, before they even land in DynamoDB. See also: #684

General Improvements(#725)

• #604 - Speeding up Lambda function deploys by using the -auto-approve flag with terraform.
• #652 - Reducing the amount of API requests made to PagerDuty during PagerDuty incident creation. This change prevents potential throttling by the API.
• #653 - More efficient rule helper functions
• #691 - Backoff (aka retry) decorators are used fairly heavily throughout StreamAlert. This release centralizes the backoff handlers to be reused and reduces the amount of duplicate code & tests.
• #707 - Almost every new component added to StreamAlert needs to load the configuration from disk. This release introduces shared logic for config loading that removes the need to write yet another load_config function.
• #725 - Alert processor status logging is now greatly simplified. Those writing new alert outputs no longer have to reason about success or failure logging.
• #784 - S3 server-side encryption adds a layer of protection to data in S3 buckets, and release 2.0.0 introduces this as the default for StreamAlert.
• #804 - Deduplicating gsuite events, thanks to @stoggi.
• #806 - Updating Athena partition refresh function to use SQS as a source. See also: #808, #810, #815
• #818 - Logger instantiation consolidated to shared module.

To view the complete list of all of the improvements in v2.0.0, including many not mentioned above, see here.

Bug Fixes

• #623 - Python absolute import warning within CloudWatch Logs during Lambda invocations. Fixes: #267
• #657 - Long-standing and sporadic Too many open files bug that was never easily tracked down. Fixes: #587
• #667 - Large amounts of memory utilized during data normalization.
• #683 - TooManyUpdates error when saving SSM parameter in StreamAlert Apps. Fixes: #682
• #688 - Unsupported types in log would not cause the log’s classification to fail. Fixes: #676
• #731 - Slack messages could be split on each character if no newline character existed, causing hundreds of message to be sent to Slack.
• #732 - Shredding the Lambda tmp directory before downloading S3 objects to prevent disk space from filling up.
• #735 - Catching KeyError during classification that could occur if log is misclassified as the wrong type.
• #741 - Supporting space-delimited logs in the csv parser by casting the unicode delimiter value to a string.

To view the complete list of all of the bugs fixed in v2.0.0, including many not mentioned above, see here.

This release also added various new schemas and rules.