Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Host Risk Score package #4019

Closed
wants to merge 66 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
505a5eb
Add yml draft
susan-shu-c Aug 17, 2022
2c7a7c5
Add transforms
susan-shu-c Aug 17, 2022
baa6404
Rename and move transform folder per specifications
susan-shu-c Aug 17, 2022
a6501ab
Add draft of ingest pipeline ymls
susan-shu-c Aug 17, 2022
f8e7232
Add draft manifest
susan-shu-c Aug 17, 2022
38a22a3
Add more ingest scripts
susan-shu-c Aug 17, 2022
77efec9
Move out some scripts to their own file
susan-shu-c Aug 17, 2022
a4076c0
Remove mappings file
susan-shu-c Aug 18, 2022
ee2367b
Hardcode space name as default_space since placeholders don't work here
susan-shu-c Aug 18, 2022
e86e38f
Move reduce script inline
susan-shu-c Aug 18, 2022
cd9e9c2
Move scripts inline
susan-shu-c Aug 18, 2022
f5e7a8c
Move map script inline
susan-shu-c Aug 18, 2022
77418e3
Remove files since I've moved the scripts inline in pivot transform
susan-shu-c Aug 18, 2022
f2004e4
Delete levels script since I've moved it inline in ingest_pipeline
susan-shu-c Aug 18, 2022
100d2b8
Add required files
susan-shu-c Aug 22, 2022
22bbdae
Remove prefix in file names
susan-shu-c Aug 22, 2022
8a66775
Add readme
susan-shu-c Aug 23, 2022
bbe80e6
Move files to transform
susan-shu-c Aug 24, 2022
343b5c7
Create subfolder under transform per spec
susan-shu-c Aug 24, 2022
272e2b2
Make ingest pipeline valid
susan-shu-c Aug 24, 2022
ff9361c
Fix indentation
susan-shu-c Aug 24, 2022
35a26bf
Add field
susan-shu-c Aug 24, 2022
e8d0b5d
Indent
susan-shu-c Aug 24, 2022
8475a2e
Update folder structure
susan-shu-c Aug 24, 2022
595a110
Update packages/host_risk_score/elasticsearch/ingest_pipeline/ingest_…
susan-shu-c Aug 25, 2022
76f1101
Backup .json form
susan-shu-c Aug 26, 2022
4a1c8ac
Update transform names to be more informative
susan-shu-c Aug 26, 2022
712c254
Add latest transform backup
susan-shu-c Aug 29, 2022
a0b409e
Add back transform
susan-shu-c Aug 29, 2022
6682f34
Change transforms to .yml (backup)
susan-shu-c Oct 6, 2022
9edc68e
Rename ingest pipeline file to see if file name is callable
susan-shu-c Oct 6, 2022
588883c
elastic-package check formatted
susan-shu-c Oct 6, 2022
64a424e
Update fields.yml
susan-shu-c Oct 6, 2022
6f9ca1d
Update pipeline name with version
susan-shu-c Oct 6, 2022
c42f52b
Change ingest pipeline file name back so Kibana and elastic-packge be…
susan-shu-c Oct 6, 2022
dd8e3d6
Change sub-field formatting via code review
susan-shu-c Oct 11, 2022
f2a188e
Update packages/host_risk_score/elasticsearch/transform/pivot_transfo…
susan-shu-c Oct 11, 2022
3193952
Apply changes from qn895 v0.1.0
susan-shu-c Dec 21, 2022
e8cbf1b
Fix fields and format with elastic-package check - install successful
susan-shu-c Dec 28, 2022
251912a
Add ecs version dependency
susan-shu-c Dec 28, 2022
c71fe95
Ensure lastest (2nd) transform works by wrapping fields in quotes
susan-shu-c Dec 28, 2022
58f0742
Use ecs field for latest transform
susan-shu-c Dec 28, 2022
5d3806a
Update screenshot
susan-shu-c Dec 28, 2022
28a4b28
Update changelog
susan-shu-c Dec 28, 2022
8ac7285
Update minimum Kibana version dependency
susan-shu-c Dec 29, 2022
791e0e6
Update readme
susan-shu-c Dec 29, 2022
12927c8
Update ingest pipeline and reduce script with new ECS
susan-shu-c Jan 16, 2023
bea2047
Attempt update fields to new ECS fields
susan-shu-c Jan 16, 2023
d4b6fc8
Update .all to .latest
susan-shu-c Jan 16, 2023
a654b92
Update ingest pipeline with new fields (tested, works)
susan-shu-c Jan 16, 2023
cf523b3
Update pivot transform to new ECS fields (tested, works)
susan-shu-c Jan 16, 2023
892c92b
Update latest transform to new ECS fields (tested, works)
susan-shu-c Jan 16, 2023
7efe1a1
Change pivot transform back to 1 hr instead of 1m for testing
susan-shu-c Jan 16, 2023
adb6ee3
Change latest transform sync field to ingest_timestamp to conform wit…
susan-shu-c Jan 16, 2023
14924da
Update package README and add screenshot
susan-shu-c Jan 16, 2023
cca52fe
Update table Markdown format
susan-shu-c Jan 16, 2023
3fa76ca
Update readme
susan-shu-c Jan 17, 2023
7efbae7
Remove trailing spaces in order for format to not merge reduce script…
susan-shu-c Jan 17, 2023
5551aff
Multipliers field to keyword, and remove event.ingested
susan-shu-c Jan 17, 2023
a9b3e10
Update ecs to 8.6.0
susan-shu-c Jan 17, 2023
3f5da48
Remove params (tested, works)
susan-shu-c Jan 17, 2023
140b331
Add note about the default index existing
susan-shu-c Jan 17, 2023
15e994d
Resolve Apoorva review about readme
susan-shu-c Jan 24, 2023
fe1c395
Update CODEOWNERS
susan-shu-c Jan 24, 2023
fdcf6dc
Add description: Pete's code review
susan-shu-c Jan 24, 2023
4516f4c
Update description wording
susan-shu-c Jan 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -215,3 +215,4 @@
/packages/gcp_metrics @elastic/obs-cloud-monitoring
/packages/airflow @elastic/obs-service-integrations
/packages/statsd_input @elastic/obs-service-integrations
/packages/host_risk_score @elastic/ml-ui @elastic/sec-applied-ml
93 changes: 93 additions & 0 deletions packages/host_risk_score/LICENSE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
Elastic License 2.0

URL: https://www.elastic.co/licensing/elastic-license

## Acceptance

By using the software, you agree to all of the terms and conditions below.

## Copyright License

The licensor grants you a non-exclusive, royalty-free, worldwide,
non-sublicensable, non-transferable license to use, copy, distribute, make
available, and prepare derivative works of the software, in each case subject to
the limitations and conditions below.

## Limitations

You may not provide the software to third parties as a hosted or managed
service, where the service provides users with access to any substantial set of
the features or functionality of the software.

You may not move, change, disable, or circumvent the license key functionality
in the software, and you may not remove or obscure any functionality in the
software that is protected by the license key.

You may not alter, remove, or obscure any licensing, copyright, or other notices
of the licensor in the software. Any use of the licensor’s trademarks is subject
to applicable law.

## Patents

The licensor grants you a license, under any patent claims the licensor can
license, or becomes able to license, to make, have made, use, sell, offer for
sale, import and have imported the software, in each case subject to the
limitations and conditions in this license. This license does not cover any
patent claims that you cause to be infringed by modifications or additions to
the software. If you or your company make any written claim that the software
infringes or contributes to infringement of any patent, your patent license for
the software granted under these terms ends immediately. If your company makes
such a claim, your patent license ends immediately for work on behalf of your
company.

## Notices

You must ensure that anyone who gets a copy of any part of the software from you
also gets a copy of these terms.

If you modify the software, you must include in any modified copies of the
software prominent notices stating that you have modified the software.

## No Other Rights

These terms do not imply any licenses other than those expressly granted in
these terms.

## Termination

If you use the software in violation of these terms, such use is not licensed,
and your licenses will automatically terminate. If the licensor provides you
with a notice of your violation, and you cease all violation of this license no
later than 30 days after you receive that notice, your licenses will be
reinstated retroactively. However, if you violate these terms after such
reinstatement, any additional violation of these terms will cause your licenses
to terminate automatically and permanently.

## No Liability

*As far as the law allows, the software comes as is, without any warranty or
condition, and the licensor will not be liable to you for any damages arising
out of these terms or the use or nature of the software, under any kind of
legal claim.*

## Definitions

The **licensor** is the entity offering these terms, and the **software** is the
software the licensor makes available under these terms, including any portion
of it.

**you** refers to the individual or entity agreeing to these terms.

**your company** is any legal entity, sole proprietorship, or other kind of
organization that you work for, plus all organizations that have control over,
are under the control of, or are under common control with that
organization. **control** means ownership of substantially all the assets of an
entity, or the power to direct its management and policies by vote, contract, or
otherwise. Control can be direct or indirect.

**your licenses** are all the licenses granted to you for the software under
these terms.

**use** means anything you do with the software requiring one of your licenses.

**trademark** means trademarks, service marks, and similar rights.
3 changes: 3 additions & 0 deletions packages/host_risk_score/_dev/build/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
dependencies:
ecs:
reference: [email protected]
5 changes: 5 additions & 0 deletions packages/host_risk_score/changelog.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- version: "0.1.0"
changes:
- description: Initial package
type: enhancement
link: https://github.com/elastic/integrations/pull/4019
32 changes: 32 additions & 0 deletions packages/host_risk_score/docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Host Risk Score

The host risk score feature highlights risky hosts from within your environment. It utilizes a transform with a scripted metric aggregation to calculate host risk scores based on alerts that were generated within the past five days. The transform runs hourly to update the score as new alerts are generated.

Each rule’s contribution to the host risk score is based on the rule’s risk score (`signal.rule.risk_score`) and a time decay factor to reduce the impact of stale alerts. The risk score is calculated using a weighted sum where rules with higher time-corrected risk scores also have higher weights. Each host risk score is normalized to a scale of 0 to 100.

Specific host attributes can boost the final risk score. For example, alert activity on a server poses a greater risk than that on a laptop. Therefore, the host risk score is 1.5 times higher if the host is a server. This boosted score is finalized after calculating the weighted sum of the time-corrected risks.

## Installation
susan-shu-c marked this conversation as resolved.
Show resolved Hide resolved

You can install the Host Risk Score package via **Management > Integrations > Host Risk Score**.

To inspect the installed assets, you can navigate to **Stack Management > Data > Transforms**.

Namely, the following transforms are installed with the Host Risk Score package:

| Transform name | Purpose | Source index | Destination index |
|---|---|---|---|
| host_risk_score.pivot_transform | Calculates the Host risk score | `.alerts-security.alerts-default` | `.alerts-security.host-risk-score-[version]` |
| host_risk_score.latest_transform | Surfaces the latest 5 days of the pivot transform results | `.alerts-security.host-risk-score.latest` | `.alerts-security.host-risk-score-latest-[version]` |

![Installation step screenshot](../img/sample-screenshot.png)

Note: This package assumes that `.alerts-security.alerts-default` exists on the cluster, which is generally true. However, in cases such as a brand-new cluster, the `.alerts-security.alerts-default` will not exist yet until the very first alert is triggered on the cluster. One solution would be to create a rule that would trigger an alert, and thus ensure the index exists.

## Additional information

This package is an alternate way to the [Host Risk Score](https://www.elastic.co/guide/en/security/current/host-risk-score.html) Elastic assets (transforms, ingest pipeline) that were originally installed via the _Security App's_ Entity Analytics dashboard.

The notable differences is that this Integration package:
* Only supports the default space, while [the _Security App_](https://www.elastic.co/guide/en/security/current/host-risk-score.html) supports other spaces.
* The Host Risk features in the _Security App_ aren't currently backed by the data coming from this Integration package.
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
processors:
- set:
field: ingest_timestamp
copy_from: _ingest.timestamp
- fingerprint:
fields:
- '@timestamp'
- _id
method: SHA-256
target_field: _id
- script:
ajosh0504 marked this conversation as resolved.
Show resolved Hide resolved
lang: painless
source: |-
double risk_score = (def)ctx['host']['risk']['calculated_score_norm'];
if (risk_score < 20) {
ctx['host']['risk']['calculated_level'] = 'Unknown'
}
else if (risk_score >= 20 && risk_score < 40) {
ctx['host']['risk']['calculated_level'] = 'Low'
}
else if (risk_score >= 40 && risk_score < 70) {
ctx['host']['risk']['calculated_level'] = 'Moderate'
}
else if (risk_score >= 70 && risk_score < 90) {
ctx['host']['risk']['calculated_level'] = 'High'
}
else if (risk_score >= 90) {
ctx['host']['risk']['calculated_level'] = 'Critical'
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
- name: host.name
external: ecs
- name: host.risk.calculated_score_norm
external: ecs
- name: host.risk.calculated_level
external: ecs
- name: host.risk.multipliers
type: keyword
- name: host.risk.rule_risks.rule_name
type: text
susan-shu-c marked this conversation as resolved.
Show resolved Hide resolved
multi_fields:
- name: 'keyword'
type: keyword
- name: host.risk.rule_risks.rule_risk
type: float
- name: host.risk.rule_risks.rule_id
type: keyword
- name: 'ingest_timestamp'
type: date
- name: '@timestamp'
type: date

# Via https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/common/utils/risk_score_modules.ts#L319-L366
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
dest:
index: ".alerts-security.host-risk-score-latest"
ajosh0504 marked this conversation as resolved.
Show resolved Hide resolved
description: This transform gets the latest risk information about hosts in the default Kibana space
frequency: 1h
susan-shu-c marked this conversation as resolved.
Show resolved Hide resolved
latest:
sort: "@timestamp"
unique_key:
- "host.name"
source:
index:
- ".alerts-security.host-risk-score.latest"
sync:
time:
delay: 2s
field: "ingest_timestamp"
_meta:
fleet_transform_version: 0.1.0
susan-shu-c marked this conversation as resolved.
Show resolved Hide resolved
order: 2
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
- name: host.name
external: ecs
- name: host.risk.calculated_score_norm
- name: host.name
external: ecs
- name: host.risk.calculated_score_norm
external: ecs
- name: host.risk.calculated_level
external: ecs
- name: host.risk.multipliers
type: keyword
- name: host.risk.rule_risks.rule_name
type: text
multi_fields:
- name: 'keyword'
type: keyword
- name: host.risk.rule_risks.rule_risk
type: float
- name: host.risk.rule_risks.rule_id
type: keyword
- name: 'ingest_timestamp'
type: date
- name: '@timestamp'
type: date

# Via https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/common/utils/risk_score_modules.ts#L253-L298
Loading