Create Host Risk Score package

[susan-shu-c]

What does this PR do?

Create "host risk score" package, replicating the core functionality of this script.

Note: the previous script version was removed from main in this commit, and the new code was added in the same commit.

The host risk score feature highlights risky hosts from within your environment. It utilizes a transform with a scripted metric aggregation to calculate host risk scores based on alerts that were generated within the past five days.

Related issues

• https://github.com/elastic/security-ml/issues/112

Screenshots

Successful installation of 2x indices and 2x transforms

Populated the pivot transform destination index

Populated the latest transform index

How to test this PR locally

1. Use elastic-package: elastic-package stack up -d -v --version 8.7.0-SNAPSHOT
2. Go to Security -> Alerts -> Manage rules -> Create new rule: Create a rule that just generates a bunch of alerts (e.g. @timestamp: * and runs every 5 seconds). Note: this is a workaround for the error that appears on a fresh stack install with no alerts generated yet - no such index [.alerts-security.alerts-default];
3. Install the host risk score package with the UI or via elastic-package install
4. In the dev tools console, run the following to inspect the results of the host risk score package transforms.

GET .alerts-security.host-risk-score-0.1.0
GET .alerts-security.host-risk-score-latest-0.1.0

GET .alerts-security.host-risk-score.all/_search
{
  "query": {
    "match_all": {}
  }
}

get .alerts-security.host-risk-score-latest.all/_search
{
  "query": {
    "match_all": {}
  }
}

For quick teardown and restart of the local Elastic stack, you can use the below script or variation to reduce manual steps.

docker kill $(docker ps -q) && docker system prune -a -f && cd /Users/susan/Documents/code/integrations/packages/host_risk_score && elastic-package check && cd /Users/susan/Documents/code/integrations && elastic-package stack up -d -v --version 8.7.0-SNAPSHOT

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

[elasticmachine]

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-01-24T20:48:44.330+0000

• Duration: 12 min 28 sec

Steps errors

Expand to view the steps failures

Test integration: host_risk_score

• Took 0 min 6 sec . View more details here
• Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

Google Storage Download

• Took 0 min 0 sec . View more details here

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[ajosh0504]

@susan-shu-c Thanks for kicking this off. Made some comments but some more general ones:

• We can simplify the naming of the assets- The ml_hostriskscore prefix is not necessary. It was a convention we were following in an older deployment mechanism.
• I'm still unsure at what point we'll be able to create mappings for the destination indices of the transforms. Maybe we'll uncover this while testing the package.
• We might want to check if there needs to be a specific naming convention for the destination indices of the transforms. I know datastreams do.

[szeitlin]

I'm still unsure at what point we'll be able to create mappings for the destination indices of the transforms. Maybe we'll uncover this while testing the package.

Let's keep an eye on this in case it ends up being a blocker.

[susan-shu-c]

Weekly update - started putting up scripts, started fixing some things based on reviews/feedback.

[szeitlin]

I believe this is not stale but, still in draft form because it's currently blocked. @susan-shu-c can you please comment with what's left/link related issues that are blocking?

[susan-shu-c]

Thank you @szeitlin - more documentation below 👍

More progress has made outside of this PR, with more information this Issue, namely:

• [ML][Fleet] Update Transform installation mechanism to support upgrade paths  kibana#142920

[szeitlin]

Not stale! Just blocked still!

[susan-shu-c]

Rebased, sorry for the notifications all 😂

[szeitlin]

Update: We are blocked on merging this until we resolve questions about workflow/installation/enablement.

[peluja1012]

Hey @susan-shu-c, just want to make sure we don't merge this until further discussion with the larger Entity Analytics team.

[susan-shu-c]

@peluja1012 Hi, have synced on Sourin some weeks ago and that was the plan! Hence we are holding off on merging this for the time being.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[susan-shu-c]

Blocked for now, when the status changes I'll clean up the tags, but will let the tags be for now.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

[susan-shu-c]

Thank you for the service, bot! This PR will remain here as an archive in the case we're able to reuse it.