Fix versions for changes in required_fileds #4640

shashank-elastic · 2025-04-23T12:38:20Z

Pull Request

Issue link(s): Identified while testing #4621

Summary - What I changed

Fix minstack versions for rules with changes in required fields
For winloag.api filed changes we have a tuning in place - [Rule Tuning] Replace legacy winlog.api usage #4647

winlog.api filed investigations

For the below rules
- Windows Event Logs Cleared
- User Added to Privileged Group
- Active Directory Group Modification by SYSTEM
We have changes in required filed winlog.api

In 8.18

        {
         "ecs": false,
         "name": "winlog.api",
         "type": "keyword"
     },

In 9.0

        {
         "ecs": false,
         "name": "winlog.api",
         "type": "unknown"
     },

For the below rule

Execution of a Downloaded Windows Script
We have changes in ecs filed file.origin_referrer_url,file.origin_url

In 8.18

     {
         "ecs": false,
         "name": "file.origin_referrer_url",
         "type": "unknown"
     },
     {
         "ecs": false,
         "name": "file.origin_url",
         "type": "unknown"
     },

In 9.0

 {
         "ecs": true,
         "name": "file.origin_referrer_url",
         "type": "keyword"
     },
     {
         "ecs": true,
         "name": "file.origin_url",
         "type": "keyword"
     },

How To Test

Unit Test to pass
Local 9.0.0 ECS schema is verified to have this filed and type recorded.

Checklist

Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
Added the meta:rapid-merge label if planning to merge within 24 hours
Secret and sensitive material has been managed correctly
Automated testing was updated or added to match the most common scenarios
Documentation and comments were added for features that require explanation

Contributor checklist

Have you signed the contributor license agreement?
Have you followed the contributor guidelines?

github-actions · 2025-04-23T12:38:34Z

tradebot-elastic · 2025-04-23T12:38:56Z

⛔️ Test failed

Results

❌ Windows Event Logs Cleared (kuery)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ User Added to Privileged Group (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Active Directory Group Modification by SYSTEM (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Execution of a Downloaded Windows Script (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta

Copilot

Pull Request Overview

This PR updates the versioning and ECS field descriptions for several Windows detection rules. Key changes include:

Updated the updated_date to "2025/04/23".
Set min_stack_version to "9.0.0" and updated min_stack_comments for accurate ECS field type changes.
Changes span four rule files related to Windows persistence, group modification, script execution, and security logs.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
rules/windows/persistence_user_account_added_to_privileged_group_ad.toml	Updated version and ECS message for winlog.api field type change.
rules/windows/persistence_group_modification_by_system.toml	Updated version and ECS message for winlog.api field type change.
rules/windows/execution_windows_script_from_internet.toml	Updated version and ECS message for file.origin_referrer_url and file.origin_url type change.
rules/windows/defense_evasion_clearing_windows_security_logs.toml	Updated version and ECS message for winlog.api field type change.

rules/windows/persistence_user_account_added_to_privileged_group_ad.toml

rules/windows/persistence_group_modification_by_system.toml

rules/windows/execution_windows_script_from_internet.toml

rules/windows/defense_evasion_clearing_windows_security_logs.toml

tradebot-elastic · 2025-04-23T12:42:32Z

⛔️ Test failed

Results

❌ Windows Event Logs Cleared (kuery)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ User Added to Privileged Group (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Active Directory Group Modification by SYSTEM (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Execution of a Downloaded Windows Script (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta

Copilot

Pull Request Overview

This PR updates the minimum stack version and related comment metadata for several Windows detection rules to reflect required changes in ECS field types between stack versions 8.18 and 9.0.0.

Updated the min_stack_version and min_stack_comments fields for rules with required field changes in winlog.api.
Updated the min_stack_version and min_stack_comments fields for the rule updating file event fields (file.origin_referrer_url and file.origin_url).

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
rules/windows/persistence_user_account_added_to_privileged_group_ad.toml	Updated metadata with new stack version and comments for winlog.api
rules/windows/persistence_group_modification_by_system.toml	Updated metadata with new stack version and comments for winlog.api
rules/windows/execution_windows_script_from_internet.toml	Updated metadata with new stack version and comments for file fields
rules/windows/defense_evasion_clearing_windows_security_logs.toml	Updated metadata with new stack version and comments for winlog.api

rules/windows/execution_windows_script_from_internet.toml

tradebot-elastic · 2025-04-23T14:04:27Z

⛔️ Test failed

Results

❌ Windows Event Logs Cleared (kuery)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ User Added to Privileged Group (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Active Directory Group Modification by SYSTEM (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta
❌ Execution of a Downloaded Windows Script (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta

shashank-elastic · 2025-04-23T14:37:07Z

Debugging the "unknown" type for filed winlog.api

Windows Schema Versions

schemas['windows'].keys()
dict_keys(['1.0.0', '1.12.1', '1.12.4', '1.14.0', '1.14.1', '1.15.0', '1.15.1', '1.15.2', '1.16.0', '1.17.0', '1.18.0', '1.19.0', '1.19.1', '1.19.2', '1.2.2', '1.20.0', '1.20.1', '1.21.0', '1.21.1', '1.22.0', '1.23.0', '1.24.0', '1.24.1', '1.25.0', '1.26.0', '1.27.0', '1.28.0', '1.28.1', '1.29.0', '1.30.0', '1.31.0', '1.32.0', '1.33.0', '1.34.0', '1.34.1', '1.35.0', '1.36.0', '1.37.0', '1.38.0', '1.39.0', '1.40.0', '1.41.0', '1.41.1', '1.42.0', '1.42.1', '1.43.0', '1.44.0', '1.44.1', '1.44.2', '1.44.3', '1.44.4', '1.44.5', '1.45.0', '1.45.1', '1.46.0', '1.46.1', '1.47.0', '1.5.0', '2.0.0', '2.0.1', '2.0.2', '2.1.0', '2.2.0', '2.3.0', '2.3.1', '2.3.2', '2.3.3', '2.3.4', '2.3.5', '2.3.6', '2.4.0', '2.4.1', '2.4.2', '2.5.0', '2.5.1', '2.5.2'])

Picking Latest

schemas['windows']['2.5.2'].keys()
dict_keys(['applocker_exe_and_dll', 'applocker_msi_and_script', 'applocker_packaged_app_deployment', 'applocker_packaged_app_execution', 'forwarded', 'perfmon', 'powershell', 'powershell_operational', 'service', 'sysmon_operational', 'windows_defender'])

Finding "winlog.api" in the schema in all possible datastreams ( the filed is not availble in perfmon and service datastreams)

 schemas['windows']['2.5.2']['applocker_exe_and_dll']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.2']['applocker_msi_and_script']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.2']['applocker_packaged_app_deployment']['winlog.api']
'keyword'
>>schemas['windows']['2.5.2']['applocker_packaged_app_execution']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.2']['forwarded']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.2']['powershell']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.2']['powershell_operational']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.2']['sysmon_operational']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.2']['windows_defender']['winlog.api']
'keyword'

Now Drilling Down on 2.5.1 version

schemas['windows']['2.5.1']['applocker_exe_and_dll']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.1']['applocker_msi_and_script']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.1']['applocker_packaged_app_deployment']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.1']['applocker_packaged_app_execution']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.1']['forwarded']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.1']['powershell']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.1']['powershell_operational']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.1']['sysmon_operational']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.1']['windows_defender']['winlog.api']
'keyword'

Now Drilling Down on 2.5.0 version

 schemas['windows']['2.5.0']['applocker_exe_and_dll']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.0']['applocker_msi_and_script']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.0']['applocker_packaged_app_deployment']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.0']['applocker_packaged_app_execution']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.0']['forwarded']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.0']['powershell']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.0']['powershell_operational']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.0']['sysmon_operational']['winlog.api']
'keyword'
>>> schemas['windows']['2.5.0']['windows_defender']['winlog.api']
'keyword'

For 9.0.0 The windows integration supported is on ^2.5.0 in all these locally the schemas everything is mapped to keyword and nothing is "unknown"

System Schema Versions

schemas['system'].keys()
dict_keys(['1.0.0', '1.1.2', '1.11.0', '1.16.2', '1.17.0', '1.18.0', '1.19.0', '1.19.1', '1.19.2', '1.19.3', '1.19.4', '1.19.5', '1.20.0', '1.20.1', '1.20.2', '1.20.3', '1.20.4', '1.21.0', '1.22.0', '1.23.0', '1.23.1', '1.24.0', '1.24.1', '1.24.2', '1.24.3', '1.25.0', '1.25.1', '1.25.2', '1.25.3', '1.25.4', '1.26.0', '1.27.0', '1.27.1', '1.28.0', '1.29.0', '1.30.0', '1.31.0', '1.31.1', '1.33.0', '1.34.0', '1.34.1', '1.35.0', '1.36.0', '1.36.1', '1.36.2', '1.37.0', '1.37.1', '1.38.0', '1.38.1', '1.38.2', '1.39.0', '1.4.0', '1.40.0', '1.41.0', '1.42.0', '1.43.0', '1.44.0', '1.45.0', '1.46.0', '1.46.1', '1.47.0', '1.47.1', '1.47.2', '1.48.0', '1.49.0', '1.49.1', '1.50.0', '1.50.1', '1.51.0', '1.52.0', '1.53.0', '1.53.1', '1.54.0', '1.55.0', '1.55.1', '1.55.2', '1.56.0', '1.57.0', '1.58.0', '1.58.1', '1.58.2', '1.59.0', '1.59.1', '1.59.2', '1.59.3', '1.59.4', '1.6.2', '1.6.3', '1.6.4', '1.60.0', '1.60.1', '1.60.2', '1.60.3', '1.60.4', '1.60.5', '1.61.0', '1.61.1', '1.62.0', '1.62.1', '1.63.0', '1.63.1', '1.63.2', '1.64.0', '1.64.1', '1.65.0', '1.66.0', '1.66.1', '1.67.0', '1.67.1', '1.67.2', '1.67.3'])

schemas['system']['1.64.0'].keys()
dict_keys(['application', 'auth', 'core', 'cpu', 'diskio', 'filesystem', 'fsstat', 'load', 'memory', 'network', 'process', 'process_summary', 'security', 'socket_summary', 'syslog', 'system', 'uptime'])

The field winlog.api is available only in application, security, system so lets analyse those keys and
For 9.0.0 The system integration supported is on ^1.64.0 lets drill down on these versions.

1.64.0

 schemas['system']['1.64.0']['application']['winlog.api']
'keyword'
>>> schemas['system']['1.64.0']['security']['winlog.api']
'keyword'
>>> schemas['system']['1.64.0']['system']['winlog.api']
'keyword'

1.64.1

>>> schemas['system']['1.64.1']['application']['winlog.api']
'keyword'
>>> schemas['system']['1.64.1']['security']['winlog.api']
'keyword'
>>> schemas['system']['1.64.1']['system']['winlog.api']
'keyword'
>>>

1.65.0

>>> schemas['system']['1.65.0']['application']['winlog.api']
'keyword'
>>> schemas['system']['1.65.0']['security']['winlog.api']
'keyword'
>>> schemas['system']['1.65.0']['system']['winlog.api']
'keyword'
>>>

1.66.0

>>> schemas['system']['1.66.0']['application']['winlog.api']
'keyword'
>>> schemas['system']['1.66.0']['security']['winlog.api']
'keyword'
>>> schemas['system']['1.66.0']['system']['winlog.api']
'keyword'
>>>

1.66.1

>>> schemas['system']['1.66.1']['application']['winlog.api']
'keyword'
>>> schemas['system']['1.66.1']['security']['winlog.api']
'keyword'
>>> schemas['system']['1.66.1']['system']['winlog.api']
'keyword'
>>>

1.67.0 , 1.67.1, 1.67.2, 1.67.3

>>> schemas['system']['1.67.0']['application']['winlog.api']
'keyword'
>>> schemas['system']['1.67.0']['security']['winlog.api']
'keyword'
>>> schemas['system']['1.67.0']['system']['winlog.api']
'keyword'
>>> schemas['system']['1.67.1']['application']['winlog.api']
'keyword'
>>> schemas['system']['1.67.1']['security']['winlog.api']
'keyword'
>>> schemas['system']['1.67.1']['system']['winlog.api']
'keyword'
>>> schemas['system']['1.67.2']['application']['winlog.api']
'keyword'
>>> schemas['system']['1.67.2']['security']['winlog.api']
'keyword'
>>> schemas['system']['1.67.2']['system']['winlog.api']
'keyword'
>>> schemas['system']['1.67.3']['application']['winlog.api']
'keyword'
>>> schemas['system']['1.67.3']['security']['winlog.api']
'keyword'
>>> schemas['system']['1.67.3']['system']['winlog.api']
'keyword'
>>>

In all these versions locally the schemas everything is mapped to keyword and nothing is "unknown"

In the latest beats , winlog.api removed this field, and as the query doesn't set a event.dataset , the code doesn't check the integrations schema. This will be tuned in a spearte PR by @w0rk3r and this PR will tackle other minstack rule.

tradebot-elastic · 2025-04-23T16:47:50Z

⛔️ Test failed

Results

❌ Execution of a Downloaded Windows Script (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta

Copilot

Pull Request Overview

This PR fixes the stack version and associated comments for the Windows rule to align with updated ECS field types in version 9.0.0.

Updated the min_stack_version from 8.15.0 to 9.0.0
Changed the min_stack_comments to reflect the required changes for ECS fields
Updated the updated_date to the current version's date

rules/windows/execution_windows_script_from_internet.toml

Co-authored-by: Copilot <[email protected]>

tradebot-elastic · 2025-04-23T16:50:59Z

⛔️ Test failed

Results

❌ Execution of a Downloaded Windows Script (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta

Copilot

Pull Request Overview

This PR updates the minimum stack version and associated comments for the "Execution of a Downloaded Windows Script" rule to reflect changes in required ECS fields for version 9.0.0.

Updated the min_stack_version, updated_date, and min_stack_comments values in the rule configuration file.

rules/windows/execution_windows_script_from_internet.toml

Co-authored-by: Copilot <[email protected]>

tradebot-elastic · 2025-04-24T00:07:31Z

⛔️ Test failed

Results

❌ Execution of a Downloaded Windows Script (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta

Copilot

Pull Request Overview

This PR updates the rule metadata to align with changes in required ECS fields.

Updated min_stack_version from "8.15.0" to "9.0.0"
Revised updated_date and min_stack_comments to highlight the ECS field type change for file.origin_referrer_url and file.origin_url

Mikaayenson

Might want to update the summary to be more accurate now that @w0rk3r has #4647

rules/windows/execution_windows_script_from_internet.toml

Co-authored-by: Jonhnathan <[email protected]>

tradebot-elastic · 2025-04-24T00:26:27Z

⛔️ Test failed

Results

❌ Execution of a Downloaded Windows Script (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta

tradebot-elastic · 2025-04-24T00:43:36Z

⛔️ Test failed

Results

❌ Execution of a Downloaded Windows Script (eql)
- coverage_issue: no_rta
- stack_validation_failed: no_rta

Fix versions for changes in required_fileds

508f076

shashank-elastic added Rule: Tuning tweaking or tuning an existing rule meta:rapid-merge labels Apr 23, 2025

shashank-elastic requested review from traut, Mikaayenson, w0rk3r, Samirbous, eric-forte-elastic and Copilot April 23, 2025 12:38

shashank-elastic self-assigned this Apr 23, 2025

botelastic bot added Domain: Endpoint OS: Windows windows related rules labels Apr 23, 2025

github-actions bot added the backport: auto label Apr 23, 2025

Copilot AI reviewed Apr 23, 2025

View reviewed changes

shashank-elastic requested a review from Copilot April 23, 2025 12:42

Copilot AI reviewed Apr 23, 2025

View reviewed changes

rules/windows/execution_windows_script_from_internet.toml Outdated Show resolved Hide resolved

Revert winlog.api minstack changes

be4f23b

shashank-elastic force-pushed the fix_versions_required_fileds branch from 51359a3 to be4f23b Compare April 23, 2025 16:47

shashank-elastic requested a review from Copilot April 23, 2025 16:47

Copilot AI reviewed Apr 23, 2025

View reviewed changes

rules/windows/execution_windows_script_from_internet.toml Outdated Show resolved Hide resolved

Update rules/windows/execution_windows_script_from_internet.toml

623fabf

Co-authored-by: Copilot <[email protected]>

shashank-elastic requested a review from Copilot April 23, 2025 16:50

Copilot AI reviewed Apr 23, 2025

View reviewed changes

rules/windows/execution_windows_script_from_internet.toml Outdated Show resolved Hide resolved

Update rules/windows/execution_windows_script_from_internet.toml

73819e1

Co-authored-by: Copilot <[email protected]>

shashank-elastic requested a review from Copilot April 24, 2025 00:09

Copilot AI reviewed Apr 24, 2025

View reviewed changes

Mikaayenson approved these changes Apr 24, 2025

View reviewed changes

w0rk3r reviewed Apr 24, 2025

View reviewed changes

rules/windows/execution_windows_script_from_internet.toml Outdated Show resolved Hide resolved

w0rk3r approved these changes Apr 24, 2025

View reviewed changes

Update rules/windows/execution_windows_script_from_internet.toml

f4081a5

Co-authored-by: Jonhnathan <[email protected]>

eric-forte-elastic approved these changes Apr 24, 2025

View reviewed changes

Merge branch 'main' into fix_versions_required_fileds

677245f

shashank-elastic merged commit 3423116 into main Apr 24, 2025
11 checks passed

shashank-elastic deleted the fix_versions_required_fileds branch April 24, 2025 00:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix versions for changes in required_fileds #4640

Fix versions for changes in required_fileds #4640

shashank-elastic commented Apr 23, 2025 •

edited

Loading

github-actions bot commented Apr 23, 2025

tradebot-elastic commented Apr 23, 2025 •

edited

Loading

Copilot AI left a comment

tradebot-elastic commented Apr 23, 2025 •

edited

Loading

Copilot AI left a comment

tradebot-elastic commented Apr 23, 2025 •

edited

Loading

shashank-elastic commented Apr 23, 2025 •

edited

Loading

tradebot-elastic commented Apr 23, 2025 •

edited

Loading

Copilot AI left a comment

tradebot-elastic commented Apr 23, 2025 •

edited

Loading

Copilot AI left a comment

tradebot-elastic commented Apr 24, 2025 •

edited

Loading

Copilot AI left a comment

Mikaayenson left a comment

tradebot-elastic commented Apr 24, 2025 •

edited

Loading

tradebot-elastic commented Apr 24, 2025 •

edited

Loading

Fix versions for changes in required_fileds #4640

Fix versions for changes in required_fileds #4640

Conversation

shashank-elastic commented Apr 23, 2025 • edited Loading

Pull Request

Summary - What I changed

How To Test

Checklist

Contributor checklist

github-actions bot commented Apr 23, 2025

Rule: Tuning - Guidelines

Documentation and Context

Rule Metadata Checks

Testing and Validation

tradebot-elastic commented Apr 23, 2025 • edited Loading

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

tradebot-elastic commented Apr 23, 2025 • edited Loading

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

tradebot-elastic commented Apr 23, 2025 • edited Loading

shashank-elastic commented Apr 23, 2025 • edited Loading

tradebot-elastic commented Apr 23, 2025 • edited Loading

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

tradebot-elastic commented Apr 23, 2025 • edited Loading

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

tradebot-elastic commented Apr 24, 2025 • edited Loading

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Mikaayenson left a comment

Choose a reason for hiding this comment

tradebot-elastic commented Apr 24, 2025 • edited Loading

tradebot-elastic commented Apr 24, 2025 • edited Loading

shashank-elastic commented Apr 23, 2025 •

edited

Loading

tradebot-elastic commented Apr 23, 2025 •

edited

Loading

tradebot-elastic commented Apr 23, 2025 •

edited

Loading

tradebot-elastic commented Apr 23, 2025 •

edited

Loading

shashank-elastic commented Apr 23, 2025 •

edited

Loading

tradebot-elastic commented Apr 23, 2025 •

edited

Loading

tradebot-elastic commented Apr 23, 2025 •

edited

Loading

tradebot-elastic commented Apr 24, 2025 •

edited

Loading

tradebot-elastic commented Apr 24, 2025 •

edited

Loading

tradebot-elastic commented Apr 24, 2025 •

edited

Loading