-
Notifications
You must be signed in to change notification settings - Fork 442
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
entityanalytics_okta: split user and device data into their own data …
…streams (#12798) The entityanalytics integrations obtain entity data from a variety of reasonably consistent data sources relating to system entities. These include users and devices, and their associated attributes. Initially the implementations of these integration only reported and indexed user data, but device data collection was added to the integrations whose data sources made that data available. In the case of the entra_id integration the user and device documents were separated into separated data streams, enabling a more refined approach to entity-based risk assessment. In this integration this was not done, resulting in user and device data being placed in a data stream labelled as user data. This can lead to confusing results being presented to analysts. So this change separates user data from device data and presents them in separate data streams, leaving any non-user, non-device data in the generalised entity data stream. The change does not alter the behaviour of the user data stream beyond ensuring that device data does not get included and so is not a breaking change. However, the upgrade path is not transparent so this is marked as a v2 and an upgrade note is added to the documentation.
- Loading branch information
Showing
28 changed files
with
1,604 additions
and
341 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,16 +1,23 @@ | ||
| # Okta Entity Analytics | ||
|
|
||
| This [Okta Entity Analytics](https://www.okta.com/) integration allows users to securely stream User Entities data to Elastic Security via the REST API. When integrated with Elastic Security, this valuable data can be leveraged within Elastic for risk-scoring scenarios (e.g., context enrichments) and detecting advanced analytics (UBA) use cases. | ||
| This [Okta Entity Analytics](https://www.okta.com/) integration allows users to securely stream User and Device Entity data to Elastic Security via the REST API. When integrated with Elastic Security, this valuable data can be leveraged within Elastic for risk-scoring scenarios (e.g., context enrichments) and detecting advanced analytics (UBA) use cases. | ||
|
|
||
| ## Compatibility | ||
|
|
||
| This module has been tested against the Core Okta API version **v1**. | ||
|
|
||
| ## Upgrading to v2 from v1 of the integration | ||
|
|
||
| In v2 of the integration the user and device data was split into separate data streams. The data ingested into your index will be the same but you may need to update device searches if you were using them. | ||
|
|
||
| **NOTE**: When you upgrade from v1 you will need to reconfigure the integration and enable it due to internal changes in the package. See [Resolve conflicts](https://www.elastic.co/guide/en/fleet/current/upgrade-integration.html#resolve-conflicts) in the Fleet documentation for details. | ||
|
|
||
| ## Data streams | ||
|
|
||
| The Okta Entity Analytics integration collects one type of data: user. | ||
| The Okta Entity Analytics integration collects two types of data: user and device. | ||
|
|
||
| **User** is used to retrieve all user logs available in an organization. See more details in the API documentation [here](https://developer.okta.com/docs/reference/api/users/#list-users). | ||
| **Device** is used to retrieve all device logs available in an organization. See more details in the API documentation [here](https://developer.okta.com/docs/api/openapi/okta-management/management/tag/Device/#tag/Device/operation/listDevices). | ||
|
|
||
| ## Requirements | ||
|
|
||
|
|
@@ -56,9 +63,9 @@ The minimum **kibana.version** required is **8.9.0**. | |
|
|
||
| ## Usage | ||
|
|
||
| The Okta provider periodically contacts the Okta API, retrieving updates for users, updates its internal cache of user metadata, and ships updated user metadata to Elasticsearch. | ||
| The Okta provider periodically contacts the Okta API, retrieving updates for users and devices, updates its internal cache of user/device metadata, and ships the updated metadata to Elasticsearch. | ||
|
|
||
| Fetching and shipping updates occurs in one of two processes: **full synchronizations** and **incremental updates**. Full synchronizations will send the entire list of users in state, along with write markers to indicate the start and end of the synchronization event. Incremental updates will only send data for changed users during that event. Changes on a user can come in many forms, whether it be a change to the user’s metadata, or a user was added or deleted. By default, full synchronizations occur every 24 hours and incremental updates occur every 15 minutes. These intervals may be customized to suit your use case. | ||
| Fetching and shipping updates occurs in one of two processes: **full synchronizations** and **incremental updates**. Full synchronizations will send the entire list of users and devices in state, along with write markers to indicate the start and end of the synchronization event. Incremental updates will only send data for changed users/devices during that event. Changes can come in many forms, whether it be a change to the user’s or device’s metadata, or a user or device was added or deleted. By default, full synchronizations occur every 24 hours and incremental updates occur every 15 minutes. These intervals may be customized to suit your use case. | ||
|
|
||
| ## Sample Events | ||
|
|
||
|
|
@@ -110,6 +117,106 @@ A user document: | |
| } | ||
| ``` | ||
|
|
||
| A device document: | ||
|
|
||
| ```json | ||
| { | ||
| "@timestamp": "2023-07-04T09:57:19.786056-05:00", | ||
| "event": { | ||
| "action": "device-discovered", | ||
| }, | ||
| "okta": { | ||
| "created": "2019-10-02T18:03:07Z", | ||
| "id": "deviceid", | ||
| "lastUpdated": "2019-10-02T18:03:07Z", | ||
| "profile": { | ||
| "diskEncryptionType": "ALL_INTERNAL_VOLUMES", | ||
| "displayName": "Example Device name 1", | ||
| "platform": "WINDOWS", | ||
| "registered": true, | ||
| "secureHardwarePresent": false, | ||
| "serialNumber": "XXDDRFCFRGF3M8MD6D", | ||
| "sid": "S-1-11-111" | ||
| }, | ||
| "resourceAlternateID": "", | ||
| "resourceDisplayName": { | ||
| "sensitive": false, | ||
| "value": "Example Device name 1" | ||
| }, | ||
| "resourceID": "deviceid", | ||
| "resourceType": "UDDevice", | ||
| "status": "ACTIVE", | ||
| "_links": { | ||
| "activate": { | ||
| "hints": { | ||
| "allow": [ | ||
| "POST" | ||
| ] | ||
| }, | ||
| "href": "https://localhost/api/v1/devices/deviceid/lifecycle/activate" | ||
| }, | ||
| "self": { | ||
| "hints": { | ||
| "allow": [ | ||
| "GET", | ||
| "PATCH", | ||
| "PUT" | ||
| ] | ||
| }, | ||
| "href": "https://localhost/api/v1/devices/deviceid" | ||
| }, | ||
| "users": { | ||
| "hints": { | ||
| "allow": [ | ||
| "GET" | ||
| ] | ||
| }, | ||
| "href": "https://localhost/api/v1/devices/deviceid/users" | ||
| } | ||
| }, | ||
| "users": [ | ||
| { | ||
| "id": "userid", | ||
| "status": "RECOVERY", | ||
| "created": "2023-05-14T13:37:20Z", | ||
| "activated": "0001-01-01T00:00:00Z", | ||
| "statusChanged": "2023-05-15T01:50:30Z", | ||
| "lastLogin": "2023-05-15T01:59:20Z", | ||
| "lastUpdated": "2023-05-15T01:50:32Z", | ||
| "passwordChanged": "2023-05-15T01:50:32Z", | ||
| "type": { | ||
| "id": "typeid" | ||
| }, | ||
| "profile": { | ||
| "login": "[email protected]", | ||
| "email": "[email protected]", | ||
| "firstName": "name", | ||
| "lastName": "surname" | ||
| }, | ||
| "credentials": { | ||
| "password": {}, | ||
| "provider": { | ||
| "type": "OKTA", | ||
| "name": "OKTA" | ||
| } | ||
| }, | ||
| "_links": { | ||
| "self": { | ||
| "href": "https://localhost/api/v1/users/userid" | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| }, | ||
| "device": { | ||
| "id": "deviceid", | ||
| }, | ||
| "labels": { | ||
| "identity_source": "okta-1" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| Full synchronizations will be bounded on either side by "write marker" documents. | ||
|
|
||
| ```json | ||
|
|
@@ -127,12 +234,14 @@ Full synchronizations will be bounded on either side by "write marker" documents | |
|
|
||
| ## Logs reference | ||
|
|
||
| ### User | ||
| ### Device | ||
|
|
||
| This is the `User` dataset. | ||
| This is the `Device` dataset. | ||
|
|
||
| #### Example | ||
| {{fields "device"}} | ||
|
|
||
| {{event "user"}} | ||
| ### User | ||
|
|
||
| This is the `User` dataset. | ||
|
|
||
| {{fields "user"}} | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
20 changes: 20 additions & 0 deletions
20
packages/entityanalytics_okta/data_stream/device/fields/base-fields.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| - name: data_stream.type | ||
| type: constant_keyword | ||
| description: Data stream type. | ||
| - name: data_stream.dataset | ||
| type: constant_keyword | ||
| description: Data stream dataset. | ||
| - name: data_stream.namespace | ||
| type: constant_keyword | ||
| description: Data stream namespace. | ||
| - name: event.module | ||
| type: constant_keyword | ||
| description: Event module. | ||
| value: entityanalytics_okta | ||
| - name: event.dataset | ||
| type: constant_keyword | ||
| description: Event dataset. | ||
| value: entityanalytics_okta.device | ||
| - name: '@timestamp' | ||
| type: date | ||
| description: Event timestamp. |
6 changes: 6 additions & 0 deletions
6
packages/entityanalytics_okta/data_stream/device/fields/beats.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| - name: input.type | ||
| type: keyword | ||
| description: Type of filebeat input. | ||
| - name: log.offset | ||
| type: long | ||
| description: Log offset. |
107 changes: 107 additions & 0 deletions
107
packages/entityanalytics_okta/data_stream/device/fields/ecs.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| - name: asset | ||
| type: group | ||
| fields: | ||
| - name: category | ||
| type: keyword | ||
| - name: costCenter | ||
| type: keyword | ||
| - name: create_date | ||
| type: date | ||
| - name: id | ||
| type: keyword | ||
| - name: last_seen | ||
| type: date | ||
| - name: last_status_change_date | ||
| type: date | ||
| - name: last_updated | ||
| type: date | ||
| - name: name | ||
| type: keyword | ||
| - name: status | ||
| type: keyword | ||
| - name: type | ||
| type: keyword | ||
| - name: vendor | ||
| type: keyword | ||
| - name: labels | ||
| type: group | ||
| fields: | ||
| - name: identity_source | ||
| type: keyword | ||
| - name: user | ||
| type: group | ||
| fields: | ||
| - name: account | ||
| type: group | ||
| fields: | ||
| - name: activated_date | ||
| type: date | ||
| - name: change_date | ||
| type: date | ||
| - name: create_date | ||
| type: date | ||
| - name: password_change_date | ||
| type: date | ||
| - name: status | ||
| type: group | ||
| fields: | ||
| - name: deprovisioned | ||
| type: boolean | ||
| - name: locked_out | ||
| type: boolean | ||
| - name: password_expired | ||
| type: boolean | ||
| - name: recovery | ||
| type: boolean | ||
| - name: suspended | ||
| type: boolean | ||
| - name: geo | ||
| type: group | ||
| fields: | ||
| - name: city_name | ||
| type: keyword | ||
| - name: country_iso_code | ||
| type: keyword | ||
| - name: name | ||
| type: keyword | ||
| - name: postal_code | ||
| type: keyword | ||
| - name: region_name | ||
| type: keyword | ||
| - name: timezone | ||
| type: keyword | ||
| - name: organization | ||
| type: group | ||
| fields: | ||
| - name: name | ||
| type: keyword | ||
| - name: profile | ||
| type: group | ||
| fields: | ||
| - name: department | ||
| type: keyword | ||
| - name: first_name | ||
| type: keyword | ||
| - name: id | ||
| type: keyword | ||
| - name: job_title | ||
| type: keyword | ||
| - name: last_name | ||
| type: keyword | ||
| - name: manager | ||
| type: keyword | ||
| - name: mobile_phone | ||
| type: keyword | ||
| - name: other_identities | ||
| type: keyword | ||
| - name: primaryPhone | ||
| type: keyword | ||
| - name: secondEmail | ||
| type: keyword | ||
| - name: status | ||
| type: keyword | ||
| - name: type | ||
| type: keyword | ||
| - name: device.serial_number | ||
| type: keyword | ||
| description: The unique serial number serves as a distinct identifier for each device, aiding in inventory management and device authentication. |
56 changes: 56 additions & 0 deletions
56
packages/entityanalytics_okta/data_stream/device/fields/fields.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| - name: entityanalytics_okta | ||
| type: group | ||
| fields: | ||
| - name: device | ||
| type: group | ||
| fields: | ||
| - name: _embedded | ||
| type: flattened | ||
| description: embedded resources related to the device. | ||
| - name: _links | ||
| type: flattened | ||
| description: link relations for the device's current status. | ||
| - name: activated | ||
| type: date | ||
| description: timestamp when transition to ACTIVE status completed. | ||
| - name: created | ||
| type: date | ||
| description: timestamp when device was created. | ||
| - name: id | ||
| type: keyword | ||
| description: unique key for device. | ||
| - name: last_login | ||
| type: date | ||
| description: timestamp of last login. | ||
| - name: last_updated | ||
| type: date | ||
| description: timestamp when device was last updated. | ||
| - name: password_changed | ||
| type: date | ||
| description: timestamp when password last changed. | ||
| - name: profile | ||
| type: group | ||
| fields: | ||
| - name: registered | ||
| type: boolean | ||
| description: Whether the device is registered. | ||
| - name: secure_hardware_present | ||
| type: boolean | ||
| description: Whether the device is using secure hardware. | ||
| - name: "*" | ||
| type: keyword | ||
| - name: status | ||
| type: keyword | ||
| description: current status of device. | ||
| - name: status_changed | ||
| type: date | ||
| description: timestamp when status last changed. | ||
| - name: transitioning_to_status | ||
| type: keyword | ||
| description: target status of an in-progress asynchronous status transition. | ||
| - name: type | ||
| type: flattened | ||
| description: device type that determines the schema for the device's profile. | ||
| - name: users | ||
| type: flattened | ||
| description: Users associated with the device. |
3 changes: 3 additions & 0 deletions
3
packages/entityanalytics_okta/data_stream/device/manifest.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| title: Collect Devices Identities logs from Okta | ||
| dataset: entityanalytics_okta.device | ||
| type: logs |
File renamed without changes.
Oops, something went wrong.