Skip to content

Commit

Permalink
[UII] Support searchAfter and PIT (point-in-time) parameters for ge…
Browse files Browse the repository at this point in the history 
…t agents list API (elastic#213486)

## Summary

Resolves elastic#206924.

This PR adds the following query parameters to the agent list API (`GET
/api/fleet/agents`) in order to enable fetching beyond the first 10,000
hits:
```
    searchAfter?: string;
    openPit?: boolean;
    pitId?: string;
    pitKeepAlive?: string;
```

The list agent API response can now include the following properties
```
    // the PIT ID used
    pit?: string;

    // stringified version of the last agent's `sort` field,
    // can be passed as `searchAfter` in the next request
    nextSearchAfter? string;
```

* `searchAfter` can be used with or without a `pitId`. If using
`searchAfter`, `page` parameter is not accepted.

* `searchAfter` expects a stringified array. (Reviewers: I couldn't get
the Kibana request schema to accept a multi-part query param and convert
it to an array... I think this would be better, please let me know if
you know how to get that to work 🙏)

* `pitKeepAlive` duration (i.e. `30s`, `1m`, etc) must be present when
opening a PIT or retrieving results using a PIT ID.

* These can be used with the existing `sortField` and `sortOrder`
params. They default to `enrolled_at` and `desc` respectively.

### Example using only `searchAfter`:

```
# Retrieve the first 10k hits
curl -X GET 'http://<user>:<pass>@<kibana url>/api/fleet/agents?perPage=10000'

# Grab the `nextSearchAfter` param from the response
# Pass it to the new request to retrieve the next page of 10k hits
curl -X GET 'http://<user>:<pass>@<kibana url>/api/fleet/agents?perPage=10000&searchAfter=<nextSearchAfter>'
```

### Example using `searchAfter` with point-in-time parameters:
```
# Retrieve the first 10k hits and open a PIT
curl -X GET 'http://<user>:<pass>@<kibana url>/api/fleet/agents?perPage=10000&openPit=true&pitKeepAlive=5m'

# Grab the `pit` ID from the response
# Grab the `nextSearchAfter` param from the response
# Pass both to the new request to retrieve the next page of 10k hits
curl -X GET 'http://<user>:<pass>@<kibana url>/api/fleet/agents?perPage=10000&searchAfter=<nextSearchAfter>&pitId=<pit id>&pitKeepAlive=5m'
```

## Testing
I recommend using `scripts/create_agents` to generate bulk agents and
testing the above requests. You can generate new agents between PIT
requests to test that using a PIT ID retains the original state. (An API
functional test was added for this)

Note: you may need to add `&showInactive=true` to all requests if your
fake agents become inactive.

TBD

### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: kibanamachine <[email protected]>
  • Loading branch information
@jen-huang @kibanamachine
jen-huang and kibanamachine authored Mar 8, 2025
1 parent c9969e7 commit 3f90203
Show file tree
Hide file tree
Showing 15 changed files with 355 additions and 156 deletions.
84 changes: 41 additions & 43 deletions oas_docs/bundle.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17292,7 +17292,6 @@
"name": "page",
"required": false,
"schema": {
"default": 1,
"type": "number"
}
},
Expand Down Expand Up @@ -17368,6 +17367,38 @@
],
"type": "string"
}
},
{
"in": "query",
"name": "searchAfter",
"required": false,
"schema": {
"type": "string"
}
},
{
"in": "query",
"name": "openPit",
"required": false,
"schema": {
"type": "boolean"
}
},
{
"in": "query",
"name": "pitId",
"required": false,
"schema": {
"type": "string"
}
},
{
"in": "query",
"name": "pitKeepAlive",
"required": false,
"schema": {
"type": "string"
}
}
],
"responses": {
Expand Down Expand Up @@ -17611,20 +17642,7 @@
"type": "number"
},
"sort": {
"items": {
"anyOf": [
{
"type": "number"
},
{
"type": "string"
},
{
"enum": [],
"nullable": true
}
]
},
"items": {},
"type": "array"
},
"status": {
Expand Down Expand Up @@ -17777,12 +17795,18 @@
},
"type": "array"
},
"nextSearchAfter": {
"type": "string"
},
"page": {
"type": "number"
},
"perPage": {
"type": "number"
},
"pit": {
"type": "string"
},
"statusSummary": {
"additionalProperties": {
"type": "number"
Expand Down Expand Up @@ -19665,20 +19689,7 @@
"type": "number"
},
"sort": {
"items": {
"anyOf": [
{
"type": "number"
},
{
"type": "string"
},
{
"enum": [],
"nullable": true
}
]
},
"items": {},
"type": "array"
},
"status": {
Expand Down Expand Up @@ -20156,20 +20167,7 @@
"type": "number"
},
"sort": {
"items": {
"anyOf": [
{
"type": "number"
},
{
"type": "string"
},
{
"enum": [],
"nullable": true
}
]
},
"items": {},
"type": "array"
},
"status": {
Expand Down
84 changes: 41 additions & 43 deletions oas_docs/bundle.serverless.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17292,7 +17292,6 @@
"name": "page",
"required": false,
"schema": {
"default": 1,
"type": "number"
}
},
Expand Down Expand Up @@ -17368,6 +17367,38 @@
],
"type": "string"
}
},
{
"in": "query",
"name": "searchAfter",
"required": false,
"schema": {
"type": "string"
}
},
{
"in": "query",
"name": "openPit",
"required": false,
"schema": {
"type": "boolean"
}
},
{
"in": "query",
"name": "pitId",
"required": false,
"schema": {
"type": "string"
}
},
{
"in": "query",
"name": "pitKeepAlive",
"required": false,
"schema": {
"type": "string"
}
}
],
"responses": {
Expand Down Expand Up @@ -17611,20 +17642,7 @@
"type": "number"
},
"sort": {
"items": {
"anyOf": [
{
"type": "number"
},
{
"type": "string"
},
{
"enum": [],
"nullable": true
}
]
},
"items": {},
"type": "array"
},
"status": {
Expand Down Expand Up @@ -17777,12 +17795,18 @@
},
"type": "array"
},
"nextSearchAfter": {
"type": "string"
},
"page": {
"type": "number"
},
"perPage": {
"type": "number"
},
"pit": {
"type": "string"
},
"statusSummary": {
"additionalProperties": {
"type": "number"
Expand Down Expand Up @@ -19665,20 +19689,7 @@
"type": "number"
},
"sort": {
"items": {
"anyOf": [
{
"type": "number"
},
{
"type": "string"
},
{
"enum": [],
"nullable": true
}
]
},
"items": {},
"type": "array"
},
"status": {
Expand Down Expand Up @@ -20156,20 +20167,7 @@
"type": "number"
},
"sort": {
"items": {
"anyOf": [
{
"type": "number"
},
{
"type": "string"
},
{
"enum": [],
"nullable": true
}
]
},
"items": {},
"type": "array"
},
"status": {
Expand Down
46 changes: 27 additions & 19 deletions oas_docs/output/kibana.serverless.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18758,7 +18758,6 @@ paths:
name: page
required: false
schema:
default: 1
type: number
- in: query
name: perPage
Expand Down Expand Up @@ -18808,6 +18807,26 @@ paths:
- asc
- desc
type: string
- in: query
name: searchAfter
required: false
schema:
type: string
- in: query
name: openPit
required: false
schema:
type: boolean
- in: query
name: pitId
required: false
schema:
type: string
- in: query
name: pitKeepAlive
required: false
schema:
type: string
responses:
'200':
content:
Expand Down Expand Up @@ -18985,12 +19004,7 @@ paths:
nullable: true
type: number
sort:
items:
anyOf:
- type: number
- type: string
- enum: []
nullable: true
items: {}
type: array
status:
enum:
Expand Down Expand Up @@ -19104,10 +19118,14 @@ paths:
- enrolled_at
- local_metadata
type: array
nextSearchAfter:
type: string
page:
type: number
perPage:
type: number
pit:
type: string
statusSummary:
additionalProperties:
type: number
Expand Down Expand Up @@ -19445,12 +19463,7 @@ paths:
nullable: true
type: number
sort:
items:
anyOf:
- type: number
- type: string
- enum: []
nullable: true
items: {}
type: array
status:
enum:
Expand Down Expand Up @@ -19793,12 +19806,7 @@ paths:
nullable: true
type: number
sort:
items:
anyOf:
- type: number
- type: string
- enum: []
nullable: true
items: {}
type: array
status:
enum:
Expand Down
Loading

0 comments on commit 3f90203

Please sign in to comment.