Request level latency tracking

[dzane17]

Description

Per Request level tracking: As part of this, we will offer further breakdown of existing took time in search response. To do this, we will introduce a new field(phase_took) in search response which will give more insights/visibility into overall time taken by different search phases(query/fetch/canMatch etc) to the clients.

% curl -XGET 'localhost:9200/_search?pretty&phase_took' -H 'Content-Type: application/json' -d'
{                                                
 "query": { "query_string": { "query": "abc" } }
}'
{
  "took" : 92,
  "phase_took" : {
    "dfs_prequery" : 0,
    "can_match" : 0,
    "query" : 66,
    "fetch" : 4,
    "expand_search" : 0
  },
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

Testing

Local testing with IDE debugger
https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#launching-and-debugging-from-an-ide

1. Create an index and ingest some data

curl -X PUT "localhost:9200/test2?pretty" -H 'Content-Type: application/json' -d' 
{                                                
  "settings": {
    "number_of_shards": 10
  }
}'

curl -X POST "localhost:9200/_bulk?pretty" -H 'Content-Type: application/json' -d'
{ "index" : { "_index" : "test2", "_id" : "1" } }
{ "field1" : "value1" }                          
{ "index" : { "_index" : "test2", "_id" : "2" } }
{ "field2" : "value2" }
{ "index" : { "_index" : "test2", "_id" : "3" } }
{ "field3" : "value3" }
{ "index" : { "_index" : "test2", "_id" : "4" } }
{ "field4" : "value4" }
'

2. Search request with phase_took query parameter

curl -XGET 'localhost:9200/_search?pretty&phase_took' -H 'Content-Type: application/json' -d'
{
 "query": { "query_string": { "query": "abc" } }
}'

3. Update and verify cluster setting

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{"transient" : {"search.phase_took_enabled" : true}}'
curl -X GET "localhost:9200/_cluster/settings?pretty&flat_settings"

4. Search request without query parameter

curl -XGET 'localhost:9200/_search?pretty' -H 'Content-Type: application/json' -d'
{
 "query": { "query_string": { "query": "abc" } }
}'

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: null ❌
• URL:
• CommitID: b0e4ea2
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[sgup432]

These are latency metrics, so why use total in naming? And also use MeanMetrics considering this is related to latency.
Same for others.

[sgup432]

We need to have version guards check around this. For now, use 3.0 version.

[sgup432]

We need to have version guards check around this change as it is a new feature. For now, use 3.0 version. We will eventually change it to 2.11 later on.