[META]Add flatten Command to PPL

[YANG-DB]

Is your feature request related to a problem?
OpenSearch Piped Processing Language (PPL) currently lacks a native command to flatten nested objects or arrays in documents. Many datasets, especially those containing JSON objects, have deeply nested fields that are difficult to work with in their raw form. The flatten command will simplify these structures and make it easier to analyze and extract data.

What solution would you like?
Introduce a flatten command in PPL that can handle arrays or nested fields, producing a flattened result that contains all the nested elements at the top level.

Syntax:

source=<data_source> | flatten <nested_field>  | fields <fields_to_select>

• The flatten command takes a nested array or object field and returns each element as part of a flat structure.

---

Example Use Cases

1. Flattening an Array Field

source=my-index  | flatten bridges | fields _time, bridges, city, country

This query flattens the bridges array field.

Example Input:

{
  "_time": "2024-09-13T12:00:00",
  "bridges": [
    {"name": "Tower Bridge", "length": 801},
    {"name": "London Bridge", "length": 928}
  ],
  "city": "London",
  "country": "England"
}

Expected Output:

[
  {
    "_time": "2024-09-13T12:00:00",
    "name": "Tower Bridge",
    "length": 801,
    "city": "London",
    "country": "England"
  },
  {
    "_time": "2024-09-13T12:00:00",
    "name": "London Bridge",
    "length": 928,
    "city": "London",
    "country": "England"
  }
]

2. Flattening a Nested Object

source=my-index | flatten details | fields _time, details 

This query flattens the details object field.

Example Input:

{
  "_time": "2024-09-13T12:00:00",
  "details": {
    "name": "Alice",
    "age": 30,
    "address": {
      "street": "Main St",
      "city": "New York"
    }
  }
}

Expected Output:

{
  "_time": "2024-09-13T12:00:00",
  "name": "Alice",
  "age": 30,
  "street": "Main St",
  "city": "New York"
}

---

Additional Considerations

• The flatten command should work efficiently with large arrays or deeply nested structures.
• It must handle complex JSON objects where multiple levels of nesting exist.
• Consider supporting multi-level flattening for more deeply nested fields (e.g., flatten details.address).

---

Support for PPL flattens functionality is required for both:

OpenSearch based PPL engine

• [FEATURE]Add flatten Command to PPL #3030

Spark based PPL engine

• [FEATURE]Add flatten Command to PPL opensearch-spark#669