gNMI filtering: Where extension

[karimra]

gNMI Filtering

Problem Statement

In the rapidly evolving landscape of network management, the gNMI
(gRPC Network Management Interface) specification has emerged as a pivotal
protocol for the configuration and monitoring of network devices. While gNMI
provides a robust framework for network telemetry, it currently lacks an
efficient and standardized filtering mechanism. This gap poses several
significant challenges:

1. High Data Volume: Modern networks generate vast amounts of telemetry
   data. Without effective filtering, network management systems are inundated
   with irrelevant or redundant data, leading to unnecessary processing overhead
   and increased storage requirements.

2. Network Bandwidth Consumption: Unfiltered data streams contribute to
   excessive bandwidth usage, which can degrade network performance and increase
   operational costs. Efficient filtering would enable the transmission of only
   pertinent data, optimizing network bandwidth utilization.

3. Performance and Scalability: The absence of filtering mechanisms impacts
   the performance and scalability of network management solutions. Systems must
   handle and process large datasets, which can strain resources and hinder
   real-time analysis and decision-making capabilities.

4. Operational Efficiency: Network operators need precise and relevant data
   to make informed decisions. The lack of filtering complicates data extraction,
   analysis, and correlation, thereby reducing operational efficiency and
   increasing the likelihood of human error.

5. Security and Compliance: Transmitting all data without filtering can
   expose sensitive information unnecessarily, raising security and compliance
   risks.
   Implementing filtering ensures that only authorized and relevant data is
   transmitted, enhancing data privacy and security compliance.

Therefore, to address these challenges and improve the overall efficacy of
network management, it is imperative to incorporate a comprehensive filtering
mechanism within the gNMI specification. This enhancement will facilitate more
efficient data handling, optimize network performance, and empower network
operators with the precise information needed for effective management and
decision-making.

Proposed Solution

To address the identified challenges, a filtering mechanism called the Where
extension will be added to the PathElem message.
This requires a new extension field to be added to the PathElem message

// PathElem encodes an element of a gNMI path, along with any attributes (keys)
// that may be associated with it.
// Reference: gNMI Specification Section 2.2.2.
message PathElem {
  string name = 1;              // The name of the element in the path.
  map<string, string> key = 2;  // Map of key (attribute) name to value.
  repeated gnmi_ext.Extension extension = 3;
}

The Where extension can be included in a PathElem message that is part of:

• A Path or Prefix in a GetRequest message
• A Prefix in a SubscriptionList message
• A Path in a Subscription message

This extension allows clients to specify conditions that must be met for a
path's children to be included in the response(s).

A single Where extension must be added to a PathElem but multiple Where
extensions can be added to a Path under different PathElem messages to
form complex conditions. All Where conditions under different PathElem in the
same Path must evaluate to true for the children of that path to be included
in the response(s).

Where Message Definition

The Where message enables filtering based on conditions defined by expressions,
paths, or values. It includes the following components:

message Where {
  oneof op {
    // Expression defines the condition that needs to be evaluated.
    // It is composed of an operation (one of WhereOp) and 2 operands
    // (left and right).
    Expression expr  = 1;
    // Path is a gNMI path relative the `PathElem` to which the
    // 'Where' message was appended.
    Path       path  = 2;
    // Value is a list of values to be evaluated against a path.
    Value      value = 3;
  }
}

Expression Message Definition

The Expression message defines an operator node in a binary expression tree,
specifying the condition to be evaluated using an operation op and two
operands called left and right.

// Expression defines the condition that needs to be evaluated.
// It is composed of an operation (one of WhereOp) and 2 operands
// (left and right).
message Expression {
  // defines the operation to be applied to left and right.
  WhereOp op    = 1;
  // defines the left operand.
  Where   left  = 2;
  // defines the right operand.
  Where   right = 3;
}

Valid and Invalid Combinations of Left and Right Operands

When constructing an Expression, it is crucial to ensure the operands are
appropriately paired to form valid expressions. The following outlines the
valid and invalid combinations:

1. Valid Combinations:

• Expression and Expression: Both left and right can be Expression types.

expr: {
  op: AND
  left: { expr: { ... } }
  right: { expr: { ... } }
}

• Expression and Path: One operand is an Expression and the other is a Path.

expr: {
  op: EQUAL
  left: { expr: { ... } }
  right: { path: { ... } }
}

• Expression and Value: One operand is an Expression and the other is a Value.

expr: {
  op: EQUAL
  left: { expr: { ... } }
  right: { value: { ... } }
}

Path and Value: One operand is a Path and the other is a Value.

expr: {
  op: EQUAL
  left: { path: { ... } }
  right: { value: { ... } }
}

2. Invalid Combinations:

• Both Operands as Values: Both left and right cannot be Value types.

// Invalid example
expr: {
  op: EQUAL
  left: { value: { ... } }
  right: { value: { ... } }
}

• Both Operands as Paths: Both left and right cannot be Path types.

// Invalid example
expr: {
  op: EQUAL
  left: { path: { ... } }
  right: { path: { ... } }
}

Ensuring the proper combination of operands is essential to maintaining the
integrity of the expression tree and preventing logical errors in the
evaluation process.

Path Message Definition

The Path message defines an operand in an Expression, it is a relative path that
begins at the PathElem to which the Where message was appended.

// Path defines a gNMI relative path as an operand of a WhereOp.
// It is relative to the `PathElem`
// to which the `Where` message was appended.
message Path {
  repeated string path = 1;
}

A Path operand returns the value of the leaf or leaf-list it points to.
If the path points to a container or a list, it returns a boolean true if the container or list exists
in the data tree; otherwise, it returns false.

Value Message Definition

The Value message defines a node in the binary expression tree, allowing
one or multiple values to be specified based on the WhereOp value.

// Value defines a message for a value node in the binary expression tree.
// one or multiple values can be specified based on the WhereOp value.
message Value {
  oneof value_type {
    int64 int_val = 1;
    uint64 uint_val = 2;
    string string_val = 3;
    bool bool_val = 4;
    double double_val = 5;
    ValueList list_val = 6;
  }
}

message ValueList {
  repeated Value values = 1;
}

WhereOp Enum Definition

The WhereOp enum defines the list of supported operations for filtering
conditions.

// WhereOp defines the list of supported operations.
enum WhereOp  {
  // Must be treated as an error
  UNSPECIFIED           =  0;
  // The AND operation returns true only if both operands are true,
  // otherwise it returns false.
  AND                   =  1;
  // The OR operation returns true if at least one of the operands is true,
  // otherwise it returns false.
  OR                    =  2;
  // The NOT operation returns true if the operand is false,
  // and false if the operand is true. It inverts the boolean value.
  NOT                   =  3;
  // The EQUAL operation returns true if both operands are equal,
  // otherwise it returns false.
  EQUAL                 =  4;
  // The NotEqual operation returns true if the operands are not equal,
  // otherwise it returns false.
  NOT_EQUAL             =  5;
  // The LessThan operation returns true if the left operand is less than
  // the right operand; otherwise, it returns false.
  LESS_THAN             =  6;
  // The GreaterThan operation returns true if the left operand is greater
  // than the right operand; otherwise, it returns false.
  GREATER_THAN          =  7;
  // The LessThanOrEqual operation returns true if the left operand is less
  // than or equal to the right operand; otherwise, it returns false.
  LESS_THAN_OR_EQUAL    =  8;
  // The GreaterThanOrEqual operation returns true if the left operand is
  // greater than or equal to the right operand; otherwise, it returns false.
  GREATER_THAN_OR_EQUAL =  9;
  // The IN operation returns true if the left operand is found within the
  // collection specified by the right operand; otherwise, it returns false.
  IN                    = 10;
  // The NOT_IN operation returns true if the left operand is not found within
  // the collection specified by the right operand; otherwise, it returns false.
  NOT_IN                = 11;
}

Implementation Details

Expression Tree Depth

Implementations can define a maximum binary expression tree depth.
If an expression exceeds this maximum depth, the server must return
an error with code ResourceExhausted.

Operations

This extension defines a list of operations used to evaluate expressions.
This list can be extended in the future.
If an implementation does not support a operation received in a Request,
it must return an error with code Unimplemented.

AND

The AND operation takes two operands of type boolean.
It returns true only if both operands are true; otherwise, it returns false.
If one of the operands is not a boolean, an InvalidArgument error must be
returned.

OR

The OR operation takes two operands of type boolean.
It returns true if at least one of the operands is true; otherwise, it returns
false.
If one of the operands is not a boolean, an InvalidArgument error must be
returned.

NOT

The NOT or ! operation takes one operand of type boolean.
It returns true if the operand is false, and false if the operand is true.
It inverts the boolean value. If the operand is not a boolean, an
InvalidArgument error must be returned.
If more than one operand is set together with the NOT operation,
an InvalidArgument error must be returned.

EQUAL

The EQUAL or == operation takes two operands of the same type.
It returns true if both operands are equal; otherwise, it returns false.

The comparison must be based on the underlying operand type.

If the two operands are not of the same type, an InvalidArgument error must
be returned.

NOT_EQUAL

The NOT_EQUAL or != operation takes two operands of the same type,
if returns true if the operands are not equal,
otherwise it returns false.

The comparison must be based on the underlying operands type.

If the two operands are not of the same type an InvalidArgument error must
be returned.

LESS_THAN

The LESS_THAN or < operation takes two operands of the same type.
It returns true if the left operand is less than the right operand;
otherwise, it returns false.
The comparison must be based on the underlying value type.

The operands must be of one of the following types: int64, uint64,
or double.
If the two operands are not of type int64, uint64, or double,
an InvalidArgument error must be returned.
If the two operands are not of the same type, an InvalidArgument error
must be returned.

GREATER_THAN

The GREATER_THAN or > operation takes two operands of the same type.
It returns true if the left operand is greater than the right operand;
otherwise, it returns false.
The comparison must be based on the underlying value type.

The operands must be of one of the following types: int64, uint64,
or double.
If the two operands are not of type int64, uint64, or double,
an InvalidArgument error must be returned.
If the two operands are not of the same type, an InvalidArgument
error must be returned.

LESS_THAN_OR_EQUAL

The LESS_THAN_OR_EQUAL or <= operation takes two operands of the same type.
It returns true if the left operand is less than or equal to the
right operand; otherwise, it returns false.

The operands must be of one of the following types: int64, uint64,
or double.
If the two operands are not of type int64, uint64, or double,
an InvalidArgument error must be returned.
If the two operands are not of the same type, an InvalidArgument error must
be returned.

GREATER_THAN_OR_EQUAL

The GREATER_THAN_OR_EQUAL or => operation takes two operands of the same type.
It returns true if the left operand is greater than or equal to the
right operand; otherwise, it returns false.

The operands must be of one of the following types:
int64, uint64, or double.
If the two operands are not of type int64, uint64, or double,
an InvalidArgument error must be returned.
If the two operands are not of the same type, an InvalidArgument
error must be returned.

IN

The IN or in operation takes two operands. The left operand's type must be one of
the following:
int64, uint64, double, string, or bool.
The right operand must be a list_val.

It returns true if the left operand is found within the collection
specified by the right operand; otherwise, it returns false.

If the left operand is not of type int64, uint64, double, string,
or bool, an error code of InvalidArgument error must be returned.
If the right operand is not of type list_val, an InvalidArgument error
must be returned.

NOT_IN

The NOT_IN or !in operation takes two operands. The left operand's type must be one
of the following:
int64, uint64, double, string, or bool.
The right operand must be a list_val.

It returns true if the left operand is not found within the collection
specified by the right operand; otherwise, it returns false.

If the left operand is not of type int64, uint64, double, string,
or bool, an error code of InvalidArgument error must be returned.
If the right operand is not of type list_val, an InvalidArgument error
must be returned.

Example Usages

Here are a few examples of how the Where extension can be used in gNMI
messages:

1. GetRequest with Path Filtering:

Get the list interface names where the oper-status is UP.

String representation:

interfaces/interface(state/oper-status == "UP")/name

Proto message:

path:  {
  elem:  {
    name:  "interfaces"
  }
  elem:  {
    name:  "interface"
    extension:  {
      where:  {
        expr:  {
          op:  EQUAL
          left:  {
            path:  {
              path:  "state"
              path:  "oper-status"
            }
          }
          right:  {
            value:  {
              string_val:  "UP"
            }
          }
        }
      }
    }
  }
  elem:  {
    name:  "name"
  }
}

2. Subscribe to Interface Counters only for interfaces with oper-status "UP".

String representation:

interfaces/interface/state(oper-status == "UP")/counters

Proto message:

subscribe: {
  subscription: {
    path: {
      elem: {
        name: "interfaces"
      }
      elem: {
        name: "interface"
      }
      elem: {
        name: "state"
        extension: {
          where: {
            expr: {
              op: EQUAL
              left: {
                path: {
                  path: "oper-status"
                }
              }
              right: {
                value: {
                  string_val: "UP"
                }
              }
            }
          }
        }
      }
      elem: {
        name: "counters"
      }
    }
    mode: SAMPLE
  }
}

3. Combine multiple conditions together

This examples shows how the Where extension expressions can be nested, allowing
to combine multiple conditions together.

String representation:

interfaces/interface(
  (subinterfaces/subinterface/state/ipv4/addresses/address/ip IN ["192.168.0.1", "192.168.0.2"])
  AND
  (state/oper-status == "UP")
)/state

Proto message:

subscribe:  {
  subscription:  {
    path:  {
      elem:  {
        name:  "interfaces"
      }
      elem:  {
        name:  "interface"
        extension:  {
          where:  {
            expr:  {
              op:  AND
              left:  {
                expr:  {
                  op:  IN
                  left:  {
                    path:  {
                      path:  "subinterfaces"
                      path:  "subinterface"
                      path:  "state"
                      path:  "ipv4"
                      path:  "addresses"
                      path:  "address"
                      path:  "ip"
                    }
                  }
                  right:  {
                    value:  {
                      list_val:  {
                        values:  {
                          string_val:  "192.168.0.1"
                        }
                        values:  {
                          string_val:  "192.168.0.2"
                        }
                      }
                    }
                  }
                }
              }
              right:  {
                expr:  {
                  op:  EQUAL
                  left:  {
                    path:  {
                      path:  "state"
                      path:  "oper-status"
                    }
                  }
                  right:  {
                    value:  {
                      string_val:  "UP"
                    }
                  }
                }
              }
            }
          }
        }
      }
      elem:  {
        name:  "interface"
      }
      elem:  {
        name:  "state"
      }
    }
  }
}

4. Subscribe to System Memory Utilization Where Utilization is Greater Than 80%

String representation:

system/memory/state(used > 80)/used

Proto message:

subscribe:  {
  subscription:  {
    path:  {
      elem:  {
        name:  "system"
      }
      elem:  {
        name:  "memory"
      }
      elem:  {
        name:  "state"
        extension:  {
          where:  {
            expr:  {
              op:  GREATER_THAN
              left:  {
                path:  {
                  path:  "used"
                }
              }
              right:  {
                value:  {
                  uint64_val:  80
                }
              }
            }
          }
        }
      }
    }
  }
}

5. Subscribe to BGP neighbor state data based on ASN numbers

String representation:

network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/state(peer-as IN [65001, 65002])

Proto message:

subscribe: {
  subscription: {
    path: {
      elem: {
        name: "network-instances"
      }
      elem: {
        name: "network-instance"
      }
      elem: {
        name: "protocols"
      }
      elem: {
        name: "protocol"
      }
      elem: {
        name: "bgp"
      }
      elem: {
        name: "neighbors"
      }
      elem: {
        name: "neighbor"
      }
      elem: {
        name: "state"
        extension: {
          where: {
            expr: {
              op: IN
              left: {
                path: {
                  path: "peer-as"
                }
              }
              right: {
                value: {
                  list_val: {
                    values: {
                      uint_val: 65001
                    }
                    values: {
                      uint_val: 65002
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

[abtuteja]

This looks fine, but this can put additional stress on the end device's CPU/memory considering the filtering can span arbitrary nodes.